In this paper, we investigate recognizing context over time using physiological signals. Using the CASE dataset we evaluate both unimodal and multimodal approaches to physiological-based context recognition, over time. For recognition, we evaluate a random forest, as well as state-of-the-art neural network. These classifiers are evaluated using accuracy, Kappa, and F1-Macro metrics. Our results suggest that the fusion of EMG signals is more accurate, at recognizing context over time, compared to the fusion of non-EMG physiological signals. Although the fusion of non-EMG has a comparatively higher accuracy, ECG data results in the highest unimodal accuracy. Considering this, we analyze how the signals are correlated, including when the are fused (i.e. multimodal). We also perform a cross-gender analysis (e.g. training on male data and testing on female data) suggesting some generalizability across gender.