Human action recognition from the unconstrained surveillance videos and anticipating human action from onset of video is a challenging task. This work focuses on the study of action recognition, action classification followed by action anticipation. We have used the UT Interaction data set containing interactive videos with six types of activities, the system developed classifies three interactive actions punching, pushing and kicking with accuracy. Anticipating action is a probabilistic process and the all possible outcomes are predicted during anticipation. The framework developed can take any interactive video as input from the web camera or any other camera connected to the laptop and will classify the three interactive actions as said above; similarly when the onset of video is given it produces the probable predictions. Action is modeled as a sequence of changes in Spatio Temporal features and histogram of the gradients helps in identifying the changes. The motion boundary descriptors, histogram of oriented gradients and optical flow features are extracted and SVM algorithm is used for classification. Bag of words approach is used for anticipation. Prediction methodologies are highly needed in surveillance environments for efficient monitoring and management.