In this paper, we present a vision system capable of analyzing underwater videos for detecting and tracking moving object. The video processing system consists of three subsystems, the video texture analysis, object detection and tracking modules. Moving object detection is based on adaptive Gaussian mixture model. The tracking was carried out by the application of the Kalman algorithm that enables the tracking of objects. Unlike existing method, our approach provides a reliable method in which the moving object is detected in unconstrained environments and under several scenarios (murky water, algae on camera lens, moving plants, low contrast, etc.). The proposed approach was tested with 20 underwater videos, achieving an overall accuracy as high as 85%.