In today’s world, the use of hand gestures to control devices has increased significantly. We see the use of hand gestures in application like controlling a presentation, mobiles, gaming, toy-cars, direction systems, automotive engines etc. A new trend has been emerging for many human based electronic products, because of the gesture recognition system. Therefore the need to study the gesture recognition system becomes a must. In our paper we are proposing a model, where we are trying to control a car buggy using hand gestures. The hand gestures will be recognized using a Near Infrared Camera (NIR). This can also be implemented using an ordinary low resolution camera. The better the camera, better the results. This system is known as the Human Vehicle Interaction System (HVI). The HVI possesses the following characteristics, namely like robustness, Stability & Reliability. In our paper we propose the use of series of algorithms to process the image and perform desired action.