Recent advances in technology have made tremendous amount of multimedia content available. The amount of video content is increasing, due to which the systems that improve the access to the video is needed. Efficiency of the retrieval system depends upon the search method used in the system. The use of inappropriate search method may degrade the performance of the retrieval system. During past years, multimedia storage grows and cost of storing digital data is also cheaper. So there is huge amount of videos available in the multimedia database. It is very difficult task to retrieve the relevant videos from the large database as per the user needs. Hence an effective video retrieval system is required to retrieve the relevant videos from the large database. In order to create an effective video retrieval system, visual perception must be taken into account. In this paper, we have proposed an algorithm for video retrieval based on genre of the video. This algorithm extracts the key frames based on motion detection, Regions of interest with the objects are detected using bounding box method and are annotated over the video and compared with the similar objects from the knowledge based prepared for various genre videos for recognition of objects. It also uses a tree based classifier to identify the genre of the query video and to retrieve the same genre videos from the video database.