When the discussion starts about a computer based automatic facial feature extraction system which can identify face, gesture etc and estimate gender, age, expirations etc. The system asks for a dependable, fast, reliable classification process. This paper presents an approach to extract effective features for face detection and gender classification system. The proposed algorithm converts the RGB image into the YCbCr color space to detect the skin regions in the color image. Finally Gaussian fitted skin color model is used to obtain the likelihood of skin for any pixel of an image. For facial feature extraction we use Gabor filters at five scales and eight orientations. To solve the classification problem this system deploys Adaboost and SVM based classifier. Biometrics is an advanced way of person recognition as it establishes more direct and explicit link with humans than passwords, since biometrics use measurable physiological and behavioral features of a person. In various biometric applications, gender recognition from facial images plays an important role. In this paper gender recognition image sequence have been successfully investigated. Gender recognition plays an important role for a wide range of application in the field of Human Computer Interaction. In this paper, we propose a gender recognition system based on Neural Networks.  The system comprises two modules: a face detector and a gender classifier. The human faces are first detected and localized in the input image. Each detected face is then passed to the gender classifier to determine whether it is a male or female. Both the face detection and gender classification modules employ the same neural network architecture; however, the two modules are trained separately to extract different features for face detection and gender classification.