Fruits contribute to an essential part of our diet because they are a major source of energy, vitamins, fiber, plant chemicals and nutrients. Fruits are naturally low in fat, sodium & calories and rich in potassium and fiber, vitamin C. A diet high in fruit can help us against cancer, diabetes, heart diseases etc. A system that gives quickly how much calories present in their diet or fruit intake that can be very useful to maintain health without expert dietitian advice. Use of image processing technique is increasing day by day in all fields and including the agriculture, food science etc. Shape, color and texture are the image features which help in classification and calorie estimation of fruits. This paper proposes an algorithm for fruit recognition and its calorie measurement based on the shape, color and texture along with histogram of gradients and GLCM with local binary pattern algorithms for texture segmentation scheme recognizing the fruits and area, major axis , minor axis are calculated by using shape feature to get more accurate calorie value. With the help of nutritional look up table these features are fed to multi SVM classifier for accurate classification. Evaluation is performed in MATLAB software using two database namely real time database and fake plastic fruit database. Results obtained are very close to real calories of the fruit.