This paper presents a novel defect segmentation of fruits based on color features with K-means clustering algorithm. The algorithms Gaussian Mixture Model (GMM), Support Vector Machine (SVM) are used for background removal and color classification respectively. Physical recognition of defected fruit is very time overwhelming. These days, most existing fruit superiority detecting and grading system have the drawback of low efficiency, low speed of grading, high cost and complexity. Although the color is not commonly used for defect segmentation, it produces a high discriminative power for different regions of image. This approach thus provides a feasible robust solution for defect segmentation of fruits. Image processing gives solution for the automated fruit size grading to give precise, dependable, unfailing and quantitative information apart from handling large volumes, which may not be achieved by employing the human graders. The hardware model can also be created by using PIC microcontroller. This will have a good aspect of application in fruit quality detecting industries.