Food processing industry requires precise and accurate quality assessment of products. This assessment effects the price of product and also the storage and transport decisions. This paper intends to address the automation of quality assessment in mangoes using computer vision. The paper addresses two issues, namely the number of surface injuries and the time-to-maturity of the fruit. A visible spectrum image of the fruits is considered to extract the Region of Interest(RoI), which contain the fruit with minimal background. Each RoI is further subjected to segmentation techniques so as to identify the injured patches as well as the portion of immature(green) area on the fruit surface. The percentage of these areas is used to determine the grade and the time-to-maturity respectively for each product.
The paper deals with two major aspects that influence the price of mangoes and their consumption patterns. They are, the surface injuries on a mango and its time-to-maturity, henceforth referenced as shelf-life.
Injuries and blemishes on the surface of mangoes are caused by either physical harm sustained during harvesting and transport or due to deficit in product quality like pests, disease or uneven ripening. This leads to black-brown patches on the surface of mangoes which are not acceptable to consumers and reduces the grade of mangoes leading to drop in prices. The paper intends to classify the fruits into four grades, Grade1 being the highest, with none or minor injuries and Grade4 being the lowest and not fit for consumption.
The shelf-life of a perishable product influences the transport and storage decisions. The fruits are to be packaged for retail such that all of them do not ripen at once. The paper proposes to estimate the shelf-life in terms of the number of days required for the fruit to mature. As mangoes mature, the skin-colour changes from green to red or yellow, thus the percentage of green area is a measure of its immaturity. The paper proposes usage of image arithmetic and thresholding techniques to identify the immature area and determine the shelf life in the range of 0 to 10 days.
Object recognition and labelling are the most challenging and fundamental tasks in computer vision. YuShe Cao et.al. proposed use of RCNN coupled with Box-fusion algorithm to detect objects in high resolution remote sensing images. The proposed method is immune to complex changes in environment thereby proving far more effective than BOVW. RCNN has better precision and higher recall rate. It cannot however be used in real-time environment due to large number of operations required.
S. Ren et.al. proposed a fully convolutional network connected to the end of a Fast RCNN, to predict bounding boxes for objects and calculate confidence scores for each identified object. With VGG-16 model, the Faster RCNN is 100 times faster as compared to RCNN.
Ross Girshik et.al. proposed a scalable CNN which gives 50% percent improvement over previous best results on PASCAL VOC 2012 data set. The importance of usage of correct training data for an RCNN is outlined. Gong Cheng et.al. used AlexNet CNN to train a network which is immune to orientation of object with respect to the image axes. The average recall was 98.6% with an mAP of 72.63%. RICNN however has low accuracy for object classification due to pooling of extracted features. Hailiang Li et.al. proposed a Same object Retrieval RCNN to detect same object in various scenes given few training samples. It used similarity score defined by cosine difference between detected and test image. SOR-RCNN has higher accuracy as compared to Fine tuned Faster-RCNN. SOR-RCNN also works better for low resolution images.
Ming-Ming Cheng et.al. explored ways to identify the foreground object. The paper proposed usage of Euclidean distance between centroids to evaluate clusters and create a background mask. The method requires unambiguous salient objects and fails when it encounters multiple objects in same image.
Estimation of shelf-life is of prominence in the food industry. Sudhir Rao Rupanagudi et.al used YCbCr colour space to find the maturity of tomatoes by evaluating the Red chroma, establishing a relationship between 'redness' and immaturity of the fruit. The method is susceptible to changes in illumination and therefore requires controlled environment for accuracy. Oscar Grillo et.al used outer colour of bakery products to evaluate their shelf-life. The relationship between moisture content and discolouration of muffins was explored. The paper establishes a non-linear relationship between surface discolouration and shelf-life.
Salim Arslan et.al. used the high contrast between blue channel of background pixels and the WBCs to identify a global threshold. Distance transform is exploited to establish coarse foreground mark for watershed segmentation.
Detection of injuries on food produce reflects possible internal damage or external infection. Jiangbo Lia et.al proposed use of multispectral imaging to identify injuries on bicoloured fruits. Imaging in Near IR spectrum renders the bicolour feature of fruits irrelevant thus increasing accuracy of simple thresholding to identify injured area. Dong-Bo Sheng et.al  proposed usage of adaptive thresholding to identify injured area on a fish. The paper advocates use of the HSV colour space to counter the effects of illumination on traditional RGB channels.
The paper proposes a solution which has three modules, namely Object Extraction, Shelf-life estimation and Grade classification. Figure 1 shows the architecture of the system.
Object extraction module is used to find and isolate mangoes from the image. The other two modules are dependent on this module.
Shelf-life estimation and Grade classification modules function independent of each other.
Kaiming He et.al. proposed a 152 layer deep residual network to create an accurate classifier for the ImageNet dataset. The paper also cites that a deeper network gives higher accuracy, but also results in longer computational time. The ImageNet has more than 100,000 synonym sets and 100 images to illustrate each set.
The training dataset contains 500 images of Alphonso mangoes with 1000 distinct mangoes on varying backgrounds. There is only one synonym, therefore usage too deep a network is not warranted.
The data is subjected to augmentation using Hue channel shift to replicate the effect of different stages of ripening, using Value channel to replicate effect of illumination and by blurring to replicate low-resolution images.
The paper proposes use of the less deeper ResNet50 to create a network that can identify each mango separately and create a bounding box around the object.
As a mango ripens, its colour changes from green to yellow or red. Thus simple comparison of shades of same colour as advocated in  is not useful. The primary problem is colour based segmentation between yellow and green. Segmentation using single channel similar to  is not appropriate as yellow and green overlap in the G channel.
The colour space La*b* seems a good choice to perform segmentation. A* channel denotes transition from green to magenta, thus the yellow colour lies just short of midpoint making it difficult to perform segmentation as can be seen in Figure2.
The second choice is Hue channel of HSV colour space, but the shade changes are influenced by both Hue and Saturation channel. Also, some shades of Red have H value higher than Green due to the circular nature, rendering simple thresholding ineffective. There is also no way to distinguish between grayscale and red shades in the hue channel as seen from Figure3. Thus a single colour space is unable to deal with this problem.
The proposed solution overcomes these problems by generating a mask for the shades of red, blue, magenta, cyan and gray before application of simple thresholding on Hue channel to highlight the exact green area. A mask is generated using equation (1)
Mask=G-R2- B2 (1)
A hue channel mask is generated using equation (2)
HMask=1 if H>27
=0 otherwise (2)
Final green region is a bitwise AND of Mask and HMask which will show the green area as varying grayscale values and everything else as black. A simple thesholding operation now highlights the immature area of the fruit.
Immature area percentage is given as
ImPer=Highlighted AreaTotal fruit Area (3)
Dark coloured patches on the mango's surface may indicate injuries sustained during harvest, diseases and infection or degradation. Thus it is important to identify these areas so that a grade may be decided depending on the affected area and for further use in disease detection.
As proposed in , adaptive thresholding on S channel in HSV colour space should highlight the areas that have lower purity of colour indicating areas of injury. the method however fails under the situation of varying illumination, due to low purity of grayscale shades as mentioned before. Figure 4 is a good example this phenomenon.
A mango being a bi-coloured fruit,  would recommend use of NIR spectrum imaging so that the injured area will be easy to localize. Multispectral imaging although requires specialized equipment and NIR imaging is susceptible to visible light noise. Thus multispectral imaging is not effective in a general environment.
Dark areas on an image will also represent valleys in a 3D graph of intensity channel. Thus it may be argued that Finding Regional minima will lead to identification of pixels that lie inside the blemished region. This method although identifies too many seeding points due to non-uniform illumination, as can be seen from Figure 5.
The proposed method takes into account these problems. As mentioned before, blemishes will be valleys in a 3D graph of intensity channel. Thus the first differential of V channel will have large magnitude at the boundaries of these blemishes. Considering this, the second differential in both directions will have high magnitude negative values just inside the blemish boundaries as can be seen from Figure 6.
Subjecting these boundaries to a morphological closing operation removes those that are very thin, thus negating the false positives generated by shadows or thin scratches.
A bitwise NOT operation highlights these boundaries, and a threshold is calculated as a mean of intensity values at these boundaries.
t= ∑v ∈Vv*boundaryV (4)
Threshold t is now used to highlight blemishes using binary thresholding from 0 to t. Final rate of injury is calculated as
As advocated in , the relation between physical appearance and shelf-life is non-linear. Thus the proposed solution takes this into account for estimating the time-to-maturity of the fruit. With a test set of over 400 mangoes whose Immature area percentage and time-to-maturity are known, lower and upper bounds in terms percentage are established. Figure 7 shows this distribution.
Similarly, the test set is also used to establish limits for classification into grades. the 4 grades are from Grade 1 to Grade 4, with grade1 being the best quality and Grade4 being deemed unfit for consumption. Figure 8 shows the limits for each grade.
Thus a relationship is established between calculated immature area and shelf-life, an also between injured area and grade
input: R: Red channel values, G: Green channel values, B: Blue channel values, H: Hue channel values for extracted Region of Interest(RoI) output: ImPer: Percentage of immature area Mask = G - (R/2) - (B/2) HMask = 255 if H 27 = 0 otherwise gProp = Mask | HMask gProp = 255 if gProp > 5 = 0 otherwise imArea = |gProp = 255| imPer = |imArea| / |G|
input: V: Value channel from HSV colour space of the extracted RoI output: InjPer: Percentage of immature area yy=?^2 v/?y^2 xx=?^2 v/?x^2 ymask = OtsuThreholding(yy) xmask = OtsuThreholding(xx) mask = ymask ^ xmask mask = morphological_closing(mask) mask = ~(mask) mask = 1 if mask = 255 = 0 otherwise t = ?_(v ?V)??(v*mask)? t = t / |V| blemish = ~(binary_threshold(V, t)) injArea = | blemish=255 | injPer = injArea / |V|
The paper proposes a solution by which the time-to maturity of a mango may be found along with its grade depending on its surface blemishes alone. Other parameters like gaseous anomalies and specific gravity of the fruit may be added so as to get a better approximation of the grade.
The proposed immature area algorithm can be generalised to work with other fruits that lose green colour as they ripen.
The algorithm proposed to detect injuries can be effectively used to detect such sort of blemishes on any product with the only restriction that the product itself should not be dark coloured.
The final estimation of shelf-life and classification into grades are performed based on normal distribution of the test data.
Thus a complete solution,, from extracting mangoes from an image containing more than one objects to independent analysis of each fruit is proposed.
We thank Mahesh Kulkarni, Assistant Professor, Dept. of Horticulture, College of Agricultre, DBSKKV university, Dapoli, for providing valuable insight and expertise that greatly assisted the research.
- Region-based convolutional neural networks for object detection in very high resolution remote sensing images Cao YuShe, Niu Xin, Dou Yong. 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).2016. CrossRef Google Scholar
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation Girshick Ross, Donahue Jeff, Darrell Trevor, Malik Jitendra. 2014 IEEE Conference on Computer Vision and Pattern Recognition.2014. CrossRef Google Scholar
- Object detection in VHR optical remote sensing images via learning rotation-invariant HOG feature Cheng Gong, Zhou Peicheng, Yao Xiwen, Yao Chao, Zhang Yanbang, Han Junwei. 2016 4th International Workshop on Earth Observation and Remote Sensing Applications (EORSA).2016. CrossRef Google Scholar
- An Improved Faster R-CNN for Same Object Retrieval Li Hailiang, Huang Yongqian, Zhang Zhijun. IEEE Access.2017;:13665-13676. CrossRef Google Scholar
- A cost effective tomato maturity grading system using image processing for farmers Rupanagudi SudhirRao, Ranjani BS, Nagaraj Prathik, Bhat VarshaG. 2014 International Conference on Contemporary Computing and Informatics (IC3I).2014. CrossRef Google Scholar
- Use of image analysis to evaluate the shelf life of bakery products Grillo Oscar, Rizzo Valeria, Saccone Rossella, Fallico Biagio, Mazzaglia Agata, Venora Gianfranco, Muratore Giuseppe. Food Research International.2014-aug;:514-522. CrossRef Google Scholar
- A color and shape based algorithm for segmentation of white blood cells in peripheral blood and bone marrow images Arslan Salim, Ozyurek Emel, Gunduz-Demir Cigdem. Cytometry Part A.2014-mar;:480-490. CrossRef Google Scholar
- Multispectral detection of skin defects of bi-colored peaches based on vis�NIR hyperspectral imaging Li Jiangbo, Chen Liping, Huang Wenqian, Wang Qingyan, Zhang Baohua, Tian Xi, Fan Shuxiang, Li Bin. Postharvest Biology and Technology.2016-feb;:121-133. CrossRef Google Scholar
- ShengDong-Bo KimSang-Bong NguyenTrong-Hai KimDae-Hwan GaoTian-Shui KimHak-Kyeong �????��?� ??��??????�??�???�?.;:32-37. Google Scholar
- Global Contrast Based Salient Region Detection Cheng Ming-Ming, Mitra NiloyJ, Huang Xiaolei, Torr PhilipHS, Hu Shi-Min. IEEE Transactions on Pattern Analysis and Machine Intelligence.2015-mar;:569-582. CrossRef Google Scholar
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Ren Shaoqing, He Kaiming, Girshick Ross, Sun Jian. IEEE Transactions on Pattern Analysis and Machine Intelligence.2017-jun;:1137-1149. CrossRef Google Scholar
- Deep Residual Learning for Image Recognition He Kaiming, Zhang Xiangyu, Ren Shaoqing, Sun Jian. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016. CrossRef Google Scholar