Deep learning based image segmentation applied to mango fruticulture

Recent advances in deep learning enabled machine vision techniques are making various orchard automation activities possible, including fruit count, fruit sizing and branch avoidance during pruning or harvesting tasks. Precise localization of on-tree objects is crucial for such tasks. A review of literature was undertaken to document the state-of-the art in fruit sizing and image segmentation of fruit and branches. With published literature focused to sizing of symmetrical (circular) fruit and vegetables, a research gap was noted for the sizing of non-circular symmetric fruit such as mango using image segmentation techniques, in real time.

As fruit sizing from canopy images requires information on camera to object distance for conversion of pixel size into real world size, an evaluation of depth cameras with a range of operational technologies was conducted. Two Time of Flight based cameras were recommended based on depth accuracy in an outdoor environment, high spatial resolution, a range of 5 m and ease of use. However, the sizing of mango fruit in canopy images using traditional image segmentation methods was challenging due to occlusion from foliage, fruit, and branches. A Mask R-CNN based instance segmentation method was developed, delivering a higher accuracy with less segmentation error than a manual segmentation method. A YOLOv8m based segmentation method was then developed, improving on the previous result, providing a real time speed (>25 fps) and accuracy (RMSE ~5 mm for fruit length) suited to real time application. YOLOv8m and v9 models outperformed the benchmark MaskR-CNN model in terms of their accuracy and inference time, achieving up to a 98.8% mAP50 on fruit predictions and 66.2% on branches in a leafy canopy. Further, the contribution of several factors to error in the sizing result was assessed, including fruit allometry, image distortion, sizing from images taken from a moving platform and accuracy of algorithms used to recognize and exclude partly occluded fruit from sizing estimates.

Further to image segmentation methods for fruit sizing, the use of segmentation was explored for recognition of branches causing partial occlusion of fruit in canopy images. This capability was adopted into a branch avoidance method for the workspace of an automated harvester arm. Branch avoidance was achieved by filtering branch point cloud within the 3D workspace using a rectangular followed iii by trapezoidal prism. The cycle time for this computation was 1.63 milli seconds using CUDA in an edge computing hardware. This timing allows for real time (>25 fps) use, which was largely facilitated by the high-speed inference of the YOLOv8m image segmentation network.

The thesis is structured into seven chapters; comprised of: (a) a general introduction providing industry and social context, and thesis objectives, (b) review of machine vision-based fruit sizing methods, (c) comparison of depth cameras, (d) machine vision-based fruit sizing methods, (e) evaluation of errors related to tools and techniques in fruit sizing, (f) fruit count, sizing and branch avoidance for real time application for robotic harvester application, and (g) conclusions and recommendation. Five of these chapters have been published in peer-reviewed journals.