The problem: finding pepper that is ready to pick
Black pepper is closely tied to Kerala, in south-west India. The authors note that Kerala is not the largest producer, and they link that in part to a shortage of skilled labour and to the difficulty of spotting harvestable pepper in time p. 1.
Their aim is to automate the spotting: given a photograph, say whether harvestable pepper is present and where. They name an automated harvesting drone as the intended use p. 1.
Not defined The paper does not give a formal definition of “harvestable”. What it does show is that the first stage looks for the red colour range, and that red and yellow separate well from green in HSV colour space p. 1, 4.
Why it is hard
- The scene is a natural canopy: leaves, trunks and branches share the frame with the berries.
- Unripe pepper has to be told apart from harvestable pepper.
- Real-time use matters, so the authors design the pipeline so that cheap checks run first and expensive ones only when needed p. 1–2.
The paper at a glance
A summary in this explainer’s own words, with the paper page each point comes from. A full claim-to-page list is in the trace file.
| Topic | What the paper reports | Paper page |
|---|---|---|
| Images | A new image set of black pepper, taken with an ordinary digital camera in natural canopies in Kerala. The network was trained on more than 500 of its images. | 1, 2, 7 |
| Method | Pre-processing, then three stages with a go/stop gate after each: red-colour segmentation in HSV; morphological clean-up and circle counting; a ResNet-50 classifier. | 2 |
| Classifier choice | ResNet-50 was picked after comparing ResNet-101, GoogLeNet, AlexNet and an SVM-based classifier on accuracy and run time. | 6 |
| Test | 115 test images, 65 with harvestable pepper and 50 without. The complete system got every one right. | 6, 7 |
| Headline | “Global accuracy” of 100%, the accuracy of the whole three-stage system, not of the classifier alone. | 5, 7 |
| Platform | MATLAB R2018b on one computer with an Intel i5 processor, 16 GB of RAM and an NVIDIA GTX 1080 GPU. | 6 |
| Next steps | Video input, automatic cutting-point finding, ground or aerial vehicles with a cutter, a farmer-facing mobile app. | 7 |
The three stages, one gate at a time
Before the stages, each photograph is cleaned up. Then it passes through three stages. After each stage the system checks a condition and either carries on or stops with the answer “no harvestable pepper”. Select a step to read what the paper says about it.
Try the gates
Illustrative, not the paper’s implementation
This explorer applies the gate logic the paper describes to values you choose. It never looks at an image. Steps that would be skipped are struck through above. The paper’s prose and its pseudo-code disagree about the stage-2 gate, so you can try both readings p. 1, 6.
Interactive demo: colour thresholding and clean-up
Illustrative, not the paper’s implementation
Stages 1 and 2 are the part of the paper that can be shown without its data. The demo below runs the same kinds of operations the paper names, on a cartoon vine scene drawn by code: a colour threshold in HSV, morphological clean-up, a minimum object size, and a circle around each surviving object. It uses none of the paper’s images, its colour ranges, or its numbers. Move the sliders and the masks update live.
What to try
- Widen the hue range or lower the brightness: the brown branch and the dry leaf join the mask. Colour alone cannot always tell them from fruit, which is why the paper adds a shape stage and a classifier.
- Switch clean-up off: isolated speckles and small holes appear. Opening removes specks; closing fills holes.
- Raise the smallest object size to discard what is left of the noise, as the paper does with a fixed size cut-off.
How this differs from the paper
- The paper pre-processes with guided filtering, local Laplacian filtering and histogram equalisation first; the demo does not.
- The paper’s red range and its 60-pixel size cut-off apply to its own 300×400 images p. 5, 6; the demo’s defaults are tuned to a small cartoon.
- In the paper, ResNet-50 is then given the input image. No classifier runs here.
ResNet-50 as the last gate
How the paper uses it
- It is stage 3, the last gate. If the earlier gates let an image through, ResNet-50 decides “pepper” or “not pepper” for the input image p. 1, 5.
- It was trained on more than 500 images from the new image set p. 1, 5.
- The paper’s pseudo-code loads a pre-trained network, takes features from one of its convolutional layers, trains a classifier on them, and then predicts p. 6. The prose speaks of training through feature extraction and deep learning.
- It was chosen by comparison with ResNet-101, GoogLeNet, AlexNet and an SVM-based classifier. ResNet-50 had the best balance of accuracy and run time; ResNet-101 was slightly less accurate p. 6.
- Used alone, the classifier’s accuracy is reported as 96.35% in the paper’s table (the prose says 98.44%, see below). Together with the first two gates, the system’s accuracy is reported as 100% p. 5.
Why a verifier at the end
Colour and size checks are cheap but fooled by look-alikes such as a reddish dry leaf. A deep network is better at telling look-alikes apart, but costs more per image. Putting it last means it only runs on images that already look promising, which is the saving in time and computing the paper describes p. 2.
Unclear Whether the network’s own weights were updated (fine-tuning) or only used to produce features, the input size it was given, and the training settings are not stated.
Inside a standard ResNet-50
Illustrative, not the paper’s implementation
This panel shows the standard ResNet-50 from He, Zhang, Ren and Sun, “Deep Residual Learning for Image Recognition,” CVPR 2016 (10.1109/CVPR.2016.90). The sizes are the standard 224×224 ImageNet configuration, not a statement about the paper’s setup. Select a block to read about it.
How “global accuracy” is defined
The unit being scored is an image, and the thing being scored is the whole three-stage system, not any one stage. For each test image the system answers “harvestable pepper present” or “not present”, and that answer is compared with the truth p. 6, 7.
| Truth \ system says | Present | Not present |
|---|---|---|
| Pepper really there | True positive (TP) | False negative (FN) |
| No pepper | False positive (FP) | True negative (TN) |
Accuracy is the share of images the system got right: (TP + TN) divided by all images. The paper also reports recall (TP over TP + FN), precision (TP over TP + FP) and F1, the harmonic mean of the two p. 5, 7. A global accuracy of 100% therefore means zero false positives and zero false negatives on the test set.
Try the arithmetic
Illustrative counts, not the paper’s results
Two accuracy numbers, and one inconsistency
The paper reports accuracy at two levels. The classifier alone (stage 3 by itself) and the whole system (“global”). The paper presents the jump from the classifier’s figure to 100% as the result of coupling the classifier with the full three-stage architecture p. 5.
Ambiguous For the classifier alone, the text gives 98.44% while the paper’s table gives 96.35% p. 5. Averaging the two diagonal entries of the paper’s confusion-matrix table reproduces the table figure, so that one looks the more consistent. The printed accuracy formula is also missing brackets, and the table’s error cells are labelled in an unusual way. None of this affects the 115-of-115 system result, but a reader should know.
The reported result
Reported in the paper p. 5–7
100%
global accuracy
Reported in the paper (not reproduced here). Definition: the share of 115 test images, 65 with harvestable pepper and 50 without, that the complete three-stage system classified correctly. The paper reports 115 of 115, with recall and precision both at 100%. This page runs no recognition model and measures no accuracy of its own.
Scan time. The paper describes the scan time as reasonable for real-time use p. 1, 6 but gives no figure for the full system.
Reading the number
- It is a statement about 115 images. The paper does not say whether the test images were kept apart from the training images p. 5–7.
- Derived here Zero errors in 115 trials is consistent with a true error rate of up to roughly 2.6% (one-sided 95% bound, about 3 divided by 115). That is this explainer’s arithmetic on the reported count, not a result from the paper.
Limitations
What the paper itself says
The paper has no limitations section. Its closest statements are that a high-performance GPU and processor are desirable p. 2, 3, and the future-work list (below), which implies the system currently works on still images, finds no cutting point, and has not been shown running on a vehicle.
Points a careful reader would raise (this explainer’s notes, not the paper’s)
- Small test set. 115 images is a modest sample, and a perfect score on it carries real uncertainty (see the bound above).
- Train and test overlap not stated. If test images came from the same scenes as training images, the score would be optimistic.
- Capture conditions. Natural canopies in Kerala are stated, but lighting, time of day, camera settings and distance are not. One crop, one region.
- Internal inconsistencies. Two different classifier-only accuracies, and a stage-2 gate that differs between prose and pseudo-code.
- Training detail is thin. Fine-tuning versus feature extraction, the input size, and the training settings are not given.
- Speed is asserted, not quantified. There is a per-classifier timing chart but no end-to-end scan time.
- Comparison with earlier systems is made in words, not on a shared image set p. 6.
- Nothing to reproduce. The dataset and code are not released, so the result cannot be independently re-run.
- 2019 methods. The work predates today’s end-to-end detectors and segmentation models.
- This explainer. Its demos use cartoon images and generic techniques. They illustrate ideas; they do not test, confirm or approximate the paper’s results.
Future scope
From the paper p. 7
Live video
Work on video streams rather than still images, for real-time identification and picking.
Cutting point
Automate finding where to cut, so that harvesting itself can be automated.
Vehicles
Mount the system on unmanned ground or aerial vehicles that record high-quality video and carry a cutting mechanism.
Mobile app
A phone app that farmers can use directly for yield estimation and harvesting.
Suggestions from this explainer (not from the paper)
- Modern one-stage or transformer detectors and instance segmentation can locate spikes directly, with boxes or masks and confidence scores.
- Quantised or distilled models on a phone or embedded board would remove the need for a desktop GPU.
- Evaluating across farms, seasons and lighting, with a published split and per-class precision and recall, would make results comparable.
Sources and gaps
Sources used
- The published paper itself (a copy supplied by a co-author). It is not included in this repository.
- Crossref record for DOI 10.1109/ICACC48162.2019.8986220 (author form, venue, pages, year). The conference name and city are not printed in the supplied copy.
- He et al., CVPR 2016, for the standard ResNet-50 architecture.
Still not answered by the paper
- A formal definition of “harvestable”.
- Whether the 500+ images are the whole set or only the training part, how it was split, and whether test images overlap training images.
- Lighting, camera model, resolution and other capture details; whether the set is available.
- Whether ResNet-50 was fine-tuned or used only for features, its input size, and training settings.
- Which stage-2 rule was actually run (zero circles, or one or fewer), and the exact red range.
- An end-to-end scan time.
- Which of 98.44% and 96.35% is the classifier-only accuracy.