A Flower Classifier and the Long Tail

The pitch is the oldest one in computer vision: point your phone at a flower, get the species back. I built it as a Flutter app with a Python inference backend, and it worked on the first try, which should have been the warning sign.

The part that was easy

Transfer learning has made this problem almost boring. Take a pretrained backbone, replace the classifier head, fine tune on a few thousand labelled images, and the validation accuracy climbs into the nineties in an afternoon. The Flutter side was equally routine: capture an image, post it, render the top three predictions with confidences.

I had a working demo in a weekend and thought I was done.

The part that was not

Then I pointed it at flowers that were not in the dataset.

The model did not say it did not know. It could not. A softmax over N classes always sums to one, so an image of something the model has never seen still produces a confident looking distribution. My classifier would look at a plant it had no concept of and report 80 percent confidence on whatever class happened to be nearest in feature space. It was never uncertain, only wrong with conviction.

This is the long tail problem, and in a flower app it is the whole problem. The dataset covers the common species. The photos people actually take are of the thing in front of them that they do not recognise, which is exactly the case most likely to fall outside training.

What I did about it

Three changes, in increasing order of how much they helped:

• Temperature scaling on the logits, fitted on a held out split. This calibrates the confidences so a reported 0.8 is closer to being right 80 percent of the time. Cheap, worth doing, does not solve unknown classes.
• A reject option. Instead of always returning the top class, return nothing when the margin between the top two classes is small or when the max softmax falls under a floor. The app says it is not sure. Users tolerate that far better than a wrong species stated confidently.
• Distance in feature space, which mattered most. Rather than trusting the softmax, compare the image embedding to the class centroids from training. An image far from every centroid is out of distribution regardless of what the classifier head says. This caught cases the softmax floor missed entirely.

What I took from it

The lesson was not about flowers. A validation score measures how a model does on data drawn from the same distribution as its training set, and that number quietly assumes the deployment distribution matches. For a classifier in the world, it does not, and the mismatch is not random: users disproportionately bring you the hard cases, because the easy ones they already recognised themselves.

Since then, the first question I ask about any classifier is not what its accuracy is. It is what it does when the answer is not in the label set.

Built with Flutter on the front end and a Python inference service behind it.