Seeing Isn't Knowing

One Photo, One Basket, A Hundred Decisions

Picture the checkout line at PRoduce, a Puerto Rico farm-to-table grocery. A shopper sets down a basket. Inside: a knob of ginger, a lone lime, a banana, a hot pepper, a root vegetable — and tangled among them, a bag of organic sugar, a jar of peppers, mustard, a pouch of guava sticks, a bag of cashews. The cashier picks up each item, hunts for a code or a label, keys it in, repeats. Loose produce has no barcode. Packaged goods do, but only if you can find it. It is slow, and the line only grows.

The dream is almost absurdly simple: snap one photo, ring up everything. Point a camera at the basket, and the system names every item, prices it, and hands the cashier a finished sale. That is the project PRoduce set out to build — its point-of-sale basket scanner.

A barcode scanner reads one labeled item at a time. A basket is a pile of dissimilar things in a single frame — that is a different, far harder problem.

Why is it hard? Because a real basket is chaos. The items don't match each other. Some have packaging and text; some are just a curve of yellow that could be a banana or a plantain. They overlap, cast shadows, and hide behind one another. A human cashier untangles all of this without thinking. Teaching a machine to do the same — reliably, in the time it takes to bag groceries — turned out to be a journey full of wrong turns and uncomfortable surprises.

And those wrong turns are the real story. What follows is not a victory lap but a sequence of business lessons — about giving precise instructions, about knowing the difference between finding something and understanding it, about why a perfect test score can be the most dangerous result of all, and about why the best speed fix didn't touch the clock. Each one cost PRoduce something to learn. Each one transfers to any leader betting on AI.

AI Does Exactly What You Ask

Once the team taught the system to find each item first, then identify it one at a time, the basket scanner got dramatically better at the hard part: seeing that there were five separate things on the counter at all. But the very first version of that "find each item" step had a flaw that turned into the most useful lesson of the whole project.

The system had been quietly told to look for one thing: produce. That instruction made perfect sense to the people who wrote it — PRoduce is a produce company. But a real basket isn't all loose fruit and vegetables. So when a cashier photographed a basket of packaged groceries — a bag of organic sugar, jarred peppers, mustard, a pouch of guava sticks, a bag of cashews — the system did exactly what it was told. It looked for produce, found almost none, and returned 1 of 5 items.

The model wasn't confused, and it wasn't underpowered. It was obedient. It had been handed a narrow goal and it pursued exactly that goal, ignoring four perfectly visible items because nobody had told it those counted. The moment the team broadened the instruction to "find every item a shopper would buy" — not just produce — the same photo, the same system, the same everything returned all 5 of 5.

The AI wasn't dumb. The instruction was too narrow. A single word — "produce" — quietly cost four of five items.

On a produce basket, the broadened instruction held up just as well — every item found, each one cleanly boxed off from its neighbors.

Here's where it gets interesting for any leader who assumes "tuning the AI" means a slow, expert-driven grind. The team didn't sit and hand-polish wording. Instead, they set several AI agents loose in parallel, each one trying a different way of phrasing the task, each one looking at its own results and grading how well it did — then they kept the winner. The humans never stopped owning the goal or the finish line; they simply let the machine do the tedious search for the best way to ask. It converged almost immediately.

Finding Is Not Identifying

Once PRoduce's basket scanner could reliably find each item in a photo, the team hit a humbling truth: spotting that something is there is a completely different skill from knowing exactly what it is. The system could draw a clean box around five objects in a basket — and then confidently mislabel them.

The mistakes were the kind a hurried human might make, too. A lime guessed as a different citrus. A banana called a plantain. The right bag of sugar identified — but credited to the wrong brand. In a grocery, those aren't rounding errors. Wrong brand means wrong price, wrong inventory, wrong margin.

Judging a product by appearance alone is like recognizing it with the label turned away.

That analogy became the team's mental model for the whole problem. Looks get you surprisingly far and then betray you at the worst moment — precisely on the look-alikes that share a silhouette, a color, a shape. Appearance is necessary, but it is never enough.

The breakthrough was to stop asking the AI to be a single, all-knowing eye and instead give it three different kinds of judgment to weigh against each other:

What it looks like — the visual impression of shape, color, and form. What the packaging says — the words and numbers printed on the label, which a lime obviously lacks but a bag of sugar carries proudly. What the catalog confirms — a lookup against PRoduce's real list of products actually sold in the store, so the answer is always something the business genuinely stocks.

No single signal is trustworthy alone. The win was teaching the AI to reconcile three.

Crucially, the system doesn't treat these as three votes to average. It lets the AI reconcile them — using the printed label to break a visual tie, using the catalog to rule out a brand the store never carries. The picture narrows the field, the text sharpens the guess, and the catalog keeps the answer honest and on-shelf.

A Perfect Score Was a Warning Sign

Late in the project, one configuration of the basket scanner did something that should have felt like a triumph: it scored a flawless, perfect result on PRoduce's two test baskets. Every item, correctly identified. The kind of number you put on a slide.

It was a trap. The system hadn't gotten smarter — it had simply memorized the answers. With only two baskets to be measured against, it had effectively learned the test rather than the skill. Hand it a third basket it had never seen, and the magic would have evaporated. A perfect score on a tiny sample isn't proof you've succeeded; it's a sign you've fooled yourself.

A perfect score on two test baskets didn't mean the system was brilliant. It meant the test was too small to be honest.

So the team deliberately chose the less impressive version — the one that scored a little lower but worked on baskets it had never seen before. That is the version you can actually trust in a busy store.

Then came the more revealing discovery. When the team examined the honest version's remaining "misses," almost none of them were the AI's fault at all. One item couldn't be matched because it simply didn't exist in the product catalog yet — a gap in the company's own data, not a failure of recognition. Another was matched to the wrong record because the label stored in the system was incorrect to begin with; the AI faithfully returned the bad information it was given. A third was a size printed on the packaging that was genuinely unreadable in the photo — no human could have read it either.

In other words: the model was doing its job. The errors lived in the data and the process around it — the catalog, the labels, the photo. These are the unglamorous, fixable problems that masquerade as "the AI got it wrong," and they are where the real work of the last mile happens.

When the Answer Is "Ask the Human"

Some calls are genuinely hard. A lime and a different citrus can look identical under fluorescent light. A banana and a plantain are cousins. The right sugar in the wrong brand bag is a coin flip. PRoduce made a deliberate choice here: rather than have the basket scanner bluff its way to a confident-sounding wrong answer, it tells the cashier exactly how sure it is.

The interface is color-coded, and it reads like a traffic light. Green means the system is confident — the cashier just glances and confirms. Yellow means "I've narrowed it to two or three likely options" — the cashier taps the right one, which takes a second. Red means "I couldn't place this" — the cashier handles it the old way. The AI does the heavy lifting on the easy majority; the human spends their attention only where it actually adds value.

A machine that admits "I'm not sure, here are your best three" is more useful than one that's confidently wrong.

This is the part that's easy to underestimate. A system that's confidently wrong is worse than one that asks, because the cashier stops checking it — and the one time it's wrong on a price, a customer notices. False certainty quietly transfers risk onto the people who trusted the tool. Confidence-aware design does the opposite: it tells the human precisely where to look, so trust is earned item by item instead of demanded all at once.

There's a compounding benefit, too. Every scan is logged — every green confirmed, every yellow corrected, every red rescued by a person. That record of real cashiers making real calls at a real register becomes the raw material for getting better over time. The product doesn't just serve today's transaction; it studies it.

Measure, Don't Assume

By this point the scanner worked. It saw the whole basket, identified the decidable items, and asked the cashier when it wasn't sure. But it felt sluggish. From the moment a cashier snapped the photo to the moment results appeared, there was about an eight-second wait — an eternity at a busy register with a line forming.

The team had three "obvious" ideas to speed it up. All three were obvious. All three were wrong. And the only reason anyone found out was that they measured instead of guessed.

Idea one: combine two steps into one. Surely doing two jobs in a single pass would be faster. It was — and it quietly halved the accuracy. Speed bought with wrong answers is not a bargain.

Idea two: attack the step that "obviously" dragged. The team was sure they knew which part was slow. When they actually timed it, that step was already as fast as it could reasonably be. The bottleneck was somewhere they hadn't suspected.

Idea three: upgrade to the newer, shinier AI model. Newer must mean better. On this specific task, the newer model performed worse. "Latest" is a marketing word, not a measurement.

Intuition about performance was wrong three times in a row. Only the stopwatch was right.

Then came the insight that mattered most — and it had nothing to do with the clock. The biggest win wasn't making the system faster. It was making it feel faster. Instead of staring at a blank screen for eight seconds, the cashier now sees the items appear within about three seconds, with the finer details filling in as they resolve. The wait barely changed. The experience of waiting changed completely.

That is the spirit of the whole project: a stream of confident assumptions, each one humbled by evidence. Which is the real takeaway for any leader betting on AI. Here is the whole story, compressed into eight things you can quote in your next meeting.

The model was rarely the problem. The assumptions were. Replace assumptions with evidence, keep a human at the finish line, and the technology does the rest.