The Mail Truck Classifier Cried Wolf: Tuning Zero-Shot CLIP With Real Data

A while back I taught my house to announce the mail truck: Frigate spots a vehicle out front, a small service crops the snapshot and runs zero-shot CLIP against a list of text prompts, and if it decides “USPS truck” or “garbage truck,” Home Assistant says so out loud through Piper. That post ended on an optimistic note — a garbage truck scoring p_trash 0.995, and a line about how a labeled dataset for future tuning would “build itself from real events.”

This is the reckoning with that optimism.

Because the classifier worked, and then it would not shut up. 143 garbage-truck announcements in seven weeks, on a street where the garbage truck comes exactly once a week. The house had become the boy who cried wolf, and my family had — correctly — started ignoring it. This post is about actually fixing that: what 200 real detections showed, why the obvious fix is a trap, and the one idea worth stealing even if you never touch CLIP.

The obvious fix is wrong #

The instinct is immediate and universal: “just only announce when it’s really sure — say, above 95% confidence.” I believed it too. It is wrong, and being able to say exactly why it’s wrong is the most useful thing in this post.

But I didn’t want to argue from intuition. I wanted to argue from the images. So I pulled the last 200 firing detections off the classifier and went to label them by hand.

The first finding arrived before I labeled anything: 114 of the 200 snapshots were already gone. Frigate’s snapshot retention had aged them out. Only 86 still existed. (Hold that thought — it turns out to be the most important operational lesson here, and it comes back at the end.) I hand-labeled all 86 by eye:

A classifier firing at 37–48% precision is not “a little noisy.” It’s a coin flip wearing a lab coat. But the false positives weren’t random noise — they had structure, and the structure is the whole story.

Two failure modes, and only one is fixable with a threshold #

Looking at the actual wrong images, the false positives sorted cleanly into two piles.

Mode 1 — look-alike vehicles. CLIP was confidently naming things that genuinely resemble the target. The single biggest culprit was the green-and-white Xfinity bucket truck — 6 trash false positives on its own. Green livery, a boom arm, a tall body: to CLIP that reads as “municipal garbage truck.” Also in the pile: a U-Haul box truck at p_mail 0.88, white cargo vans, white work pickups, a cone trailer, and — my favorite — a flatbed hauling a garbage truck body.

Mode 2 — background bleed. The classifier pads the detection box by 15% a side (CROP_PAD = 0.15) before cropping. That padding regularly drags a second vehicle into frame. Several detections whose tracked object was an ordinary parked car scored p_trash 1.00 — because a garbage truck was genuinely passing through the top of the crop. CLIP wasn’t wrong about the pixels; it was describing the scene instead of the detected object. The canonical example: a silver Toyota Camry sitting in a driveway, scored 1.00 trash, because a real garbage truck is visible above it.

Mode 2 is the reason the “just raise the threshold” fix is doomed. Those false positives are already at 1.00. There is no confidence gate above 1.00. You cannot threshold your way out of a classifier that is looking at the wrong object.

Why “only fire above 95%” measured out as an off switch #

Here’s the part that deserves the most space, because it’s a category error a lot of people make with these models.

A CLIP softmax score is not a calibrated probability. It’s the softmax over whatever prompt list you happened to write. Add a prompt, remove a prompt, and every score shifts, because the denominator changed. “95% confident” doesn’t mean “95% likely to be right” — it means “this prompt won the argmax against the other prompts in this particular list by this much.” Treating that number as a probability is treating a beauty-contest margin as a lie-detector result.

So I measured what a hard 0.95 gate actually does, on the labeled set:

Read that middle row again. A 0.95 gate takes mail announcements from 9-of-11 deliveries down to 2. And that’s the charitable number — measured after I’d already improved the prompt list. On the original v1 prompts, a 0.95 gate fired on zero of the 29 mail detections in the review set. You haven’t made the classifier precise. You’ve turned it off and kept the icon on the dashboard so it still feels like a feature.

There’s a second half to this lesson, and it’s just as important: measure recall in units your family actually experiences — visits, not events. One mail delivery generates several Frigate events, and the announcement already has a 30-minute cooldown. So only one of those events has to clear the bar for the delivery to get announced. Per-event recall looks catastrophic; per-visit recall is fine. I grouped events into visits by a >1-hour gap, and that regrouping is the only reason the numbers mean anything. Measuring per-event, I’d have concluded a config that works in practice was broken, and thrown it away.

The fix that actually shipped #

Not a threshold. Three sentences of English. I added negative prompts describing the specific confusers from the review, and — modestly — raised the thresholds and widened the margin:

    # negatives added from a hand-labeled review of 86 firing detections
    "a green and white Xfinity Comcast cable utility truck with a boom bucket lift on the back",
    "a white U-Haul rental moving box truck with graphics on the side",
    "a flatbed tow truck or car hauler carrying another vehicle",

- MAIL_THRESHOLD=0.45   TRASH_THRESHOLD=0.45   MARGIN=0.10
+ MAIL_THRESHOLD=0.55   TRASH_THRESHOLD=0.55   MARGIN=0.25

On the same labeled set:

Adding three sentences did what raising the threshold could not — killed the false positives without gutting recall. That’s the deep point about zero-shot classification: the negative prompts are the decision boundary. If there’s no prompt for the thing that keeps fooling the model, its score has nowhere to go but onto your positive class. You’re not tuning a dial; you’re drawing the boundary in words.

Within a minute of the rebuild, it caught a real APEX Waste Solutions truck at p_trash 0.94. The wolf was real that time.

The trap: the most obvious negative prompt is the most destructive #

Several of the mail false positives were plain white cargo vans. So the obvious next negative prompt writes itself:

"a plain white Chevrolet Express or Ford Transit cargo van with no windows"   # DO NOT ADD

Do not add it. A USPS LLV is a small white windowless van — so this prompt describes the mail truck better than my mail prompt does, and it wins the argmax against the real thing. I ablated it: on its own it dropped mail from 9/11 visits to 6/11, and it beat the correct answer on 7 different real USPS LLVs.

The general rule, and it’s a sharp one: a negative prompt that describes your positive class in more generic words will eat it. Write negatives that describe the confusers, not the category. The Xfinity/U-Haul/flatbed prompts are safe precisely because nothing about them also describes an LLV. You only find this by ablating one prompt at a time against labeled data — guessing would have shipped the van prompt and quietly broken the mail alert.

The technique worth stealing: dump logits once, search prompts for free #

Here’s the method that made all of the above cheap enough to actually do.

The softmax over a subset of prompts can be recomputed from the raw logits of the full set — you just re-normalize over the columns you kept. So I scored every image once against the union of every candidate prompt, saved the raw logits, and then evaluated any prompt subset offline by re-softmaxing just those columns. Plain Python, no GPU, no model reload:

def softmax(v):
    m = max(v); e = [math.exp(x - m) for x in v]; s = sum(e)
    return [x / s for x in e]

# `sub` = the prompt indices in this candidate set; 0 = mail, 1 = trash
p = softmax([row[i] for i in sub])

That turned an exhaustive search over thousands of (prompt-subset × threshold × margin) combinations into something instant, off a single model run over 86 images. Ablating 15 candidate negatives individually would otherwise have meant 15 full re-scoring passes.

Two things I tested that did not work — because the negative results are the honest part:

• Tightening CROP_PAD to 0 to fight background bleed. It cost mail recall (9/11 → 7/11 visits) and removed no false positives. Left at 0.15.
• Bigger, sharper positive prompts. A more elaborate USPS description (“right-hand-drive LLV with a blue eagle logo and a red and blue stripe”) was more precise but cut recall further. The short “stubby white van” phrasing survives.

And two gotchas for anyone replaying snapshots:

• Replay is not live scoring. The live service classifies whatever frame is current during the event; the stored snapshot is the final frame. Replayed scores are valid for comparing prompt sets against each other, but they won’t reproduce a specific logged score. Several p_trash 1.00 background-bleed cases don’t reproduce on replay at all — the truck had driven on by the time the final snapshot was written.
• This Frigate build ignores ?bbox=0. Every crop still carries the drawn orange box and burnt-in car: 96% ... label text. CLIP is known to be sensitive to text in images, so this is a real, unresolved question for a future round — I’m flagging it, not claiming I fixed it.

The real headline: your NVR’s retention is not your dataset #

Remember that 114 of the last 200 detections were already gone before I could label them? That’s the finding with the longest shelf life. Frigate’s snapshot retention is far shorter than a useful tuning window, so the parent post’s cheerful line about a dataset that “builds itself from real events” was simply wrong. Nothing was building itself, and more than half of it had already evaporated.

So the classifier now keeps its own copy — bucketed by how useful each sample is for tuning:

review/YYYY-MM-DD/fired/   every announcement           -> where false positives live
review/YYYY-MM-DD/near/    didn't fire, but a positive
                           class scored >= 0.25          -> where the MISSED trucks live
review/YYYY-MM-DD/other/   1-in-20 of the remainder      -> negative coverage
review/YYYY-MM-DD/index.jsonl

Why bucket instead of keeping everything? About 1,240 events/day pass the zone+label filter. Fourteen days of all of it is ~17,000 images — only ~456 MB, which is nothing on a disk with 119 GB free. But 17,000 images is not a set a human will ever review. Bucketing keeps 100% of what’s diagnostic and thins the boring remainder to ~100–400 images/day. The storage was never the constraint; attention was. (One image per event per bucket, too — the classifier re-runs every ~5 s over an event’s life, and 30 near-identical crops of the same truck are worthless.)

The detail worth copying: index.jsonl stores the full probability vector for every sample, plus a prompts_version hash of the prompt list.

{"ts": 1787065483.471, "event_id": "1787065483.180278-2quxox", "bucket": "fired",
 "fired": "garbage_truck", "probs": [0.0, 0.9419, 0.0111, ...],
 "prompts_version": "8a31452f", "file": "2026-08-18/fired/150443_m0.00_t0.94_..."}

Because the raw probs are stored, a threshold or margin change can be re-scored from the index alone — no CLIP run, no images, just re-apply the rule to the vectors. Only a prompt change needs the model re-run over the crops. And the prompts_version fingerprint stops you from silently pooling days that were captured under different prompt lists — which would quietly corrupt every comparison.

Retention is a nightly cron with a lock:

15 3 * * * /usr/bin/flock -n /tmp/truck-prune.lock \
  /home/vlouvet/docker/truck-classifier/prune-review.sh >> .../prune-review.log 2>&1

The window comes from REVIEW_KEEP_DAYS=14, with a 4 GB size backstop that drops the oldest days if a busy week outruns the window. Whole day-directories are the unit of retention, so a surviving index.jsonl never points at crops that were already deleted.

One nice little sysadmin beat to close on: the container writes those files as root, and this host has no passwordless sudo — so a normal user crontab can’t delete them. Rather than escalate privileges, the prune script does its deletions through a throwaway container that bind-mounts the same directory. Reading (find, du) works fine as my user; only the unlink needed the uid.

in_container() { docker run --rm -v "$REVIEW:/review" "$IMAGE" bash -c "$1"; }

Honest caveats #

I’d rather state these plainly than let the tidy numbers oversell it:

• 86 images is a small set, hand-labeled by eye by one person. The direction of every result here is solid; the exact percentages are not precise to the point.
• The set is biased by construction. Every image in it fired under the old config. So it measures precision well and confirms the new config keeps what it had — but it cannot see a real mail truck the old config already missed. True recall is unknown, and could be worse than 9/11. That blind spot is exactly why the near/ bucket exists, and the honest reason there will need to be a v3.
• A couple of labels were judgment calls — a garbage truck being hauled past on a flatbed isn’t a collection pass, but it’s visually a garbage truck.
• “0 false positives” means zero on this 86-image set, not zero forever.

Takeaways #

• A confidence threshold can’t fix a classifier that’s looking at the wrong object. Some of the worst false positives scored 1.00. Diagnose which failure mode you have before you tune anything.
• Zero-shot softmax scores are not probabilities. “Only fire above 95%” sounds like rigor and measured out as an off switch — 2 of 11 mail deliveries.
• In a zero-shot classifier, the negative prompts are the decision boundary. Three sentences of English beat every threshold change I tried.
• But a negative that describes your positive class in more generic words will eat it. Describe the confusers, not the category — and ablate one prompt at a time against labels.
• Measure recall in units the user actually experiences — visits, not events.
• Dump raw logits once and re-softmax subsets to make prompt search essentially free.
• If you plan to tune from production events, keep your own copy. Your NVR’s retention is not your dataset — bucket by usefulness, because storage is cheap and attention isn’t.

The house announces the mail again, and — for now — only the mail. Whether that survives contact with a delivery van I haven’t seen yet is a question for v3, which, this time, will have the data to answer it. (If you’re new here, the Frigate alerts-vs-detections trap is the other post where a confident-looking dashboard turned out to be lying to me.)

• Frigate
• Home-Assistant
• Clip
• Computer-Vision
• Machine-Learning
• Mqtt
• Docker
• Self-Hosted
• Linux
• Homelab
• Cron