Teaching the House to Announce the Mail Truck: A Zero-Shot CLIP Classifier on the Homelab
I wanted my house to say, out loud, “the mail truck is out front” — the moment the USPS truck pulls up, spoken through the same Home Assistant TTS pipeline that runs the rest of the place. Not “a vehicle was detected.” Specifically the mail truck.
The gap between those two sentences is the whole project. My NVR (Frigate) is very good at “there’s a truck in the driveway zone” and completely incapable of “that’s a USPS LLV.” Closing that gap meant adding a small second-stage classifier that looks at the snapshot Frigate already captured and decides what kind of truck it is — running on hardware I already own, with no cloud vision API. It ended up as a zero-shot CLIP classifier, and along the way it picked up a second job I never planned: catching the garbage truck too.
The Mail Truck Classifier Cried Wolf: Tuning Zero-Shot CLIP With Real Data
A while back I taught my house to announce the mail truck: Frigate spots a vehicle out front, a small service crops the snapshot and runs zero-shot CLIP against a list of text prompts, and if it decides “USPS truck” or “garbage truck,” Home Assistant says so out loud through Piper. That post ended on an optimistic note — a garbage truck scoring p_trash 0.995, and a line about how a labeled dataset for future tuning would “build itself from real events.”
This is the reckoning with that optimism.
Because the classifier worked, and then it would not shut up. 143 garbage-truck announcements in seven weeks, on a street where the garbage truck comes exactly once a week. The house had become the boy who cried wolf, and my family had — correctly — started ignoring it. This post is about actually fixing that: what 200 real detections showed, why the obvious fix is a trap, and the one idea worth stealing even if you never touch CLIP.