Teaching the House to Announce the Mail Truck: A Zero-Shot CLIP Classifier on the Homelab

I wanted my house to say, out loud, “the mail truck is out front” — the moment the USPS truck pulls up, spoken through the same Home Assistant TTS pipeline that runs the rest of the place. Not “a vehicle was detected.” Specifically the mail truck.

The gap between those two sentences is the whole project. My NVR (Frigate) is very good at “there’s a truck in the driveway zone” and completely incapable of “that’s a USPS LLV.” Closing that gap meant adding a small second-stage classifier that looks at the snapshot Frigate already captured and decides what kind of truck it is — running on hardware I already own, with no cloud vision API. It ended up as a zero-shot CLIP classifier, and along the way it picked up a second job I never planned: catching the garbage truck too.

The core problem: Frigate gives you “truck,” not “mail truck” #

Frigate’s stock detectors (CPU/Coral/OpenVINO) use COCO-class models. They know car, truck, person — generic classes. There is no “USPS mail truck.” (There isn’t even a reliable truck if your model is the bundled ssdlite_mobilenet_v2, which is a trap I hit separately — the mail LLV is small and usually classifies as car anyway.)

So every approach is really two stages:

1. Stage 1 (Frigate): “a car/truck entered the front-of-house zone.”
2. Stage 2 (the specificity layer): “…and it is the mail truck.”

Stage 1 already exists. The project is Stage 2, and where to run it.

The architecture decision: don’t touch the busy box #

Frigate runs on fileserv (.21), which is already loaded with containers. I did not want to pile a vision model onto it. The classifier is a separate, self-contained service that:

1. Listens to Frigate’s MQTT event stream (broker lives on my Home Assistant box, .18).
2. On a car/truck event in the front zone, pulls the event snapshot from Frigate’s API (http://192.168.1.21:5099/api/events/<id>/snapshot.jpg).
3. Runs a classifier on the image.
4. Publishes a verdict to home/mailtruck/detected.
5. Home Assistant (.18) reacts → Piper TTS → the speaker announces it.

The only change to the Frigate box is a one-line addition of truck to the tracked objects. Everything else is new and isolated. Clean separation: Frigate keeps doing Frigate, the classifier does exactly one job.

Why zero-shot CLIP #

For “is this specifically a USPS truck,” the tempting path is “collect a few hundred labeled images and fine-tune a binary classifier.” That works, but it’s a lot of upfront labeling for a first cut. CLIP (Contrastive Language–Image Pre-training) skips it: you hand it the image and a list of candidate text labels, and it scores how well the image matches each phrase. USPS LLVs are visually distinctive — white body, right-hand drive, blue eagle — so plain-language prompts separate them well:

labels = [
  "a white USPS United States Postal Service mail delivery truck",
  "a brown UPS delivery truck",
  "a FedEx delivery van",
  "an Amazon delivery van",
  "an ordinary car or SUV",
  "an empty street",
]

Take the argmax, apply a confidence threshold and a margin over the runner-up, and you have a verdict. If zero-shot ever proves too fuzzy, Frigate is conveniently already saving every snapshot — so you can collect a labeled set from real events and fine-tune later. But you don’t need that to ship v1.

I ran CLIP through OpenVINO (ViT-B/32), which keeps inference in the millisecond range on modest hardware.

The hardware saga (or: where does the GPU live, actually?) #

This is the part that turned into a comedy of homelab reality. The plan was to run the classifier on a GPU. The plan met the facts:

• .62 — the box with the actual RTX 3060 it was written for — was powered off and unreachable.
• .52 has an Intel Arc A310, but its BIOS has ReBAR off, so it can’t do GPU compute. Great for video decode, useless for this.
• .42 has no NVIDIA at all.

So after all that, the classifier landed on .42 running on CPU — which is completely fine: CLIP ViT-B/32 is small, and a phone-truck snapshot every few minutes is nowhere near a load. The CUDA Dockerfile still lives in git history for the day .62 comes back. The lesson I keep re-learning: match the deployment to the hardware that’s actually on and reachable, not the hardware you wish you were using. A working CPU service beats an idle GPU one.

The build, and the dependency-pinning bite #

The service is a small Python container: paho-mqtt for the event stream, requests to pull snapshots, optimum[openvino] + transformers for CLIP, Pillow for the image. The first build failed on a dependency conflict that’s worth calling out because it’s easy to miss:

optimum[openvino]==1.21.4  requires  transformers <4.44.0
transformers==4.44.2       ← conflict

Pinning transformers==4.43.4 resolved it. If you wire CLIP through optimum, watch the transformers upper bound — the newest release is often just past what optimum allows.

The compose file passes the render device through for the (intended) iGPU path, using the numeric host GID of renderD128 rather than the group name, because the group name inside the container image doesn’t necessarily map to the host’s render gid:

    group_add:
      - "993"   # host gid of renderD128, so the container can use the iGPU

The Home Assistant side #

Whatever produces the verdict, it ends as an MQTT message; HA announces via TTS to the MPD media player, with a couple of cheap reliability gates — a front-of-house zone so a truck on the cross-street doesn’t trigger, and a cooldown so one delivery equals one announcement, not five as the truck lingers:

automation:
  - alias: "Announce mail delivered"
    trigger:
      - platform: mqtt
        topic: home/mailtruck/detected
    condition:
      - condition: template
        value_template: >
          {{ (now() - states.input_datetime.last_mail_announce.last_changed).total_seconds() > 1800 }}
    action:
      - service: tts.speak
        data:
          entity_id: tts.piper
          media_player_entity_id: media_player.mpd
          message: "The mail truck is out front."
      - service: input_datetime.set_datetime
        target: { entity_id: input_datetime.last_mail_announce }
        data: { datetime: "{{ now() }}" }

Debugging “it dies silently” #

The first end-to-end test was a classic distributed-systems head-scratcher: I published a test message by hand and nothing came out of the speaker, and the command seemed to “die silently.” The instinct is to blame the publish. It was innocent.

mosquitto_pub prints nothing on success and exits — that is the silent success, not a failure. Adding -d showed a clean CONNACK (0) and PUBLISH with rc=0. A subscriber confirmed the message round-tripped the broker fine. So the transport layer — broker, credentials, topic — was all working. The dead end was entirely on the Home Assistant side: the new automation hadn’t been reloaded, so HA simply wasn’t reacting to the topic. The proof was elegant: publishing to the known-good mail topic did speak, which isolated the problem to the one automation that hadn’t loaded. Developer Tools → Reload Automations, and it spoke.

Lesson: when a message-driven automation is silent, prove each hop independently — publish, broker delivery, then the consumer. mosquitto_pub -d plus a mosquitto_sub subscriber tells you in ten seconds whether you’re debugging the network or the app. Ninety-five percent of the time, “it dies silently” is the last hop, not the first.

It grew a second job #

Once the mail-truck path worked, extending it to the garbage truck was almost free — same snapshot, same CLIP call, one more candidate prompt and a second output topic (home/trashtruck/detected). I consolidated both into a single truck_classifier container publishing both topics (running two overlapping classifiers would double-announce the mail). One process, two verdicts.

The payoff moment: a real Waste Management pass at 08:34 one morning scored p_trash 0.995 and fired the announcement on its own — and the snapshot got saved to a labeled review/trash/ folder as a validation sample, so the set for any future fine-tune builds itself from real events.

Takeaways #

• Two-stage is the right shape. Let Frigate do generic detection; bolt on a specificity classifier as a separate service. Don’t try to make one model do both.
• Zero-shot CLIP is a fantastic v1. No labeling, plain-language prompts, argmax + threshold. Fine-tune later if you need to, from snapshots the NVR already saved.
• Deploy to the hardware that’s actually on. A CPU CLIP service that runs beats a GPU one that’s powered off. CLIP ViT-B/32 is small enough that CPU is fine for event-rate inference.
• Watch optimum ↔ transformers version bounds, and pass the render device by numeric GID.
• Debug message pipelines hop by hop. mosquitto_pub -d + a subscriber isolates “broker vs consumer” instantly; “dies silently” is usually a reload away from working.

The house says “the mail truck is out front” now. It also tattles on the garbage truck. Both run on a spare CPU and a 151-million-parameter model that never had to be told what a mail truck looks like — I just asked it in English.

• Frigate
• Home-Assistant
• Clip
• Openvino
• Mqtt
• Docker
• Self-Hosted
• Linux
• Computer-Vision
• Homelab
• Piper