Forty Cents of Agent Research, One Hallucinated Email, and the Script That Caught It
Vicente Louvet
- 8 minutes read - 1668 wordsNeonBench is in public beta, and the people most likely to care are the schools and studios that still teach glass bending. There is no directory of them, and finding them is tedious web research, which is exactly what a cheap agent is for, and exactly where a cheap agent will make things up. So I spent forty cents on ten agent jobs, got back a spreadsheet of trade schools, and built a deterministic verifier that refused to trust any of them. It caught a hallucinated email address. The whole thing is the subject of this post.
The agent is the same Hermes/DeepSeek setup I use for code jobs on the homelab GPU box (.62): an open-source agent framework in a container, a DeepSeek V4 Flash model through OpenRouter, briefs written as Markdown files, one job per brief, run in parallel. For research jobs it gets the web, file and terminal toolsets and a plain output directory instead of a git worktree. A pleasant surprise: the framework’s web search works with no API key at all (it falls back to a free search backend), so the research side cost nothing beyond model tokens.
The punchline, spoiled up front: the model followed every rule except the one that mattered, a dumb script was the only thing that noticed, and the forty cents bought a lesson about rules versus controls that I’d happily have paid more for.
Why schools #
NeonBench is a local-first pattern tool for hand-bent neon: a single binary that vectorizes a logo into tube centerlines and prints a 1:1 paper pattern with a bend list. The people learning to bend glass today are in the schools and studios that keep the trade alive, and a tool that prints 1:1 paper patterns has to earn its keep at their benches first. There is no directory of them. Web searches, a country list and some judgment is the actual method, and it is an aggravating one to do by hand.
The pipeline #
Ten briefs became ten jobs; the whole thing processed as a script chain. Each step is small and checkable, which is the point:
make_briefs.py— one template plus a seed CSV of regions and school types becomes 7 briefs (US west / central / east, Canada+UK+Ireland, Europe, Asia-Pacific+Latin America, plus one by programme type). Each brief states the output CSV contract verbatim and the rule “never construct or guess an email address; only record one you saw on a page, with the URL”.stage.shcopies each job’s brief plus the runner into the container;docker exec -dlaunches them.collect.shpulls each job’s CSV back.merge.pydedupes by registered domain, not by name, which turned out to matter: 13 cross-job duplicates surfaced, because the type-based job overlaps the regional ones and the same school appears under slightly different names.verify_emails.py— for every row, re-fetch the cited URL and require the literal address to appear on the page. A--playwrightmode renders JavaScript-heavy pages with a real Chromium; an SSL-unverified fallback handles the schools with broken certificates.make_gapfill_briefs.py— a second, smaller round of 3 jobs aimed only at the schools that came back without a verified address.render_drafts.py --postal-address "..."— one Markdown draft per contact plus a mail-merge CSV from a template. It refuses to run without a postal address, because CAN-SPAM requires a physical mailing address in every commercial email and I did not want the option to forget. Nothing in the pipeline sends mail; sending is a separate, human step.
A code block for the shape of the verifier’s core check — the control, not the rule:
def verify(row): # in outline
html = fetch(row["url"])
ok = row["email"] in html # literal address present on the page
if not ok and row.get("js"):
html = playwright_render(row["url"]) # Chromium for JS-heavy pages
ok = row["email"] in html
return ok
What came back #
Ten jobs (7 regional plus 3 gap-fill) ran in parallel at 3 to 5 minutes each and made 22 to 53 tool calls apiece, none hitting the 60-call cap. The model bill was $0.33 for the first seven and $0.07 for the three gap-fills: about $0.40 total.
| measure | count |
|---|---|
| research jobs | 10 |
| unique schools after dedupe | 60 |
| with an email found | 55 |
| verified by re-fetch | 50 (49 by script, 1 by hand from a home page) |
| left unverified | 5 (one behind a 403 wall the fetcher cannot pass) |
| form-only, no public address | 7 |
| cross-job duplicates removed | 13 |
| finally sent | 49 |
The geographic spread of the deduped sixty: 40 US, 7 UK, 4 Australia, 2 France, 2 Hong Kong, and one each from Canada, Germany, Japan, Poland and Sweden. Dedupe by registered domain removed 13 cross-job duplicates on the way to those 60.
The address the model invented #
Cloudflare’s email-obfuscation feature replaces addresses in the HTML with the literal text [email protected] and decodes them client-side. One job hit such a page, and the model, despite the never-construct rule sitting there in bold at the top of its instructions, turned the obfuscation marker into a plausible address at the school’s domain and recorded it with a straight face and a real citation URL. The verifier re-fetched the page, found no such address, and failed the row.
Three things about this, in order of how much they changed my thinking:
- The rule was written down. The model is not malicious; it is completing a pattern. Read an obfuscation marker that happens to look like an email the way a model reads it and the plausible completion writes itself. A rule is not a control.
- The control is a deterministic step that re-derives the fact from the source. The agent’s confidence, the citation, and the tidy CSV are not evidence. The page is evidence. I keep having to relearn this: an AI’s output is a claim, and a claim gets verified against the thing it cites, not given credit for citing it.
- The same verifier caught the opposite problem too. One school’s only discoverable address came from a donor-society PDF, so it was valid but the wrong office: there’s a difference between “this address exists” and “this is where you should write.” Validity and provenance are different checks, and that row was excluded by hand.
The decision rule that fell out of it: include the 12 rows marked low confidence (address verified, “is this the right office” genuinely unclear), skip anything unverified. Never send an unverified row. That rule is what let the 49 go out while the 5 unverified ones wait for a hand check.
Two small traps #
node -e and argv. With node -e "<script>", process.argv[1] is the first argument, not a script path. My first Playwright pass read the wrong argv index and silently rendered nothing, so every JavaScript-heavy page reported “no email,” and I only caught it because the zero was too round. The lesson: when an otherwise-working pipeline returns a suspiciously clean zero, suspect the harness before the data.
Dedupe on names fails on the long tail. Abbreviations, “Studio” vs “School”, trailing city names: name matching is fine for the popular fifty and falls apart on the exact schools that made this list worth having. Registered domain is the stable key for an organisation, and it is what merged the 13 cross-job duplicates without a human eyeballing each pair.
Sending it #
The send went through my own self-hosted Odoo CRM, as plain mail.mail records through its SMTP relay — so every outreach contact exists as a tagged CRM record and the whole campaign can be undone by filtering on the tag. I imported 68 partner records (48 schools as companies, 20 named contacts), sent one test message to myself, then sent the 49. Replies land in my normal mailbox.
Three Odoo XML-RPC traps, each one sentence:
mail.mail.send()returnsNone, and the XML-RPC layer raises a marshalling fault after the send already happened; catch it and re-read the record state instead of retrying.- The message author must be the RPC user’s own partner record; using the company as author fails with a generic security error.
- Private ORM helpers (names starting with an underscore) are not callable over RPC; look up external ids with
search_readonir.model.datainstead.
No replies to report yet; it only just went out. I will not claim an outcome that has not happened.
Takeaways #
- A rule is not a control. I wrote the never-construct rule in bold in every brief and a model invented an email anyway, because completing a pattern is what a model does. Make the control a deterministic step that re-fetches the source and checks.
- The agent’s citation is not evidence. A URL the model wrote next to a fact proves the model wrote it, nothing more. Re-derive the fact from the page yourself.
- Validity and provenance are different. “This address exists” and “this is the right office” are separate checks; label them separately and keep the split visible.
- Never send an unverified row. Decide the boundary before anything goes out, and let the verifier’s failures keep those rows in the queue.
- Dedupe organizations by registered domain, not by name. The long tail is where names stop being reliable.
- Validate your harness, not just your data. A silently zero output from a buggy argv read masqueraded as a finding until the cleanliness of the number gave it away.
- Bake the legal gate into the software. The render step refusing to run without a postal address made CAN-SPAM compliance un-forgettable by construction, which is worth more than a checklist item.
Sixty schools is not every neon school in the world; it is what ten cheap searches found, and 13 programmes without a public address were never contacted. One person — me, not the model — hand-labelled “right office / wrong office,” and that judgment is mine to own. And forty cents is the model bill; my own hours writing briefs and reviewing CSVs were the real cost. I’d call it the cheapest lesson about trust I’ve bought this year.