Speaker-Diarized Call Transcription on a CPU — No Cloud, No GPU, No Hugging Face Token

I wanted a speaker-labeled, timestamped transcript of a two-person phone call — the kind an LLM can actually reason about (“what did the customer commit to?”, “when did the price come up?”) — without shipping the audio to a cloud speech-to-text service.

One hard constraint shaped every decision: it all had to run on hardware already in the rack. No new GPU, no API keys, no “just use the cloud for the hard part.” A 35-watt desktop chip that already earns its keep running other things.

It worked. And, as usual, the interesting parts weren’t in the happy path — they were in why the whisper I already had running couldn’t do the job, how to diarize speakers on a CPU with no Hugging Face token, and a genuinely great debugging story that ends with a 114-byte file quietly corrupting every timestamp in the system.

A note on the transcript examples below: the real test call involved an actual third-party business and named individuals who never agreed to be quoted. Every name, company, and phone number in the excerpts here is synthetic — invented for illustration. In many US states, Colorado included, recording calls and reusing what was said carries real legal weight. If you build one of these, redact before you publish. I’m practicing what I preach.

The box

Everything runs on one node in the home lab — dockerhost, an Intel i5-10500T:

Intel(R) Core(TM) i5-10500T CPU @ 2.30GHz
12 threads / 15 GiB RAM / no NVIDIA GPU
Ubuntu, Python 3.12.3

No GPU. Every model below is CPU-only inference. That’s central to the story: this is a low-power 35 W chip doing large-model speech recognition and speaker diarization at roughly real time.

The lab already ran a Home Assistant voice stack — a wyoming-protocol set of containers:

So the natural first thought was: I already have whisper running. I’m basically done.

I was not basically done.

Why the whisper you already have can’t do this

This is the point most people get wrong, so it’s worth being precise.

The Wyoming protocol returns a flat text blob. It’s built for voice-assistant turn-taking: stream audio in, get a string back. There are no word timestamps, no segment timestamps, and no speaker information — because a voice assistant doesn’t need any of that to turn your lights off.

But speaker labeling fundamentally requires timestamps. To attribute a word to a voice, you have to know when it was spoken so you can line it up against “who was speaking then.” The convenient endpoint on port 10300 is structurally incapable of producing a diarized transcript. Not misconfigured — incapable.

And a second, deeper point: whisper doesn’t do diarization at all, in any deployment. Whisper is an ASR model; it turns audio into text. Knowing that two different people are talking is an entirely separate class of model — speaker segmentation, then speaker embedding, then clustering. “Whisper with speakers” is always whisper plus something else.

So the plan wrote itself: leave the Home Assistant container completely untouched (it’s a live service the household depends on) and build a parallel, offline pipeline that calls the faster_whisper Python library directly — same engine family, same box, different entry point, and full access to word-level timestamps.

Three obstacles that only happen on a real, lived-in server

No stereo channels to exploit

The cheap trick for call diarization: many PBX systems record each party on a separate stereo channel. If you have that, “diarization” is just ffmpeg -map_channel and you get a perfect speaker split for free.

I checked. I did not have that:

codec_name=gsm_ms
sample_rate=8000
channels=1

Mono, 8 kHz, GSM. No free lunch — and about as hostile as speech recognition gets: narrowband telephony, a lossy GSM codec, half the sample rate of standard 16 kHz speech models. Real diarization was mandatory.

No venv, no root

$ python3 -m venv ~/stt-transcribe/venv
The virtual environment was not created successfully because ensurepip is not available.

$ sudo -n true
sudo: a password is required

Rather than block on a password I couldn’t type, uv solved it — installs to ~/.local/bin with no root, and builds environments without ensurepip:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 ~/stt-transcribe/venv
uv pip install --python ~/stt-transcribe/venv/bin/python faster-whisper sherpa-onnx

uv as a root-free escape hatch keeps earning its place on machines you don’t fully own.

A 9-byte “model”

The sherpa-onnx speaker-embedding models live on a GitHub release whose tag contains a typo that’s been there long enough to be load-bearing:

.../releases/download/speaker-recongition-models/...
                              ^^^^^^^^^^^^

That’s speaker-recongition-models — the correctly spelled URL 404s. My first download silently produced a 9-byte file (the GitHub “Not Found” body), which then failed at model-load time with a useless error.

Lesson: curl -O against a 404 gives you a file, not an error. Always check the size of a downloaded model. A 9-byte ONNX file is not an ONNX file.

Model choices, and why

Why large-v3 and not the Home Assistant small-int8? The HA container runs the small model because voice commands must return in under a second. This job is offline batch — nobody is waiting on it in real time. That flips the calculus: spend the CPU, buy the accuracy. On 8 kHz GSM phone audio, the jump from small to large-v3 is dramatic. int8 quantization keeps the memory footprint sane and runs much faster than fp32 on CPU, at negligible quality cost here.

Why sherpa-onnx and not pyannote.audio? The obvious diarization library is pyannote, but its models are gated on Hugging Face — accept terms, supply an HF_TOKEN. sherpa-onnx redistributes the same underlying pyannote segmentation model as a plain ONNX file with no token required, and runs it through ONNX Runtime instead of PyTorch. On a CPU-only box with no CUDA, dodging the entire torch dependency stack is a real win in install size and complexity.

The pipeline

  .WAV (8 kHz mono GSM)
        │
        ├─► ffmpeg ──────────────► 16 kHz mono float32 PCM
        │                                │
        │              ┌─────────────────┴─────────────────┐
        │              ▼                                   ▼
        │   sherpa-onnx diarization              faster-whisper large-v3
        │   (segmentation → embedding            (word_timestamps=True)
        │    → clustering, k=2)                          │
        │              │                                 │
        │      [(start, end, speaker), ...]      [(word, start, end), ...]
        │              │                                 │
        │              └────────────┬────────────────────┘
        │                           ▼
        │              per-word speaker assignment
        │              (maximum temporal overlap)
        │                           ▼
        │              merge consecutive same-speaker words
        │                           ▼
        └──────────────►  transcript.json  +  transcript.md

The key structural decision: both branches consume the same decoded sample array in memory. Decode once, feed both models. If you decoded twice with slightly different parameters, the two timebases could drift and the speaker assignment would smear. One decode guarantees whisper and the diarizer are looking at an identical clock.

Decode once, with boring tools

def decode_to_16k_mono(src: Path) -> np.ndarray:
    """ffmpeg-decode anything (GSM/WAV/mp3/...) to float32 mono @16k."""
    tmp = Path(tempfile.mkstemp(suffix=".wav")[1])
    try:
        subprocess.run(
            ["ffmpeg", "-y", "-loglevel", "error", "-i", str(src),
             "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", str(tmp)],
            check=True,
        )
        with wave.open(str(tmp), "rb") as w:
            raw = w.readframes(w.getnframes())
        return np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
    finally:
        tmp.unlink(missing_ok=True)

Deliberately just ffmpeg + the stdlib wave module + numpy — no soundfile, no librosa, no extra native deps. ffmpeg handles the GSM decode that Python audio libraries generally refuse.

Tell the diarizer how many people are on the call

cfg = sherpa_onnx.OfflineSpeakerDiarizationConfig(
    segmentation=...,   # pyannote-segmentation-3.0 ONNX
    embedding=...,      # titanet-large ONNX
    clustering=sherpa_onnx.FastClusteringConfig(num_clusters=num_speakers),
    min_duration_on=0.3,
    min_duration_off=0.5,
)

num_clusters=2 is the single highest-leverage knob. Diarization can either estimate the speaker count from a distance threshold or be told outright. Telling it “there are exactly two people” removes an entire class of error — it can no longer decide that a cough or a burst of line noise is a third participant. When you actually know the answer, supply it. (There’s an honest caveat to this coming up.)

min_duration_on=0.3 / min_duration_off=0.5 govern how aggressively short speech and short silences collapse, which stops single-syllable backchannel — “yeah,” “mhm” — from shattering the segment list.

Whisper parameters tuned for bad phone audio

segments, info = model.transcribe(
    samples,
    word_timestamps=True,
    vad_filter=True,
    beam_size=5,
    condition_on_previous_text=False,   # phone audio loops without this
)

Two of these are specifically telephony hardening:

• condition_on_previous_text=False — whisper normally feeds prior text back in as context. On noisy narrowband audio with long silences, that triggers the notorious degenerate loop where the model emits the same phrase dozens of times. Turning it off trades a little coherence for a lot of robustness.
• vad_filter=True — voice-activity detection strips silence before it reaches the model, which speeds things up and removes another hallucination trigger.

The actual join: assign speakers per word

This is the heart of the thing — the logic that marries two models that never talked to each other:

def speaker_for(start: float, end: float, turns) -> int:
    """Speaker whose diarization segment overlaps [start,end] most; else nearest."""
    best, best_ov = None, 0.0
    for s, e, spk in turns:
        ov = min(end, e) - max(start, s)
        if ov > best_ov:
            best, best_ov = spk, ov
    if best is not None:
        return best
    mid = (start + end) / 2
    return min(turns, key=lambda t: min(abs(mid - t[0]), abs(mid - t[1])))[2] if turns else 0

Assignment happens per word, not per whisper segment — and that matters. Whisper’s own segmentation follows prosody and punctuation, and a single whisper segment routinely straddles a speaker change ("…is Owen available? — Yeah, who’s calling?"). Assign at segment granularity and you smear that whole exchange onto one person. Word granularity lets the speaker boundary land wherever the diarizer actually put it.

The fallback matters too. A word that overlaps no diarization segment — which happens at the very edges and around VAD-trimmed gaps — is assigned to the nearest segment by midpoint distance, rather than silently dropped or defaulted to speaker 0.

Make JSON the source of truth, markdown a projection

The pipeline writes .json first, then renders .md from that JSON via a shared render_md(data). A separate relabel.py imports the same function.

My first relabel.py scraped and patched the already-generated markdown — which was wrong: markdown became the source of truth for some fields and JSON for others, and they drifted the instant anything changed. Making JSON canonical and markdown a pure projection means renaming a speaker regenerates a byte-identical document apart from the labels — and the expensive transcription never re-runs just to fix presentation.

Results on a real call

Test input: a ~6-minute outbound call, FreePBX MixMonitor recording, 8 kHz mono GSM.

[1/4] decoding WAV -> 16k mono            359.5s
[2/4] diarizing (2 speakers)              59 speech segments
[3/4] transcribing: faster-whisper 'large-v3' (int8, 10 threads)
[4/4] 35 turns -> writing transcript

• 359.5 s of audio → 59 diarization segments → 35 merged speaker turns
• Wall clock: about 5 minutes for 6 minutes of audio — roughly 1× realtime on a 12-thread i5-10500T with no GPU, using large-v3
• Peak CPU: ~980% (10 threads saturated)
• Language detection: en, p = 1.00

Transcription quality on narrowband GSM was genuinely good — proper nouns, business names, even a spoken email address came through. The output format, with synthetic names (see the note at the top):

**[00:00:26] Ridgeline Brewing:** Brewing, this is Marcus. How may I help you?

**[00:00:37] Dana (Halcyon Sign Co.):** Thanks. Hi, Owen. My name is Dana. I'm
calling on behalf of Halcyon Sign Co. We're a print and sign shop here in Denver...

The honest accuracy caveats

Two real limitations showed up, and glossing over them would be dishonest:

1. There were actually three voices, not two. An IVR auto-attendant greeting, then one person, then a second person after a hand-off. Because clustering was forced to k=2, the IVR got folded into one cluster and the two humans on the far end merged into the other. Forcing the speaker count is a trade-off, not a free win — correct when you truly know the count, actively wrong when a call gets transferred. Re-running with k=3 separates them.
2. One clear attribution bleed. Around 00:00:27, a “who’s calling?” line that belonged to the far end was attributed to the caller. That’s expected for overlapping speech on a mono mixdown — when two people talk over each other there is exactly one waveform, and the embedding is a blend of both voices. Stereo-per-leg recording is the only real fix.

One more thing to be clear about: the diarizer separates voices; it does not identify them. Speaker names are assigned afterward — by a human reading the transcript, or an LLM inferring from content. Nothing here fingerprints voices against known identities.

The best bug in the whole project: the 114-byte timezone file

Partway through, the timestamps were wrong. Chasing that turned into the most satisfying debugging of the project — and it’s the part I’d keep even if I cut everything else.

Symptom

The FreePBX container reported UTC when it should have been Mountain:

$ docker exec freepbx date
Wed Aug 12 13:45:21 UTC 2026        # should be 07:45 MDT

Everything is configured correctly — and that’s the problem

$ docker exec freepbx sh -c 'echo $TZ; cat /etc/timezone'
America/Denver
America/Denver

$ docker exec freepbx ls -la /etc/localtime
... /etc/localtime -> /usr/share/zoneinfo/America/Denver

TZ is set. /etc/timezone is right. /etc/localtime points at Denver. And date still says UTC. Worse:

$ docker exec freepbx sh -c 'TZ=America/Denver date'
Wed Aug 12 13:46:02 UTC 2026

Explicitly requesting Denver still returns UTC. Every usual suspect is eliminated — which is exactly what makes this one fun.

The actual cause

# docker-compose.yml
    volumes:
      - ./data/izpbx:/data
      - /etc/localtime:/etc/localtime:ro     # <-- the culprit

The host is Etc/UTC. That line bind-mounts the host’s timezone into the container — a common, usually-harmless idiom.

But Docker resolves the container-side path’s symlink before mounting. The container’s /etc/localtime is a symlink to /usr/share/zoneinfo/America/Denver, so Docker followed it and mounted the host’s UTC zoneinfo data on top of the Denver file itself.

The result: /usr/share/zoneinfo/America/Denver inside the container literally contained UTC data. Every timezone lookup faithfully read the Denver file and correctly got UTC out of it. Nothing was “misconfigured” — the timezone database itself had been corrupted by a bind mount.

The smoking gun

# before fix
-rw-r--r-- 1 root root  114 ... /usr/share/zoneinfo/America/Denver
# after fix
-rw-r--r-- 5 root root 2469 ... /usr/share/zoneinfo/America/Denver

114 bytes vs 2469 bytes. A real America/Denver zoneinfo file is ~2.4 KB because it encodes a century of daylight-saving transitions. 114 bytes is the size of the UTC file, which has none. That size gap is the single clearest diagnostic — the detail that flips this from “baffling” to “obvious in hindsight.”

The fix

Delete one line, recreate the container. TZ=America/Denver was already correct and simply started working once the zoneinfo tree was intact:

cp docker-compose.yml docker-compose.yml.bak-tzfix
sed -i '/^\s*-\s*\/etc\/localtime:\/etc\/localtime:ro\s*$/d' docker-compose.yml
docker compose config -q          # validate YAML before touching a live service
docker exec freepbx asterisk -rx "core show channels"   # confirm 0 active calls first
docker compose up -d freepbx

The takeaway PSA: do not bind-mount /etc/localtime into a container that already sets TZ. The idiom predates images shipping their own tzdata and setting TZ properly. When both are present, the mount silently corrupts the container’s timezone database in a way that’s nearly impossible to diagnose from the usual checks — because every file you’d think to inspect is correct.

The second-order timezone bug: parse the epoch, not the timestamp

Same class of problem, one layer up. The transcript header read:

- **Transcribed:** 2026-08-11T20:01:07+00:00

The script used datetime.now().astimezone(), and the host is intentionally Etc/UTC (unlike the container, the host stays UTC on purpose). .astimezone() faithfully picked up the system zone. Fix: never infer the zone from the host.

DEFAULT_TZ = os.environ.get("STT_TZ", "America/Denver")
data["transcribed_at"] = datetime.now(tz).isoformat(timespec="seconds")

But the call time was the more interesting case. FreePBX recording filenames look like this:

out-7205550142-102-20260811-193221-1786476741.25.WAV
    │           │   │        │      └── Asterisk uniqueid = UNIX epoch
    │           │   │        └───────── HHMMSS  (PBX local time -- was UTC!)
    │           │   └────────────────── YYYYMMDD
    │           └────────────────────── extension
    └────────────────────────────────── dialed number  (synthetic here)

The YYYYMMDD-HHMMSS field is written in the PBX container’s local time — which, thanks to the bug above, was UTC. So the filename claimed 19:32:21 for a call that actually happened at 13:32:21 Mountain.

But Asterisk’s uniqueid (1786476741) is an absolute UNIX timestamp — immune to whatever the PBX thinks its timezone is:

def call_started(src: Path, tz: ZoneInfo):
    """Prefer the epoch in the filename; fall back to file mtime.

    The <epoch> is Asterisk's uniqueid: an absolute UNIX timestamp, correct
    regardless of the PBX container's (currently UTC) timezone -- unlike the
    YYYYMMDD-HHMMSS field, which is written in the container's local zone.
    """
    m = re.search(r"-(\d{9,11})\.\d+$", src.stem)
    if m:
        return datetime.fromtimestamp(int(m.group(1)), tz), "filename epoch"
    return datetime.fromtimestamp(src.stat().st_mtime, tz), "file mtime"

The general principle — and a clean closing beat: when a filename carries both a formatted local timestamp and an absolute epoch, always parse the epoch. Formatted timestamps encode the writer’s timezone assumptions; epochs don’t. As a bonus, recordings made before the tz fix (UTC-named) and after it (Mountain-named) are read correctly by the same code path — no renaming, no migration. Which matters, because renaming would break FreePBX’s own CDR references.

What you end up with

On the workstation, ~/.local/bin/call-transcript is a small wrapper that scps the audio up, runs the pipeline on dockerhost, and pulls .md + .json back:

call-transcript recording.WAV --names 'Me,Customer'

And a transcript header (synthetic names, again):

# Call transcript

- **Call started:** 2026-08-11T13:32:21-06:00  _(from filename epoch)_
- **Duration:** 00:05:59 (359s)
- **Engine:** faster-whisper `large-v3` (int8, CPU) + sherpa-onnx
  pyannote-segmentation-3.0 / titanet-large diarization
- **Detected language:** en
- **Speaking time:** Dana (Halcyon Sign Co.) 272s, Ridgeline Brewing 55s

The lessons that outlived the project

1. The whisper you already run probably can’t diarize. Wyoming returns a flat string by design; speaker labels need timestamps, and whisper doesn’t do speakers at all. “Whisper with speakers” is always whisper plus a segmentation + embedding + clustering stack.
2. You can diarize on a CPU with no Hugging Face token. sherpa-onnx redistributes the pyannote model as plain ONNX and skips the entire torch dependency tree — a real win on a GPU-less box.
3. Tell the model what you know. num_clusters=2 erased a whole error class — right up until a call transfer made the true answer 3. Supplying known facts is powerful and a trade-off; own both halves.
4. Check the size of anything you download. A 9-byte “ONNX model” is a 404 in disguise. A 114-byte “America/Denver” is UTC in disguise. File size is an underrated diagnostic.
5. Never bind-mount /etc/localtime into a container that sets TZ — Docker resolves the symlink and corrupts the zoneinfo file the container reads.
6. Parse the epoch, not the formatted timestamp. Epochs are absolute; formatted times carry the writer’s assumptions.

A privacy coda worth stating out loud

The audio here never leaves the LAN — that was the whole point. But analyzing a transcript with a cloud LLM is a disclosure, and it’s the first thing a careful reader will ask about. Local inference and downstream cloud analysis are genuinely different privacy postures; don’t let “it’s all self-hosted” quietly stop being true at the last step. And whatever you build, keep real people’s names out of what you publish — which is exactly why every name on this page is invented.

• Speech-to-Text
• Whisper
• Diarization
• Sherpa-Onnx
• Faster-Whisper
• Self-Hosted
• Linux
• Freepbx
• Asterisk
• Docker
• Timezone
• Cpu-Inference
• Privacy