Building a Headless AI Workstation: Ubuntu Server + RTX 3060 + Ollama, Zero-Touch from a USB Stick

- 6 minutes read - 1076 words

Several of my homelab projects want a local GPU for inference — running a vision LLM for OCR, offloading work that would otherwise hit a cloud API, keeping a small model resident and fast. So I built a dedicated box for it: a Dell Precision 3640 (i3-10105F, 16 GB) with a Gigabyte RTX 3060 12 GB, running Ubuntu Server and serving a vision model through Ollama.

Two things made this worth writing up. First, the actual GPU-serving setup has a few non-obvious choices (which driver, no CUDA Toolkit, exposing the port safely). Second — the fun part — I made the whole install zero-touch: plug in a USB stick, walk away, and the box comes up on a static IP with SSH pubkey auth ready, no monitor or keyboard ever attached. This post covers both, and both live as reusable docs/scripts in my infra-config repo.

Why this box exists #

The concrete driver was my jf-barcode-search tool, whose OCR backend can escalate hard-to-read covers to a vision LLM (VLM). Running that VLM in the cloud is fine until you’re doing it constantly; running it locally on a $200-ish used Dell + a mid-range GPU pays for itself fast and keeps the images on your LAN. The same box then becomes the natural home for any other “needs a GPU but not a datacenter” job.

The GPU serving setup #

The software stack is deliberately minimal:

NVIDIA driver — the easy way. For an Ampere card like the 3060, ubuntu-drivers picks the right one (550+):

sudo ubuntu-drivers install

No CUDA Toolkit. This trips people up: you do not need to install the full CUDA Toolkit to run Ollama. Ollama bundles its own CUDA runtime. Installing the toolkit separately just adds gigabytes and a second thing to keep in sync with the driver. Skip it.

A BIOS/hardware note that saves grief: on the Dell, turn Secure Boot off. Otherwise the DKMS-built NVIDIA module has to be MOK-enrolled on every kernel update, which means an unattended box silently loses its GPU after a routine apt upgrade. (The 3060 also just needs its single 8-pin power lead — no dual-cable drama.)

Ollama as a systemd service, exposed on the LAN. By default Ollama binds to localhost; a systemd drop-in makes it listen on the network so other hosts can use it:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then pull the model — a small vision model is plenty for OCR:

ollama pull qwen2.5vl:3b

Confirm it’s actually on the GPU, not silently falling back to CPU:

ollama ps        # should show 100% GPU
nvidia-smi       # the ollama process resident in VRAM

Don’t leave an unauthenticated inference port open #

OLLAMA_HOST=0.0.0.0:11434 has no auth. Anyone who can reach the port can use your GPU. So the port must only be reachable over trusted networks:

# Tailscale for remote reach + UFW to fence the port to LAN/tailnet
sudo ufw allow from 192.168.1.0/24 to any port 11434 proto tcp
sudo ufw allow in on tailscale0 to any port 11434 proto tcp

Then the app wiring is a two-line change in jf-barcode-search — flip the OCR engine to auto-escalate and point it at the new box:

- OCR_ENGINE=tesseract
+ OCR_ENGINE=auto
- VLM_URL=http://127.0.0.1:11434/v1/chat/completions
+ VLM_URL=http://<workstation-ip>:11434/v1/chat/completions
  VLM_MODEL=qwen2.5vl:3b

auto runs fast Tesseract first and only escalates to the VLM when the cheap path fails — the GPU box earns its keep on the hard cases, not every scan.

The zero-touch install #

Here’s the part I’m happiest with. I did not want to attach a monitor and keyboard to this thing — it lives on a shelf. The goal: plug in a USB stick, boot, and later just ssh in. Ubuntu’s autoinstall (subiquity) makes this possible, but with two gotchas most guides gloss over.

Gotcha 1: subiquity stops to confirm unless you tell the kernel not to. A user-data seed alone isn’t enough — the installer still pauses at a confirmation screen. You have to bake the autoinstall kernel parameter into the ISO’s GRUB config, or it’s not truly hands-off. So I wrote a small remaster script that takes a stock Ubuntu Server ISO and:

  • embeds the NoCloud seed (user-data + meta-data) at /server on the ISO, and
  • rewrites the GRUB kernel line to add autoinstall ds=nocloud;s=/cdrom/server/.

The one fiddly detail: that semicolon must be GRUB-escaped as \; in the sed, or GRUB truncates the parameter. (I burned a good ten minutes proving to myself that sed’s backslash handling was doing the right thing — the fix is \; inside the single-quoted sed expression.)

Gotcha 2: no secrets in git. The seed needs a console password hash, but the repo is shared. So the user-data ships with a REPLACE_ME_PW_HASH placeholder, and the build script substitutes a hash you generate at build time (openssl passwd -6) — nothing sensitive is committed. SSH itself is pubkey-only (allow-pw: false); the console password is just a fallback.

The seed itself pins everything the headless box needs to come up ready:

autoinstall:
  identity: { hostname: ai-workstation, username: vlouvet, ... }
  ssh:
    install-server: true
    allow-pw: false
    authorized-keys: [ "ssh-ed25519 AAAA... " ]   # my key, baked in
  network:
    ethernets:
      primary:
        addresses: [192.168.1.62/24]
        routes: [{ to: default, via: 192.168.1.1 }]
        nameservers: { addresses: [192.168.1.12] }

Static IP .62, my SSH key pre-authorized, password login disabled. Flash the remastered ISO, boot the Dell once (it auto-selects the installer), and after the automatic first reboot the box is sitting at 192.168.1.62 with SSH waiting. I validated the seed with python3 -c "import yaml; yaml.safe_load(...)" before ever burning it — an autoinstall YAML typo means a wasted trip to physically reboot the machine, so parse it first.

Takeaways #

  • Ollama bundles CUDA — don’t install the CUDA Toolkit. The driver (ubuntu-drivers install) is all you need.
  • Turn off Secure Boot for a headless GPU box, or a kernel update silently un-loads the NVIDIA module via unattended MOK enrollment.
  • OLLAMA_HOST=0.0.0.0 has no auth — fence the port with UFW + Tailscale.
  • True zero-touch autoinstall needs the autoinstall kernel arg in GRUB, not just a seed file — and the NoCloud semicolon must be GRUB-escaped.
  • Keep the password hash out of git with a build-time placeholder substitution; make SSH pubkey-only.
  • Validate the autoinstall YAML before burning — a typo costs a physical reboot on a headless box.

The result: a self-provisioning inference node that I can rebuild from a USB stick and an infra-config checkout in about the time it takes to make coffee — and it never needs a monitor.