Several of my homelab projects want a local GPU for inference — running a vision LLM for OCR, offloading work that would otherwise hit a cloud API, keeping a small model resident and fast. So I built a dedicated box for it: a Dell Precision 3640 (i3-10105F, 16 GB) with a Gigabyte RTX 3060 12 GB, running Ubuntu Server and serving a vision model through Ollama.

Two things made this worth writing up. First, the actual GPU-serving setup has a few non-obvious choices (which driver, no CUDA Toolkit, exposing the port safely). Second — the fun part — I made the whole install zero-touch: plug in a USB stick, walk away, and the box comes up on a static IP with SSH pubkey auth ready, no monitor or keyboard ever attached. This post covers both, and both live as reusable docs/scripts in my infra-config repo.