Spinning Up a Local AI Lab: OpenCode, llama.cpp, and LoRA

Most “local AI” write-ups stop at a chat window. One model. One prompt box. A screenshot that proves the weights loaded.

That is a demo. It is not a lab.

A lab is what you run when vendor defaults start to feel expensive—and when you still want skin in the runtime. I keep one so I can route work, measure capacity, and change a model without asking a salesperson for a new SKU. The point is not to boycott the cloud. The point is to own the harness so the model is swappable.

This is a howto. It is more operational than most pieces on this site, and it is still a leadership argument: if you cannot interrogate the farm, you are renting judgment. You do not need my hardware. You do need the constraints.

Why a Local Lab Now

The token bill is a meter you do not own. Seats were a project. Agent loops are an operating cost. Hard-capping the invoice rations the people who actually get leverage. Leaving the meter unowned produces fat tails and unattributed retries.

LLMs are becoming a commodity. Durable advantage sits in workflow: which model takes which turn, what context is always in the prompt, what a human still has to stand behind. If that layer lives only inside one vendor’s product, you rented the meter.

A local lab makes that argument tactile.

• You feel KV cache as a budget, not a slide.
• You feel why a “fast” model and a “thinking” model are different products.
• You feel why an agent that can edit the tree is more expensive than an agent that can only search it.
• You feel why review capacity is the real constraint—generation is cheap once the farm is up.

This is the AI-runtime sibling of the leadership-grade homelab. That piece is GitOps, blast radius, and restore. This one is inference you own, agents you direct, and adapters you train only when the data deserves it.

What You Are Actually Building

Not one 70B chat. A four-specialist farm behind one OpenAI-compatible /v1, consumed by OpenCode agents that pick a model per role.

The stack that earns its keep:

Layer	What I run	Why it is the layer
Client	OpenCode on the workstation, optional OpenAI-compatible clients	Agents, permissions, and pins—not a single prompt box
Contract	OpenAI /v1 plus a bearer token	Anything that speaks the contract can swap in
Edge (optional)	A reverse proxy or tunnel in front of the host	Remote use without exposing the inference process itself
Engine	One llama-server in router mode	Four models in one process, one listen port
Weights	Unsloth GGUFs from Hugging Face	Quantized, dated, replaceable without a vendor SKU

The four specialists in the current farm:

Role	Model	Job
Think	Qwen3.8 27B Q5, plus a vision projector	Planning, architecture, images
Code	Qwen3-Coder 30B-A3B Q4	Build, explore, general implementation
Fast	Nemotron 3.5 Lightning 30B-A3B Q4	Scout, title, summary, compaction, short tool loops
Route	Arch-Router 1.5B Q4	Intent only. Never a chat model.

Lightning’s KV is cheap because it is a hybrid Mamba stack—most layers are not dense attention. That is why it can hold a large context pool and several parallel slots while Coder cannot. The farm is an economics decision, not a trophy case.

Exclusive modes are a product, not a footnote. The same box cannot honestly run the four-model farm, a single-model “burn the memory on one engine” mode, and a speech stack at the same time. I flip modes on purpose. If everything is always up, you have not sized the machine. You have lied to yourself about capacity.

Hardware Constraints That Matter

I run the farm on an AMD Strix Halo class APU—Ryzen AI MAX+ with on the order of 128 GiB unified memory. The interesting property is not the logo. It is that weights and KV live in one pool the iGPU can actually use.

A few constraints were settled the hard way. They transfer to any high-memory APU, and several of them transfer to discrete cards.

Leave the BIOS VRAM carveout low. The Windows habit is to gift the GPU a huge static slab. On Linux, llama.cpp wants GTT at runtime. Raising the carveout shrinks what the farm can allocate. Confirm free memory is in the neighborhood of the DIMM size, not the carveout size.

Raise GTT, then reboot. A 110 GiB ceiling is what makes four resident models plus production-sized KV possible on this class of machine. If you skip the reboot, you will debug the wrong layer for an hour.

The render node is a permission problem. Headless, there is no seat ACL on /dev/dri/renderD128. The service user needs render and video. Miss it and llama.cpp falls back to CPU without a dramatic error. Throughput collapses. The fix is groups plus a launcher that refuses to start when --list-devices prints nothing.

mmap = 0 on this APU. Full GPU loads are faster and safer than paging the GGUF. Flash attention stays on; quantized V cache requires it. Do not cargo-cult flags from a CUDA write-up.

Disk is a swap budget, not “enough for the files.” The four GGUFs plus projector and router are on the order of 80+ GiB. Keep room for the previous GGUF during a swap. A lab that cannot roll back a weight is not a lab.

Train somewhere else. Unsloth’s comfortable path is Ampere-class NVIDIA, 24 GiB or more, for 7B–14B QLoRA. A Pascal-era card will infer. It is a poor first trainer. Rent GPU hours if that is what you have. Do not pretend the inference host is also the fine-tune host just because both have a GPU in the name.

You can run a smaller farm on less memory. Cut parallel slots. Drop a specialist. Do not keep four full context pools and hope.

Phase How-To

Each phase ends with a check. If the check fails, stop. Later phases assume it passed.

Phase 1 — Host baseline

Install a current Ubuntu-class image, or the vendor AI image for the APU. Wired Ethernet. SSH with a key. Headless once you are done installing: a desktop session competes for the same iGPU the farm needs.

free -h                          # unified memory in the DIMM neighborhood
ls /dev/dri/                     # card0 and renderD128
id -nG                           # must include render and video after you add them

Build llama.cpp with the backend the GPU actually speaks. On this APU that is Vulkan / RADV, not the ROCm path I evaluated and set aside.

sudo apt-get install -y build-essential cmake git \
  vulkan-tools libvulkan-dev glslc
vulkaninfo --summary             # a Radeon device, not an empty report

git clone https://github.com/ggml-org/llama.cpp.git ~/llama.cpp
cd ~/llama.cpp
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j"$(nproc)"
./build/bin/llama-server --list-devices

Pin a llama.cpp tag you have smoked. Router mode, long context, and newer hybrid architectures all have minimum builds. “Latest main” is not a release process.

Check: --list-devices shows a Vulkan (or CUDA) device. (none) means you are about to serve on CPU. Fix groups and re-login before you download a single GGUF.

Phase 2 — Weights

Install the Hugging Face CLI in a user tool, not a PEP-668 system Python.

pipx install huggingface-hub
export PATH="$HOME/.local/bin:$PATH"

mkdir -p ~/models && cd ~/models

hf download unsloth/Qwen3.8-27B-GGUF \
  --include "*UD-Q5_K_M.gguf" "mmproj-F16.gguf" --local-dir .

hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  --include "*UD-Q4_K_XL.gguf" --local-dir .

hf download unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF \
  --include "*UD-Q4_K_XL.gguf" --local-dir .

hf download katanemo/Arch-Router-1.5B.gguf \
  --include "Arch-Router-1.5B.Q4_K_M.gguf" --local-dir .

Quantization is a product choice. Q5 on the thinking model is extra fidelity you can feel in plans. Q4 on Coder and Lightning is how they coexist with production-sized KV. Do not “just use Q8” on all four unless you have measured the GTT leftover.

Check: the four GGUFs plus the mmproj are on disk, and you still have space for one old weight during a swap.

Phase 3 — Router INI

Section names are the OpenAI model IDs. If OpenCode says qwen3-coder-30b-a3b and the INI says coder, every pinned agent 404s. Match them character for character.

[*]
n-gpu-layers = 99
flash-attn = on
fit = off
jinja = 1
mmap = 0
cache-type-k = q8_0
cache-type-v = q8_0
batch-size = 2048
ubatch-size = 512

[qwen3.8-27b-q5]
model = ~/models/Qwen3.8-27B-UD-Q5_K_M.gguf
mmproj = ~/models/mmproj-F16.gguf
ctx-size = 262144
parallel = 1
reasoning = auto

[qwen3-coder-30b-a3b]
model = ~/models/Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf
ctx-size = 524288
parallel = 2
kv-unified = on

[nemotron-lightning-q4]
model = ~/models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q4_K_XL.gguf
ctx-size = 1048576
parallel = 4
kv-unified = on
reasoning = off

[arch-router-1.5b]
model = ~/models/Arch-Router-1.5B.Q4_K_M.gguf
ctx-size = 8192
parallel = 4
temp = 0.0

Context pools are sized so a slot can take a full window without stealing from a sibling until the unified cache is exhausted. I learned that as "Context size has been exceeded" during real agent turns, not in a bench script.

Advertise a consistent context to the client even when the host pool is larger. OpenCode should see one contract. The host can keep a bigger unified cache behind it.

Check: the INI parses, paths resolve, and you have not added a fifth section. Four is the budget on this class of machine.

Phase 4 — Serve and smoke

Put the API key in a file the launcher reads, mode 0600. Do not put it on the llama-server command line. ps and the journal will keep it forever.

# Router mode is "a models file plus a max."
# Empty preset = single-model bisect. Useful. Not the farm.

export LLAMA_API_KEY=YOUR_API_KEY
~/llama.cpp/build/bin/llama-server \
  --models-preset ~/.config/lab/models.ini \
  --models-max 4 \
  --host 127.0.0.1 \
  --port 8080 \
  --api-key-file ~/.config/lab/api-key

Bind loopback until you have a proxy and a token you trust. A LAN bind is a product decision, not a default.

curl -sf -H "Authorization: Bearer $LLAMA_API_KEY" \
  http://127.0.0.1:8080/v1/models

You want exactly the four IDs. Then one short chat per specialist. If Lightning is fast and Coder is silent, you have a load problem, not a “models are hard” problem.

If you publish the API past the machine, put a keepalive-aware proxy in front. Long agent turns look like dead connections to some edges. A tunnel is a pattern. It is not a substitute for the bearer token.

Check: /v1/models lists the four IDs, and each specialist completes a one-paragraph smoke prompt.

Phase 5 — OpenCode as the client

Install OpenCode on the workstation. Point a custom provider at the farm with @ai-sdk/openai-compatible. Model IDs in this file must match the INI.

{
  "model": "lab/auto",
  "small_model": "lab/nemotron-lightning-q4",
  "provider": {
    "lab": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local lab",
      "options": {
        "baseURL": "https://llm.example.com/v1",
        "apiKey": "YOUR_API_KEY"
      },
      "models": {
        "auto": { "name": "Auto (Arch-Router)" },
        "qwen3.8-27b-q5": { "name": "Qwen3.8 27B (thinking)" },
        "qwen3-coder-30b-a3b": { "name": "Qwen3-Coder 30B-A3B" },
        "nemotron-lightning-q4": { "name": "Nemotron Lightning 30B-A3B" }
      }
    }
  },
  "agent": {
    "build": { "model": "lab/auto" },
    "plan": { "model": "lab/qwen3.8-27b-q5" },
    "fast": { "model": "lab/nemotron-lightning-q4" },
    "explore": { "model": "lab/qwen3-coder-30b-a3b" },
    "scout": { "model": "lab/nemotron-lightning-q4" }
  }
}

auto is not served by llama.cpp. It is a picker label. A small plugin calls Arch-Router once per build session, maps nano / coder / qwen onto Lightning / Coder / Qwen, then sticks. Explicit picks are never rewritten. Vision forces Qwen. plan is pinned to Qwen. fast and scout stay on Lightning so they do not steal Coder slots.

If the plugin cannot reach the router, fail open to Coder—not to a hang, and not to a model that was never a chat model.

Check: a plan turn hits Qwen, a mechanical fast turn hits Lightning, and a build with the Auto picker does not send the literal string auto to /v1/chat/completions.

Phase 6 — A real agent turn

Smoke scripts lie in a friendly way. The farm is proven when OpenCode completes a turn that looks like work:

1. @plan — propose an approach for a real change in a repo you own.
2. @scout or @explore — gather files and docs with read-only permissions.
3. @build — implement a thin slice. Watch which model the router chose.
4. You review the diff. The lab does not merge itself.

If step 4 feels optional, you built a toy. Cognitive debt still applies: the human keeps intent, boundaries, and outcomes.

Check: you can name which model did which step without opening a vendor dashboard.

OpenCode Is the Control Plane

The weights are not the product. The pins are.

Treat agents as capacity, the way a platform team treats namespaces.

• plan is allowed to think. It is not allowed to thrash the Coder slots.
• explore is read-only. If it can bash and edit, it will, and you will pay for it in KV and in review.
• scout is research that must not queue behind a refactor.
• fast is mechanical: git, search, short loops. Cap the step count. Deny doom loops.
• build is the only agent that should get Auto. Auto is a privilege, not a default for every role.

This is how workflow, not vendor, becomes a lab instead of a slogan. You can move plan to a different thinking model next month. You cannot move it if “the assistant” is one anonymous blob.

Permissions are part of the farm. A read-only explore agent is the cheapest concurrency you will ever buy: it cannot corrupt the tree, and it cannot start a second write loop while Coder is already at parallel = 2.

Practical LoRA

The daily farm above is inference. There is no LoRA training loop in that stack today. Say that out loud before you schedule a fine-tune. Adapters are the next layer, not a badge you sew onto the README.

An adapter earns a slot when the failure is distributional: the base model can talk, but it will not reliably emit your tool JSON, your incident voice, your domain nouns, your “we never do X” shape. A better prompt or an OpenCode skill is cheaper when the failure is contextual. Most teams try LoRA because the prompt was vague.

That is the same argument as training models with models: architecture is not the moat. Labeled examples are. Start with a few hundred expert rows you would be willing to put your name on. If you cannot describe the eval, you are not ready to train.

When not to fine-tune

• You have not written a twenty-example eval you can score without vibes.
• The task changes weekly. You will be chasing last week’s adapter.
• You are trying to teach a 30B model your whole company. That is a data program, not a weekend LoRA.
• You have not tried a skill, a pinned prompt, or a cheaper specialist.

Train

Use Unsloth (or PEFT + TRL) on Ampere-class NVIDIA, 24 GiB+. First adapter: a 7B–14B instruct checkpoint in the same family as the specialist you will replace. 30B QLoRA is possible. It is a bad first lab experiment.

from unsloth import FastLanguageModel
from trl import SFTTrainer
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-Coder-7B-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

dataset = load_dataset("json", data_files="seed.jsonl", split="train")

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=4096,
)
trainer.train()

model.save_pretrained("lora_adapter")
# Or merge and export GGUF when the eval passes.

seed.jsonl is the whole product. Each line is an input/output pair a domain expert would defend. If you generate synthetic rows, a human still gates the ones that enter training. That is training models with models, not an excuse to skip the gate.

Hold out an eval set before you train. Score format, refusals, and the two or three behaviors you actually care about. Do not promote on “it feels sharper.”

Export and serve

Two honest paths:

1. Keep the adapter. llama.cpp can load a base GGUF plus --lora. Useful while you A/B.
2. Merge, then quantize to GGUF. Simpler operations: one file, one INI section, one rollback (keep the previous GGUF).

Promote one specialist. Usually Coder, or a new domain worker—not Qwen, not the router. Change one section in the INI. Leave Arch-Router and Lightning alone. If the adapter is worse, flip the filename back. That is the whole rollback story.

Do not train on the inference host “because it is idle tonight.” Exclusive modes exist because memory is a lie detector. A failed train job that fragments GTT is an outage with extra steps.

Check: the eval set beats the base model on the behaviors you named, and a farm smoke still returns four IDs after the swap.

What to Refuse

Public-by-default. Loopback, then a token, then a proxy you understand. “It is only on my LAN” is how labs become incident reports.

Prompt archives as a shared drive. Local does not mean free of retention rules. Treat traces like production logs. If you would not paste the prompt into the company ticket system, do not write it to a world-readable file.

The farm as a substitute for review. Local generation still produces diffs a human has to own. Faster unowned code is still cognitive debt. The lab should make review more honest, not optional.

One anonymous blob. If every turn hits the same model, you did not build a farm. You built a chat UI with extra YAML.

Fine-tune as identity. An adapter is a patch. It is not a strategy. Most of the leverage is still pins, permissions, and evals.

Gear theater. A louder GPU that never runs a scored eval is the LED rack of the AI era. Constraints beat capacity. That is the same rule as the homelab.

Leadership Checklist

Answer these without a vendor slide:

1. Can you name the model that should take planning versus mechanical work—and change the pin without a procurement cycle?
2. What happens to the farm when context is large and three agents run at once? Have you watched it, or imagined it?
3. If the public API disappeared tomorrow, which workflow still completes on hardware you control?
4. Who owns the bearer token, the prompt traces, and the rollback GGUF?
5. What is the eval for the next adapter—and who is allowed to promote it into a live slot?
6. After a local agent writes a diff, who still stands behind the merge?

If those answers are fuzzy, you do not have a tooling gap. You have an unowned harness.

You Need a Farm You Can Interrogate

You do not need this BOM. You need a contract you can point other clients at, specialists you can justify, and a promotion path that includes “no.”

A leadership-grade homelab keeps infrastructure judgment current. A local AI lab keeps harness judgment current: routing, capacity, eval, and the difference between a model and a workflow. I use both for the same reason. Abstraction without practice produces decks.

Skin in the Runtime is the longer argument—review capacity, cognitive debt, measurement, agent governance. If you would rather start with a conversation than a farm, get in touch.

I take on selective fractional and advisory work where AI-stakes platform decisions need battle-tested judgment. For how that engagement is scoped, see The Fractional CTO Playbook.

---

Standing up a local farm—or trying to get the harness out of a vendor default? Connect with me on LinkedIn to continue the conversation.