Local LLMs, Political Bias, and Why llama.cpp Still Matters

TL;DR. Public frontier models are excellent at code and ops. They are not politically neutral. The Neutrality Project benchmark puts numbers on that lean. If you need an opinion that is not pre-framed by a lab’s safety stack, a local model via llama.cpp is still the clean path. Memory is the real blocker today. When TurboQuant-style KV compression lands in llama.cpp, routing writing through a local uncensored model and everything else through the best cloud models becomes a normal agent config, not a hobbyist stunt.

The quiet problem with “just ask the model”

I use cloud LLMs every day. For Spark job rewrites, Ceph runbooks, sizing notes, and agent orchestration they are hard to beat. The failure mode is different when the task is judgment, not code.

Ask a public model to draft a political brief, score a contested claim, or summarize a cultural fight, and you do not only get facts. You get a default framing: which options sound “reasonable”, which caveats appear first, which sides are described as extreme. That framing is not a bug you can prompt away in one shot. It is baked into training data, RLHF, and refusal policy.

During an election cycle the gap is obvious. You do not need a conspiracy theory. You need a second opinion that does not share the same corporate safety layer. Local inference is still the only practical way I know to get that.

What the Neutrality Project actually measured

The Neutrality Project runs an open Political Neutrality Benchmark: thousands of real survey questions, six anchored axes (economy, social values, foreign policy, environment, religion, national identity), and a self-anchoring design so each model is scored against its own far-left and far-right personas rather than against the authors’ preferred center. In practice that means a model’s score is relative to the range it would generate if explicitly prompted to argue from its own extremes, not to some fixed political ruler every model gets measured against.

The Release 01 results are blunt:

• 97 of 108 measured axis positions landed left of center (across the TNP-run set they highlight).
• Average overall position about −0.41 on a −1 (progressive) to +1 (conservative) self-anchored scale.
• Strongest lean on environment (about −0.82); foreign policy closest to center (about −0.11).
• Every model they highlight leans progressive overall. xAI’s Grok family is the main near-center exception, not a full counterweight across the board.

Read the caveats on their site. Positions are relative to each model’s own anchors. Refusals can skew some axes. Community runs wait for re-verification. None of that erases the shape of the chart: if you only talk to public API models, you are sampling from a progressive-leaning distribution of worldviews.

That is the empirical case for keeping a local stack. Not “cloud models are useless.” Cloud models are biased instruments. Treat them like any other instrument with a known offset.

Cloud models are biased instruments. Treat them like any other instrument with a known offset.

Why llama.cpp still has a future

llama.cpp is boring infrastructure in the best sense. C/C++ inference, GGUF weights, CPU and consumer GPU paths, a local OpenAI-compatible server if you want one. No account, no content policy rewrite overnight, no “we updated the model and your prompts broke.”

Local models keep three properties that matter when the question is opinion rather than autocomplete:

• Weight control. You pick the base model and the quant. Abliterated or lightly aligned checkpoints stay available in the open ecosystem even when labs push harder safety defaults on their APIs.
• Offline and private. Drafts, notes, and agent memory never leave the machine. That is useful for politics. It is also useful for client architecture notes you should not paste into a SaaS chat.
• Stable behavior. The binary and the GGUF on disk do not silently change between Tuesday and Thursday because a lab shipped a new system prompt.

So yes, local LLMs have a future even if frontier APIs keep winning on raw MMLU. The product is not “smartest chat.” The product is a second opinion you can actually own.

It is already easy. Memory is what hurts.

Running a small model with llama.cpp is not hard anymore:

• build or grab a release binary,
• download a GGUF (Q4_K_M or Q5_K_M is the usual sweet spot),
• start llama-server or llama-cli,
• point your agent or editor at localhost.

A 7B to 14B class model on a mid laptop is enough for drafting, rewriting, and critique. Quality is not the main wall for that class of work. RAM and VRAM are.

Weights are only part of the bill. Long agent sessions burn KV cache. Context length is the feature you want for real writing and tool loops, and it is exactly what multiplies memory. That is why a “small” model still feels tight once you stack system prompts, retrieved notes, and multi-turn agent state.

Put a number on it. An 8B model with grouped-query attention runs about 1 GB of KV cache per 8K tokens of context at fp16, on top of roughly 5 GB of Q4 weights. Push that to a 32K-token agent session and the cache alone climbs past 4 GB, often more than the weights themselves. That gap is exactly what KV compression is trying to close.

What I actually run today

Concretely, two GGUFs cover most of it. For hardware with nothing to spare: dolphin-x1-nano-Q6_K.gguf, small, fast, no refusal theater. For sessions where I want more judgment quality than a nano model gives me: Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-AMD-MTP-IQ4_XS.gguf, a 27B-class merge tuned to drop refusals, quantized down to IQ4_XS so it still fits on a single consumer GPU or a unified-memory Mac. Yes, that second filename is exactly what franken-merge naming looks like after a model has been through several rounds of community tuning. Neither needs a data center for short-context drafting and critique.

If you want to see the gap yourself rather than take my word for it: pick a policy trade-off question sitting on the axis where Release 01 found the strongest lean, something like scoring the costs and benefits of a carbon tax on heavy industry, and run it through your local model and a frontier API side by side. The Neutrality Project’s benchmark is open and reproducible, so nothing stops you from running your own stack through it and comparing notes. I would be curious what routing rule you land on once you have.

TurboQuant and the missing piece in llama.cpp

TurboQuant (Google Research, ICLR 2026) is online vector quantization aimed at near-optimal distortion for things like KV cache compression: keep more context in the same memory envelope with less quality loss than naive low-bit tricks. A CPU implementation landed as a real llama.cpp pull request (#21089): two new types, tbq3_0 at about 3 bits per element for roughly 5.2x compression versus fp16, and tbq4_0 at about 4 bits for 3.9x, matching q4_0 perplexity in early testing. On June 2, 2026 the maintainer closed it unmerged, citing insufficient evidence of a gain over quant types already in the repo. vLLM already ships an equivalent path, and community forks, ik_llama.cpp among them, keep working CUDA, ROCm and Metal kernels alive outside mainline. Treat “when TurboQuant lands in llama.cpp” as a near-term infrastructure bet that just got a bit more contested, not a shipping checkbox.

When that class of KV compression is boring and local, the practical setup I want is simple:

• Writing, critique, political or cultural judgment: local uncensored (or lightly aligned) model on llama.cpp. No lab PR layer in the loop.
• Code, infra, math, tool-heavy agent work: best-in-class public models. If they lean progressive on culture war prompts, I do not care while they are renaming Kafka topics or reviewing a Ceph upgrade plan.

That split is already how multi-model agents should work, and the plumbing for it exists on both sides. On the cloud side, a proxy like OpenRouter puts every major provider behind one API key and one bill, so switching from one frontier model to another is a config change, not a new account and a new card on file with each lab. On the local side, llama-server is already an OpenAI-compatible endpoint, so swapping the GGUF it points at is the same one-line change: no new server, no new client code. Hermes Agent and OpenClaw sit on top of both and do the part that actually matters: they let one human prompt fan out across several models on the same board, instead of you copy-pasting between chat windows. Today the local leg is memory-constrained. Better KV quant shrinks that gap without forcing every task onto a 70B that does not fit.

Side project stack, production habit

This reads like a weekend setup: a GGUF on a home box, an agent board, a local endpoint for “say what you actually think” drafts. It is also the same discipline I want on serious AI platforms.

On a data or AI platform audit I care about three boring questions: what is the default path, where does state live, and what happens when the vendor moves the goalposts. Swap “S3 SDK” for “chat API” and the questions stay the same. If every reasoning step goes through one external model family, you have a single point of framing failure the same way a single overloaded metadata path is a single point of performance failure.

Routing by task is not ideology. It is capacity planning for cognition. Put uncensored local capacity where judgment matters. Put frontier cloud capacity where raw skill and tools matter. Measure both. That is the small bridge between the chill local-LLM hobby and the pro big-data work: own the critical path, size the bottleneck (here, memory), and do not pretend a convenient default is a neutral one.

Practical takeaway

• Keep using frontier APIs for code and heavy tools. They earn their invoice there.
• Do not outsource contested judgment to the same stack. The Neutrality Project chart is your prior.
• Run a small local model with llama.cpp now if you only need short-context drafting. It already works.
• Watch KV compression (TurboQuant and relatives) landing in local runtimes. That is what turns “cute laptop demo” into “default writing worker in the agent mesh.”
• Configure agents (Hermes, OpenClaw, or your own router) with an explicit policy: local for writing voice and political sensitivity, cloud for everything else.

Local models are not a nostalgia project. They are how you keep a second opinion that no lab can silently edit. Election season just makes the need harder to ignore.

Local models are not a nostalgia project. They are how you keep a second opinion that no lab can silently edit.

What model, or routing rule, do you currently reach for when the task is judgment rather than code? I’m curious whether other people running local stacks have landed on the same split.

Further reading

• The Neutrality Project and the benchmark results
• llama.cpp
• TurboQuant paper (arXiv)
• Hermes Agent
• On this blog: Model fusion beats the frontier model and distilling frontier reasoning into a local model

If you are sizing an AI platform and the hard part is not the demo but memory, routing, and failure modes under real load, that is the work I do in data platform performance audits. Local LLM stacks have the same capacity questions as any other inference tier.