Why run a model locally in 2026?

Three hard reasons have converged. Cost: cloud API subscriptions add up fast — ChatGPT Plus costs 20 USD/month (≈18 EUR), while a local model runs on electricity you already pay for. Privacy: under the GDPR, European businesses and professionals can avoid the legal headache of sending customer data, internal documents, or unreviewed code to US servers by keeping inference entirely on‑device. Capability: models you can run on a single consumer GPU have crossed a threshold — they handle code generation, document summarisation, translation into European languages, and even agentic tool use without breaking a sweat.

The headline numbers say it: open‑weight models like Mistral Small 4 (French, released March 2026), Qwen3.5 2B, the DeepSeek‑R1‑0528 distilled 8B, and Google's Gemma 4 20B all run comfortably on 8–16 GB of VRAM with 4‑bit quantisation — and you can try them tonight without spending a cent.

What hardware do you actually need?

The single number that matters is VRAM (video memory). Rule of thumb for 4‑bit quantised models in 2026:

• 8 GB VRAM (RTX 4060 Ti, RTX 5060, Radeon RX 7600 XT, Apple M4 with 16 GB unified memory): runs 7B–12B models at 30–50 tokens/sec. Good enough for chat, summarisation, basic coding.
• 16 GB VRAM (RTX 5060 Ti, RTX 5070, Radeon RX 7800 XT, Apple M4 Pro): runs 20B–32B models at 15–30 tokens/sec. This is the sweet spot — you get Mistral Small 4 or Gemma 4 20B with room for a 32K context window.
• 24 GB+ VRAM (RTX 5090, Apple M4 Max/Ultra): 70B models, speculative decoding, large batch sizes. Overkill for most solo users but relevant if you run a local agent with MCP tool calling.

Some quick maths for the European buyer. An NVIDIA RTX 5060 Ti 16 GB retails around 410 EUR (455 USD) in the Czech Republic, while an AMD Radeon RX 7800 XT 16 GB goes for roughly 380 EUR. Both run local LLMs well — but NVIDIA still has the edge in software compatibility (CUDA). On our own AI Arena test rig — an RTX 5060 Ti 16 GB — we measure Mistral Small 4 at 4‑bit delivering 44 tokens/sec with a time‑to‑first‑token of 280 ms, and Gemma 4 20B at 22 tokens/sec with a 410 ms TTFT. Both are perfectly usable for real‑time chat.

System RAM matters too if you offload layers to CPU (llama.cpp's GPU‑offload split). Aim for at least 32 GB — DDR5 kits are cheap in 2026, and the extra headroom saves you from OOM crashes when the context grows.

Two tools that make it dead simple

Ollama — the terminal way

Ollama remains the zero‑friction option. One command serves a model over a local OpenAI‑compatible REST API:

ollama run mistral-small:latest

From there, any tool that speaks the OpenAI API — VS Code, Continue.dev, opencode, LangChain — can use your local model. Ollama handles quantisation, model pulling, and GPU detection automatically. It works on Linux, macOS (Apple Silicon), and Windows, and the model catalogue includes everything from Mistral's French‑developed models to DeepSeek's reasoning distillations.

LM Studio — the GUI power option

LM Studio has evolved dramatically through 2026. Version 0.4.0 (January 2026) added server deployment with parallel requests and continuous batching. In July 2026 it shipped Bionic — a built‑in AI agent for open models that can browse the web, run code, and work with files. It also supports the Model Context Protocol (MCP), speculative decoding, and the MLX engine for blistering performance on Apple Silicon Macs. For users who prefer clicking to typing, LM Studio is the most polished local‑LLM experience on the market — and it is free for commercial use.

Which models should a European user pick?

Not all models speak European languages well or respect GDPR‑friendly workflows. Here is a curated shortlist for mid‑2026:

Mistral Small 4 deserves a special mention for Europeans: developed by a Paris‑based company subject to EU regulation, it handles French, German, Spanish, and Italian natively, and its open‑weights licence (Apache 2.0 for non‑frontier models) means you can run it in production inside your firewall without negotiating a contract. That is the sort of sovereignty Brussels keeps asking for — and it already works.

Real electricity cost: what does it actually add to your bill?

This is where a local LLM costs more than zero — but far less than you might think. In the Czech Republic, the residential electricity price in mid‑2026 hovers around 0.14 EUR/kWh. A desktop with an RTX 5060 Ti draws roughly 320 W at full inference load. Running the GPU continuously for one hour consumes 0.32 kWh, costing approximately 0.045 EUR. For a typical user generating 2–3 hours of active inference per day, the monthly power bill for the LLM comes to roughly 4 EUR — compared to a ChatGPT Plus subscription at 18 EUR/month, the local rig pays for its own electricity many times over. Even in Germany (≈0.32 EUR/kWh), the monthly cost would be around 9 EUR — still less than a cloud subscription.

GDPR and data residency: what changes when you stay local

When you use ChatGPT, Gemini, or Claude, your prompts and uploaded files leave the EU — they land on servers in the United States, and even with Data Processing Agreements in place, the legal comfort is shaky. When you run a model locally, no data leaves your machine. That means:

• You do not need a Data Processing Agreement (DPA) with a third party.
• You do not need to perform a Transfer Impact Assessment (TIA) for cross‑border data flows under the AI Act's transparency obligations.
• You can process employee records, customer data, unreleased code, or medical information without worrying about the next Schrems ruling or US executive order.

For European SMEs, schools, and public institutions, this is not a luxury — it is the path of least legal resistance. The EU AI Act, which entered full application in August 2026, classifies general‑purpose AI used internally as low‑risk, but the moment you pipe data to a third‑party cloud provider, the transparency and documentation obligations escalate.

Getting started: a 15‑minute checklist

1. Download Ollama or LM Studio — both are free, both work on Windows, macOS, and Linux.
2. Pick a model — start with Mistral Small 4 (22B, 4‑bit ≈12 GB VRAM) or Qwen3.5 2B (if RAM/VRAM is tight).
3. Connect your tools — VS Code with Continue.dev, opencode on the terminal, or any chat UI that speaks the OpenAI API at http://localhost:11434 (Ollama) or http://localhost:1234 (LM Studio).
4. Test and tune — adjust context length (32K is a good default), try speculative decoding for a speed boost, and enable Flash Attention if your GPU supports it.
5. Optional: expose to LAN — both tools let you serve the model on your local network, so a single powerful PC can serve your whole household or small office.

What are the real limitations?

Local models are not a drop‑in replacement for GPT‑5 or Claude 4.0 — yet. They are weaker at very long reasoning chains, occasionally struggle with nuanced multilingual tasks (Czech, Polish, and Hungarian still lag behind French and German), and you need to manage your own updates when new models drop. Internet‑connected search, when you want it, requires an additional retrieval pipeline. That said, for 80 % of what people actually do with AI assistants — writing, summarising, translating, coding, explaining — a local 20B model in mid‑2026 is genuinely hard to distinguish from a cloud model costing 20 EUR/month.