GuideIntermediate

Local Models for Engineers: From GGUF to Production

You already know how to start a model on your machine and call it through an API. This guide starts there and gives you what is missing to use it for real: the technical why and the engineering cases. You will understand what takes up memory when a model runs (the weights, the quantization, the KV cache that grows with every word of the conversation) and what limits its speed, so you can calculate, before downloading anything, whether a model fits your hardware and how fast it will go. You will choose a model with your own speed and quality measurements, not rankings; pick between llama.cpp, Ollama (with its MLX engine on Apple Silicon), headless LM Studio, mlx-lm, and vLLM depending on the case; and write a single client that works the same against a local model, a cloud model, or both with a fallback, with a lock that prevents by design any sensitive data from leaving for the internet. Then come the cases that fail most often locally: structured output and tool calling, with their real, measured limits; a RAG that works with the network off, with evaluated retrieval and verifiable citations; and coding agents with a local model, with the configuration that makes them usable and a way to measure them on your own repository. The guide closes with verified privacy (what leaves the machine and how to check it), the licenses of open models, and the final project: a private document assistant packaged in Docker that answers with citations and has no way out to the internet. Everything happens in one case: Altamira Clinic, a network of three clinics that needs an assistant for its staff, with one non-negotiable condition: no document leaves the building.