GuideAdvanced
AI Gateways and Model Routing
When a company has several apps, several teams and more than one model provider, calling each API directly stops working: keys get copied everywhere, nobody knows which team spends what, every app reinvents its own retries, and a provider outage becomes a product outage. An AI gateway is the layer that brings order in the middle: one API for every app and one door to the providers, where routing, fallbacks, budgets, limits, caching, guardrails and observability live. You learn it by building the gateway of **Ondaria**, a fictional payments platform for small businesses with three teams that use LLMs: a live support chat, a nightly summary batch and a team that tries new models. First you measure how Ondaria behaves without a gateway during a simulated outage. Then you build **minigate**, your own gateway written with Python's standard library only, module by module: a proxy that translates between the OpenAI API and the Anthropic API, routing by cost with a quality floor measured by evals, by quality and by latency, retries with backoff and a retry budget, fallbacks and circuit breakers, virtual keys with per-team budgets, rate limits in tokens per minute, a cache with a per-team namespace, input and output guardrails, and a request log with metrics in the Prometheus format. In every module you set up the same behavior with **LiteLLM proxy** (v1.106.0), so you know what an open source tool does inside and can decide with judgment between building and adopting. Everything runs on your laptop with Docker and two simulators that imitate the Anthropic and OpenAI APIs, with their formats, their errors and their streams. You need no accounts and no API keys. An optional `--live` mode points minigate at the real providers with your own keys. Prices and models as of October 2026.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Free
- access
Outcomes
What you'll be able to do
- Explain what an AI gateway does (data plane and control plane), what it costs, and when you do not need one
- Build a proxy that receives Chat Completions and translates it to Anthropic Messages, with normalized errors and streaming
- Route requests by cost with a quality floor measured by evals, by quality, and by observed latency
- Retry with exponential backoff, jitter and Retry-After under a retry budget, fall back to another provider, and cool down failing deployments
- Issue virtual keys per team and app, keep a spend ledger with the real cost, and enforce budgets
- Limit by tokens per minute with a token bucket that reserves and reconciles, and return 429 errors a client can act on
- Add an exact-match cache with a per-team namespace, and input and output guardrails that also work on streams
- Observe the gateway with one log line per request, Prometheus metrics and W3C traces, and run it in production
- Set up the same behavior with LiteLLM proxy and decide between building, adopting or buying a managed service
Before you start
What you need to bring
It's for you if...
- Platform and backend engineers who support several LLM apps in the same company
- AI engineers who already call a model from code and now have to answer for cost, availability and control
- Tech leads who have to decide whether to build a gateway, adopt an open source one, or buy a managed one
Requirements and materials
- Having called an LLM API from code: tokens, system prompt, `max_tokens`, the `usage` object (covered by the Building with LLMs in Code guide)
- Intermediate Python: functions, classes, `dataclass`, dictionaries, `json`
- Basic HTTP (methods, headers, status codes) and `curl`
- Docker and Docker Compose at the level of `up`, `logs`, `restart` and `down`
- Python 3.10 or later for the client scripts. No API key needed
Content
The syllabus, module by module
Open any of them to see its lessons.
- 1. Introduction: policies that do not depend on each app
- 2. What to cache in the gateway
- 3. Cache keys and namespaces
- 4. TTL, invalidation and cache-control
- 5. Guardrails that belong in the gateway
- 6. Masking and blocking in practice
- 7. Caching and guardrails in LiteLLM
- 8. Project: Ondaria's cache and policy
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!