The live session: your first model running on your laptop

This is the full guide to the live session, on a single page. We go in five parts, in this order. Follow them top to bottom: each one uses what you set up in the one before.

Remember the workshop analogy: the cloud is a restaurant (a huge chef, charges per dish, sees your ingredients) and local is a cook in your own kitchen (doesn't charge per dish, nobody sees anything, but it has to fit and it knows less). Every new term is explained when it shows up; if you want the full map, it's in the Workshop vocabulary, in the welcome lesson.

Before we start: your starting point

You need to bring this from the prep. If something is missing, that's exactly what we fix in the first few minutes:

• The ~/local-model-workshop folder with expenses.csv and requirements.txt, and the .venv virtual environment with the openai library.
• LM Studio installed and qwen/qwen3-4b-2507 downloaded.
• (Optional) Your OpenRouter key, for part 3.

Open the folder in VS Code and a terminal inside it. Activate the virtual environment:

Mac and Linux:

# Activate the project's virtual environment
source .venv/bin/activate

Windows (PowerShell):

# Allow the activation script only in this window, and activate the environment
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\.venv\Scripts\Activate.ps1

The terminal line has to start with (.venv).

Part 1: What fits on your laptop

Before writing code, let's understand what happens inside your computer when a model answers. With that, the numbers you'll measure in part 3 will make sense.

In this part you'll understand:

1. What happens when LM Studio "loads" a model.
2. What a token is and how a model generates text.
3. Why speed depends so much on your machine.
4. What context is and why it also takes up memory.

1. Loading a model: from the storeroom to the kitchen

You already know the model is a file of about 2.5 GB on your disk. To answer, all of that file has to be in RAM. That's what "loading" means: copying it from the disk into memory. It takes a few seconds, and while the model is loaded your laptop feels heavier.

There are two ways to load it. You, by hand (with the app's model selector or with lms load), or LM Studio on its own, when the first request arrives for a model that isn't in RAM. LM Studio calls that last one JIT (just in time), and it's on by default. If it's off and you ask a model that isn't loaded for something, it answers with the error No models loaded.

Before loading it, LM Studio can tell you how much memory it will need, without loading anything:

# Estimate how much memory the model would ask for, without loading it
lms load qwen/qwen3-4b-2507 --estimate-only

On a Mac with an Apple chip, the estimate says something like this (Confidence: LOW only means it's an approximate calculation, and GiB is almost the same as GB: 1 GiB is about 1.07 GB):

Model: qwen/qwen3-4b-2507
Estimated GPU Memory:   2.33 GiB
Estimated Total Memory: 2.33 GiB
Confidence: LOW

Now load it for real and turn on the server (in the app: the model selector at the top and the Developer tab; or in the terminal):

# Turn on the LM Studio local server
lms server start

# Load the model into RAM
lms load qwen/qwen3-4b-2507

# Show what's loaded in RAM right now
lms ps

Open your system monitor (Activity Monitor on Mac, Task Manager on Windows) and look at the memory used before and after loading. That jump is the cook walking into your kitchen.

2. What a token is and how a model writes

First, a word you'll hear all through the workshop: the text you send the model (your question, your instruction) is called a prompt. In the analogy, it's the order you give the cook.

A model doesn't read or write single letters, or whole words: it works with tokens, pieces of words. A short, common word is usually one token; a long or unusual word gets split into several.

  "Explain what a CSV file is"

  [Expl][ain][ what][ a][ CSV][ file][ is]     <- about 7 tokens
  (the exact split depends on each model)

And it writes one token at a time. For each new token, it does the math with all of its parameters, picks the most likely next piece, adds it to the text, and starts again:

  "A CSV is"  -> model -> " a"
  "A CSV is a"  -> model -> " file"
  "A CSV is a file"  -> model -> " of"
  ...  and so on, one token per round, until it finishes

In the analogy: the cook serves one bite at a time, and for each bite goes over everything they know.

3. What speed depends on

Since each token requires going through the 2.5 GB of parameters, speed depends above all on how fast your computer can read its memory:

Speed is measured in tokens per second: how many pieces of words the model writes each second. You'll measure it yourself in part 3.

4. Context also takes up memory

The context is all the text the model has in front of it in one request: your instructions, your question, what it already answered. It's its work table: it only reasons with what's on the table. And the table also takes up RAM: a bigger context asks for more memory, on top of the model's.

  RAM used by the loaded model
  +------------------------------+----------------------+
  |  Parameters (about 2.5 GB)   |  Context (the table) |
  |  fixed: always the same      |  grows with the text |
  +------------------------------+----------------------+

For this workshop the texts are short, so the context size LM Studio sets by default is enough.

Check your understanding

1. You send a request to a model that isn't loaded, and the answer takes much longer than the next ones. What happened?

2. Why is a Mac with an Apple chip usually faster than a laptop without a graphics card, with the same amount of RAM?

Summary

• Loading means copying the model from the disk into RAM; while it's loaded, it takes up memory even if you don't talk to it.
• A model writes one token at a time, going through all of its parameters for each one. That's why speed depends on how fast your memory is read.
• The context is the model's work table, and it also takes up RAM.

In the analogy: the cook (the model) waits in the storeroom (the disk). Loading it means bringing it into your kitchen (RAM), where it also needs room for its worktable (the context). It serves one bite (token) at a time, and how fast it serves depends on how nimble your kitchen is.

Part 2: Your code, talking to your laptop

Now, code. By the end of this part you'll have a Python script that sends a question to your local model and shows you the answer. And you'll see why that same script, changing its three configuration lines, would talk to the cloud.

In this part you'll understand:

1. What a compatible API is and why LM Studio imitates it.
2. How a request is built from Python: client, model, and messages.
3. How to read a Python error without panicking.

If a Python term doesn't ring a bell (variable, list, dictionary, function, import), it's explained in the Prepare your laptop lesson, in idea 6.

What the compatible API is and why it matters

Cloud providers accept requests in a very specific format: which address they're sent to, which fields they carry (the model, the messages), and how the answer comes back. That format became a standard, and LM Studio imitates it: its server accepts exactly the same order ticket as a cloud provider.

  The same request (the same order ticket)
  +--------------------------------------------+
  | model: "qwen/qwen3-4b-2507"                |
  | messages: [ {role: user, content: "..."} ] |
  +--------------------------------------------+
            |                         |
            v                         v
  http://localhost:1234/v1    https://openrouter.ai/api/v1
  (your kitchen: LM Studio)   (the restaurant: OpenRouter)

That's why the library is called openai even though we won't use that company: it's the library that knows how to write that order ticket, and it works with any server that understands it.

1. Create chat.py

In VS Code, in the Explorer, create a new file called chat.py inside local-model-workshop, paste this, and save (Cmd + S on Mac, Ctrl + S on Windows and Linux):

# chat.py: sends a question to a model and shows you the answer
import sys

from openai import OpenAI

# Where the request goes: the LM Studio server, on your own laptop
BASE_URL = "http://localhost:1234/v1"
# Which model answers: the one you loaded in LM Studio
MODEL = "qwen/qwen3-4b-2507"
# The key: LM Studio doesn't check it, but the library requires some text
API_KEY = "lm-studio"

# The client knows how to build the request in the format the API understands
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)

# The question comes from what you typed after "python chat.py"
question = " ".join(sys.argv[1:])
# If you didn't type anything, we use an example question
if not question:
    question = "Explain in two sentences what a CSV file is."

# We send the request: which model and which messages make up the conversation
response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": question}],
)

# The answer text comes in the first choice (choices[0])
print(response.choices[0].message.content)

If during the prep curl .../v1/models showed you a different id for your model, change the MODEL text to that id, exactly as it appears.

What each part does

And the Python lines that let the script receive your question:

role: "user" means "the person is saying this". In part 4 you'll see other roles.

2. Run it

python chat.py

Wait for the answer. Now with your own question, in quotes:

python chat.py "Give me three ideas to save money at the grocery store. One line each."

What you might see:

When Python fails, it shows a long block of text (the traceback). It looks scary, but the important part is on the last line: that's where it says what type of error it was and its message. Read that one first.

The real test: disconnect your wifi and run python chat.py again. It answers just the same. Your code, your server, and your model are on the same machine.

3. The same code, pointed at the cloud

For this same script to talk to a model in the cloud, the three configuration lines change, and you have to add import os at the top, because the key is read with os.environ. This is what the diff would look like (- is removed, + is added):

+  import os
-  BASE_URL = "http://localhost:1234/v1"
+  BASE_URL = "https://openrouter.ai/api/v1"
-  MODEL = "qwen/qwen3-4b-2507"
+  MODEL = "google/gemma-4-31b-it:free"
-  API_KEY = "lm-studio"
+  API_KEY = os.environ["OPENROUTER_API_KEY"]

Don't change it yet: in part 3 we'll save both configurations in a separate file so you don't have to edit by hand every time. What matters is the idea: the code that builds the request and reads the answer is identical. The only thing that decides whether your data leaves your laptop is an address.

Check your understanding

1. Why do you need API_KEY = "lm-studio" if LM Studio doesn't check keys?

2. What would you have to change in chat.py for it to talk to the cloud?

Summary

• LM Studio accepts the same order ticket as cloud providers. That's why we use the openai library with base_url pointing to http://localhost:1234/v1.
• You created chat.py: it builds a request with model and messages and shows choices[0].message.content.
• When there's an error, read the last line of the traceback: server off, model not loaded, different id, or environment not activated.
• To go to the cloud, the three configuration lines change (address, model, and key), plus an import os.

In the analogy: your script writes the order ticket (the request in the API's format), and the client, which is the waiter, carries it to the pass of your kitchen (localhost:1234). The restaurant accepts exactly the same order ticket: to order there, you only change the address and show your customer number (the key).

Part 3: Measuring: your laptop against the cloud

"Local models are worse" or "they're just as good" are phrases that are useless without measuring. In this part you send the same requests to your laptop and to a model in the cloud, and you compare time, speed, and correct answers with your own eyes.

In this part you'll understand:

1. How to save two destinations without writing your key in the code.
2. What gets measured and how to measure it fairly.
3. How to read the results.

1. Create providers.py: the two addresses

Create a new file called providers.py in local-model-workshop and paste this:

# providers.py: the two addresses you can send your requests to
import os
import sys

import openai
from openai import OpenAI

PROVIDERS = {
    # LM Studio, on your laptop: nothing goes out to the internet and it needs no key
    "local": {
        "base_url": "http://localhost:1234/v1",
        # LM Studio doesn't check the key, but the library requires some text
        "api_key": "lm-studio",
        "model": "qwen/qwen3-4b-2507",
    },
    # OpenRouter, in the cloud: your key travels with every request
    "cloud": {
        "base_url": "https://openrouter.ai/api/v1",
        # The key is read from an environment variable, never written here
        "api_key": os.environ.get("OPENROUTER_API_KEY", ""),
        "model": "google/gemma-4-31b-it:free",
    },
}


def get_provider(name):
    # If the name isn't "local" or "cloud", we warn you and exit
    if name not in PROVIDERS:
        sys.exit(f"Unknown destination: '{name}'. Use 'local' or 'cloud'.")
    provider = PROVIDERS[name]
    # Without a key, the cloud would answer with an authentication error
    if not provider["api_key"]:
        sys.exit("Your key is missing: set OPENROUTER_API_KEY in this terminal.")
    return provider


def make_client(provider):
    # The same client works for both destinations: only the address changes
    return OpenAI(base_url=provider["base_url"], api_key=provider["api_key"])


def explain_error(error):
    # Nobody is listening at the address: the server is off
    if isinstance(error, openai.APIConnectionError):
        return "No connection. If it's 'local': is the LM Studio server on?"
    # The key isn't valid or was pasted incomplete
    if isinstance(error, openai.AuthenticationError):
        return "The key isn't valid. Check OPENROUTER_API_KEY."
    # Too many requests: the cloud's free plan slowed you down
    if isinstance(error, openai.RateLimitError):
        return "Request limit (rate limit). Wait a bit or switch models."
    # LM Studio answers, but it has no model loaded in memory
    if "No models loaded" in str(error):
        return "LM Studio is on but has no model loaded. Load it and run again."
    # Any other error with a code: we show it as is
    if isinstance(error, openai.APIStatusError):
        return f"Error {error.status_code}: {error.message}"
    # Any other problem: the error text, without the traceback
    return str(error)

This file doesn't run on its own: measure.py and agent.py will use it. It saves in a dictionary (a list of name: value pairs) the two configurations that were loose in chat.py, and it brings three functions:

The new Python in this file:

The cloud model: google/gemma-4-31b-it:free is a free OpenRouter model with 31 billion parameters: about eight times more than yours. List checked on October 14, 2026. If it no longer shows up, change that text to nvidia/nemotron-3-super-120b-a12b:free.

2. Give your key to the terminal, not to the code

providers.py reads the key from an environment variable: a named value that lives in the terminal while it's open, and that the programs you run from there can read. That way the key is never written in a file you could share or upload to the internet by accident.

If you don't have an OpenRouter key, skip this step and the cloud measurement: measure only your laptop.

Mac and Linux:

# Asks for the key without showing it on screen; paste it and press Enter
read -s OPENROUTER_API_KEY
# Makes it available to the programs you run from this terminal
export OPENROUTER_API_KEY

Windows (PowerShell):

# Asks for the key; paste it and press Enter (don't share your screen at this moment)
$env:OPENROUTER_API_KEY = Read-Host "Paste your OpenRouter key"

The variable lasts as long as that terminal is open. If you open another one, repeat this step.

3. Create measure.py: the same requests, measured

Create a new file called measure.py in local-model-workshop and paste this:

# measure.py: runs the same prompts against one destination and measures each answer
import sys
import time

import openai

from providers import explain_error, get_provider, make_client

# Each test brings its prompt and the correct answer, so you can compare
TESTS = [
    # 1. Follow a short instruction and its format
    {
        "name": "classify",
        "prompt": "Classify this expense into a single category: food, transport, home or fun. "
        "Answer with only the category.\nExpense: Taxi home",
        "expected": "transport",
    },
    # 2. Pull data out of a text and return it as JSON
    {
        "name": "extract",
        "prompt": "From the text, extract the category (food, transport, home or fun) and the amount. "
        'Answer with only JSON like this: {"category": "...", "amount": 0.0}\n'
        "Text: Yesterday I paid 35.00 for the internet bill.",
        "expected": '{"category": "home", "amount": 35.0}',
    },
    # 3. Arithmetic with decimals
    {
        "name": "sum",
        "prompt": "Add up these amounts and answer with only the total: 54.20, 6.50, 61.75, 11.90, 58.30",
        "expected": "192.65",
    },
    # 4. Combine two facts and draw a conclusion
    {
        "name": "reason",
        "prompt": "I spent 385.15 this month and my goal was to spend less than 350. "
        "Did I meet the goal? How much did I go over or how much was left? Answer in one sentence.",
        "expected": "You didn't meet it: you went over by 35.15",
    },
    # 5. Summarize without making anything up
    {
        "name": "summarize",
        "prompt": "Summarize in one sentence: I want to know where my money goes every month. "
        "My goal for October is to spend less than 350 in total. "
        "I suspect food is the most expensive, but I haven't added it up.",
        "expected": "(open: you decide if the summary is faithful)",
    },
]


def run_test(client, model, test):
    # We take the time right before sending the request
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": test["prompt"]}],
        # temperature=0: the most likely answer, to compare fairly
        temperature=0,
    )
    # How many seconds passed until we had the complete answer
    seconds = time.perf_counter() - start
    # usage counts the tokens the model wrote (some providers don't send it)
    tokens = response.usage.completion_tokens if response.usage else 0
    # Speed: tokens written per second
    speed = tokens / seconds if seconds > 0 else 0
    # The answer text, without extra spaces at the start and end
    answer = (response.choices[0].message.content or "").strip()
    return seconds, tokens, speed, answer


def main():
    # The first argument picks the destination: local (by default) or cloud
    name = sys.argv[1] if len(sys.argv) > 1 else "local"
    # We take that destination's configuration and create its client
    provider = get_provider(name)
    client = make_client(provider)
    print(f"Destination: {name} · model: {provider['model']}\n")

    # Here we collect each test's result to write the file at the end
    rows = []
    for test in TESTS:
        # If a test fails (server off, rate limit), we explain and stop
        try:
            seconds, tokens, speed, answer = run_test(client, provider["model"], test)
        except openai.OpenAIError as error:
            sys.exit(explain_error(error))
        # We show each test as soon as it finishes, with the expected answer next to it
        print(f"[{test['name']}] {seconds:.1f} s · {tokens} tokens · {speed:.1f} tokens/s")
        print(f"  answer:   {answer}")
        print(f"  expected: {test['expected']}\n")
        rows.append((test["name"], seconds, speed, answer, test["expected"]))

    # We save everything to a file to compare local against cloud afterward
    out_path = f"results-{name}.md"
    with open(out_path, "w", encoding="utf-8") as out:
        out.write(f"# Results: {name} ({provider['model']})\n\n")
        out.write("| test | seconds | tokens/s | answer | expected |\n")
        out.write("| --- | --- | --- | --- | --- |\n")
        for test_name, seconds, speed, answer, expected in rows:
            # Multi-line answers get flattened so they fit in the table
            flat = answer.replace("\n", " ").replace("|", "/")
            out.write(f"| {test_name} | {seconds:.1f} | {speed:.1f} | {flat} | {expected} |\n")
    print(f"Saved to {out_path}")


if __name__ == "__main__":
    main()

What it measures and why this way

The five tests aren't random: each one tests something different, and all of them have an answer you can check (except the summary, which you judge).

Three details in the code that make the comparison fair:

• temperature=0: a model doesn't always pick the most likely token; the temperature controls how much it's allowed to vary. At 0 it almost always picks the most likely one, so if you run the test twice, the answer barely changes. For measuring, that's what you want.
• time.perf_counter(): a high-precision stopwatch. We start it right before the request and stop it when we have the complete answer.
• usage.completion_tokens: the server tells us how many tokens the model wrote. Divided by the seconds, it gives the speed.

And the new Python in measure.py:

4. Run the measurement on your laptop

python measure.py local

You'll see the five tests, one by one, in this shape:

Destination: local · model: qwen/qwen3-4b-2507

[classify] (seconds) s · (tokens) tokens · (speed) tokens/s
  answer:   (what your model answered)
  expected: transport

...

Saved to results-local.md

Look at the first test: if the model wasn't loaded yet, LM Studio loads it with that first request (JIT) and the time shoots up. If that happens to you, run the measurement again: the second round is the one that counts.

5. Run the same measurement in the cloud

If you set your key in step 2:

python measure.py cloud

It's saved to results-cloud.md. If you see Request limit (rate limit), the free model is overloaded: wait a bit, or change the "cloud" model in providers.py to the alternative one and run again.

6. Compare

The results are saved in .md (Markdown) files: plain text with a very simple format, where the lines with | make up a table. Open results-local.md and results-cloud.md in VS Code, side by side, and fill in this table with your numbers:

How to read it:

• Look at correct answers first, then speed. A fast, wrong answer is useless.
• The sum is the trickiest test for any model, and even more so for a small one: the model writes the number that looks right. When we prepared this workshop (October 14, 2026, with temperature=0), Qwen3 4B got this sum wrong in English (292.65, all three times we ran it) and right in Spanish (192.65): the same numbers, another language, another result. If your task depends on exact math, the model shouldn't do it: it should ask your code to do it. That's exactly what we do in part 4.
• Narrow tasks (classifying, extracting to a fixed format) are where a small model usually gets closest to the big one. Check it with your numbers.
• The cloud's speed includes the trip across the internet and the free service's queue. Your laptop's has neither of those, but it depends on your machine.
• Don't trust tokens per second on very short answers. Before writing, the model has to read your whole prompt, and that takes time too. In classify the answer is one or two tokens, so the "speed" comes out very low even when the model is flying. To compare speed, look at the test with the longest answer (summarize).

7. Add your own prompt

The measurement that really helps you is the one for your task. Add a test at the end of the TESTS list in measure.py, before the closing bracket ]:

    # Your own test: a request from your work, with example data
    {
        "name": "my-task",
        "prompt": "Write here the request you'd make at work, with made-up data.",
        "expected": "Write here how you'd know the answer is right",
    },

And run both measurements again:

python measure.py local
python measure.py cloud

Remember: send the cloud only example data. You can send your laptop whatever you want: it doesn't leave your machine.

Check your understanding

1. Why does measure.py use temperature=0?

2. Your laptop was slower on the first test than on the rest. Is that a problem with the model?

3. Why does the key go in an environment variable and not written in providers.py?

Summary

• providers.py saves the two destinations and reads the cloud key from an environment variable.
• measure.py sends five checkable requests to one destination, with temperature=0, and measures seconds, tokens, and tokens per second. It saves everything to results-local.md or results-cloud.md.
• To compare: first correct answers and format, then speed. Arithmetic is the weak spot; narrow tasks are a small model's strength.
• The measurement that matters is the one for your own task: add it to TESTS.

In the analogy: you gave the same five orders to the cook at home and to the restaurant's chef, with a fixed recipe (temperature=0, so the cook doesn't improvise), and you timed the service. Now you know which dishes your cook gets right and which ones not yet.

Part 4: Tools with a small model

In part 3 you saw that a model isn't a calculator. The solution isn't to ask it to add better: it's to give it tools. Let the model decide what data it needs, and let your code calculate it. In this part you give the model on your laptop two tools and see what it can handle and where it breaks.

In this part you'll understand:

1. What a tool is and who actually runs it.
2. What the cycle of a request with tools looks like.
3. Where a small model usually breaks, and how your code protects it.

1. What a tool is

A tool is a function in your code that you describe to the model: its name, what it's for, and what data it receives. The model doesn't run it: the model only produces text. What it does is answer with a structured request like "I want you to run category_total with category = food". Your code reads it, actually runs the function, and gives it back the result.

In the analogy: the cook doesn't have the key to the pantry. They tell you "bring me the total of what you spent on food", you go to the pantry (expenses.csv), do the math, and bring it to them. They decide what to ask for; you do the exact work.

  Your question: "How much did I spend on food?"

  Your code (agent.py)                       Model (in LM Studio)
  |                                                   |
  |-- question + tool menu -------------------------->|
  |<-- "run category_total(category='food')" ---------|  round 1
  |                                                   |
  (your code reads expenses.csv and adds: 192.65)     |
  |                                                   |
  |-- result: {"total": 192.65} --------------------->|
  |<-- "You spent 192.65 on food." -------------------|  round 2

Each outgoing arrow is a complete request to the model. A question with tools takes, at a minimum, two rounds.

2. Create agent.py

Create a new file called agent.py in local-model-workshop and paste this:

# agent.py: the model asks for tools, your code runs them and gives it back the result
import csv
import json
import sys
from pathlib import Path

import openai

from providers import explain_error, get_provider, make_client

# The expenses file lives next to this script
CSV_PATH = Path(__file__).with_name("expenses.csv")
# Round limit: if the model doesn't finish in 8, we cut it off
MAX_ROUNDS = 8


def load_rows():
    # We read the CSV as a list of dictionaries (one row = one dict)
    with open(CSV_PATH, newline="", encoding="utf-8") as file:
        return list(csv.DictReader(file))


def list_categories():
    # We lowercase everything: to the computer "Food" and "food" are different
    categories = {row["category"].strip().lower() for row in load_rows()}
    return {"categories": sorted(categories)}


def category_total(category):
    # We compare in lowercase, same as in list_categories
    wanted = category.strip().lower()
    rows = [row for row in load_rows() if row["category"].strip().lower() == wanted]
    # If the category doesn't exist, we return an error with the valid ones
    if not rows:
        return {
            "error": f"Unknown category '{category}'",
            "valid_categories": list_categories()["categories"],
        }
    # We add up the amounts and round to cents
    total = round(sum(float(row["amount"]) for row in rows), 2)
    return {"category": wanted, "total": total, "rows": len(rows)}


# The menu the model sees: name, what it's for, and which arguments it takes
TOOLS = [
    # Tool 1: receives nothing and returns the categories
    {
        "type": "function",
        "function": {
            "name": "list_categories",
            "description": "List the expense categories that exist in expenses.csv.",
            "parameters": {"type": "object", "properties": {}},
        },
    },
    # Tool 2: receives a category and returns its total
    {
        "type": "function",
        "function": {
            "name": "category_total",
            "description": "Total amount spent in one category of expenses.csv.",
            # Arguments are described with JSON Schema: name, type, and what it's for
            "parameters": {
                "type": "object",
                "properties": {
                    "category": {
                        "type": "string",
                        "description": "One category, as returned by list_categories.",
                    }
                },
                "required": ["category"],
            },
        },
    },
]

# Which Python function runs when the model asks for each tool
FUNCTIONS = {"list_categories": list_categories, "category_total": category_total}

# The rules the model reads before your question
SYSTEM_PROMPT = (
    "You are a personal expense assistant. For any amount, use the tools: "
    "never make up figures. Answer in English and in a few lines."
)


def run_tool(name, raw_arguments):
    # Arguments arrive as JSON text: if they can't be read, it's the model's mistake
    try:
        arguments = json.loads(raw_arguments or "{}")
    except json.JSONDecodeError:
        return {}, {"error": "Arguments are not valid JSON"}
    # The model asked for a tool that isn't on the menu
    if name not in FUNCTIONS:
        return arguments, {"error": f"Unknown tool '{name}'"}
    # We run the function; if the arguments don't fit, we return the error
    try:
        return arguments, FUNCTIONS[name](**arguments)
    except TypeError as error:
        return arguments, {"error": f"Bad arguments: {error}"}


def run_agent(client, model, question):
    # The conversation starts with the rules (system) and your question (user)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question},
    ]
    for round_number in range(1, MAX_ROUNDS + 1):
        # Each round is a new request, with the tool menu
        response = client.chat.completions.create(model=model, messages=messages, tools=TOOLS)
        # The model's message: either a final text, or a list of tool requests
        message = response.choices[0].message
        # If the model didn't ask for tools, that's its final answer
        if not message.tool_calls:
            print(f"[round {round_number}] the model answers")
            return message.content
        # We save what it asked for, so it remembers it in the next round
        messages.append(
            {
                "role": "assistant",
                "content": message.content or "",
                "tool_calls": [call.model_dump() for call in message.tool_calls],
            }
        )
        for call in message.tool_calls:
            # Your code (not the model) runs the tool
            arguments, result = run_tool(call.function.name, call.function.arguments)
            print(f"[round {round_number}] calls {call.function.name}({arguments}) -> {result}")
            # We give it back the result, tied to the id of its request
            messages.append(
                {"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}
            )
    return f"(The agent didn't finish in {MAX_ROUNDS} rounds: it got stuck going in circles.)"


def main():
    # First argument: local or cloud. What follows: your question
    name = sys.argv[1] if len(sys.argv) > 1 else "local"
    question = " ".join(sys.argv[2:]) or "How much did I spend on food in September?"
    # We take that destination's configuration and create its client
    provider = get_provider(name)
    client = make_client(provider)
    # We show which model and which question we'll work with
    print(f"Destination: {name} · model: {provider['model']}")
    print(f"Question: {question}\n")
    # If something fails in the connection, we explain it instead of showing the raw error
    try:
        answer = run_agent(client, provider["model"], question)
    except openai.OpenAIError as error:
        sys.exit(explain_error(error))
    # The model's final answer
    print(f"\n{answer}")


if __name__ == "__main__":
    main()

The pieces, in order

And the new Python in agent.py:

The MAX_ROUNDS limit is your safety net: a confused model can ask for tools over and over without ever answering. Without a limit, your program would never finish. Why 8? Because "how much did I spend in total?" done right already takes 6 rounds: list the categories, ask for the total of each of the four, and answer. A limit of 6 would leave no margin.

3. Ask it, and watch each round

python agent.py local "How much did I spend on food in September?"

Every time the model asks for a tool, your code prints a line. This is what came out when we tried it with Qwen3 4B:

Destination: local · model: qwen/qwen3-4b-2507
Question: How much did I spend on food in September?

[round 1] calls list_categories({}) -> {'categories': ['food', 'fun', 'home', 'transport']}
[round 2] calls category_total({'category': 'food'}) -> {'category': 'food', 'total': 192.65, 'rows': 5}
[round 3] the model answers

You spent $192.65 on food in September.

Notice what it did: first it asked which categories exist, then it asked for the food total. Each line tells you what the model asked for (the name and the arguments) and what your code gave back to it. That way you don't have to take its word for it: you see the exact data it had in front of it. The correct food total is 192.65, with 5 rows (the four food ones and the Food one).

Now, questions that demand more:

python agent.py local "How much did I spend on transport?"
python agent.py local "Which category did I spend the most on, and how much?"
python agent.py local "How much did I spend in total in September?"

The second and third ones need several steps: knowing which categories exist and asking for the total of each one. The correct values:

Look at the third question: there's no "grand total" tool. The model has to ask for the four categories and add them up itself. Compare its answer with 385.15.

4. What happened when we tried it

We ran these questions several times with Qwen3 4B while preparing the workshop. The first two came out fine. The other two failed every time, in two ways worth recognizing.

The "which category did I spend the most on" question took a shortcut. In Spanish it went like this every time:

[round 1] calls list_categories({}) -> {'categories': ['food', 'fun', 'home', 'transport']}
[round 2] calls category_total({'category': 'food'}) -> {'category': 'food', 'total': 192.65, 'rows': 5}
[round 3] the model answers

and it answered that food was the highest, with 192.65. The answer is right, but by luck: it only asked for the food total and assumed it was the biggest, without checking the other three categories. With other data, it would have answered wrong with the same confidence. That's why we print every round: the answer alone doesn't tell you whether the model did the work. (In English, in our run, it did ask for all four totals. Run it yourself and look at the rounds.)

The total question didn't even use tools. This came out every time:

[round 1] the model answers

I cannot determine the total spending for September as no data about monthly expenses is available. Please provide more details or check the dataset for specific time-based spending.

The model gave up. Why? Read the tool descriptions in TOOLS: they talk about "categories" and "expenses.csv", but none of them says which month the expenses are from. The model saw "September" in the question, found nothing about months in the menu, and concluded it didn't have that data. With the same question without "in September", it did ask for the five tools and answered 385.15.

5. Fix it: tell the model what's in the pantry

The model only knows about your tools what their descriptions say. If it's missing a piece of information to decide, it doesn't guess: it gives up or gets it wrong. So you give it that information.

In agent.py, inside TOOLS, change the description of list_categories:

            "description": "List the expense categories in expenses.csv (all expenses of September 2026).",

And the one for category_total:

            "description": "Total spent in one category in September 2026, from expenses.csv.",

Save and ask again:

python agent.py local "How much did I spend in total in September?"

With that change, when we tried it, the model asked for all five tools and answered In September 2026, you spent a total of $385.15 on food, fun, home, and transport. Same model, same question: the only thing that changed was one sentence in the menu.

Watch out for what it didn't fix: in Spanish, the "which category did I spend the most on" question kept taking the shortcut. That isn't missing information, it's the model not being careful, and that's where a small model has less room than a big one. The real fix is the same as in part 3: give it a tool that does the exact work (for example, one that returns all the totals sorted), instead of hoping the model gets it right on its own.

6. Where a small model breaks

A model with 4 billion parameters can use tools, but it has less room for error than a big one. Along with what you've already seen, this is what you might run into, and what your code does in each case:

Notice the pattern: tools with complete descriptions and clear errors help the model decide and correct itself. category_total doesn't just say "error": it says which categories are valid. And one sentence about the month in the description turned an "I cannot" into the right answer. That's designing tools with a small model in mind.

7. The same question, in the cloud

If you have your key set, run the same questions against the big model:

python agent.py cloud "Which category did I spend the most on, and how much?"
python agent.py cloud "How much did I spend in total in September?"

Compare: how many rounds did each one need? Did either one ask for a tool wrong? Did either one add wrong? That difference, measured with your questions, is what tells you whether a task with tools can go local today.

Watch out for privacy: in the cloud, each round sends the provider your question and the tool results. If expenses.csv had your real expenses, they would go in the request. Locally, nothing leaves your laptop.

Check your understanding

1. When the model "uses" category_total, who reads expenses.csv and does the sum?

2. Why does category_total return the list of valid categories when it can't find one?

3. What is MAX_ROUNDS for?

Summary

• A tool is a function in your code described in a menu (TOOLS). The model asks; your code runs it and gives it back the result with role: "tool".
• Each request with tools takes at least two rounds. agent.py prints each one: what the model asked for and what you gave back to it.
• A small model might not ask for the tool, give up for lack of context, take shortcuts, ask for it with the wrong arguments, write the format wrong, or get stuck going in circles. Complete descriptions, clear errors, and a round limit protect it.
• You saw it live: one sentence in the description ("September 2026") turned an "I cannot" into 385.15.
• Exact math goes in your code, not in the model.

In the analogy: the cook doesn't have the key to the pantry (expenses.csv): they ask for what they need and you bring it, with the exact math done. But they only ask for what they know exists: if the tool menu doesn't say the pantry is from September, they think there's nothing from September. And sometimes, in a hurry, they check one shelf and not all four.

Part 5: When to go local

You now have your own data: how long your laptop takes, what it gets right and what it gets wrong, and how well it uses tools. Now we put them on a scale. The question isn't "local or cloud?" in general: it's "local or cloud for this task?".

In this part you'll understand:

1. Which four questions decide whether a task goes local or to the cloud.
2. How to turn your measurement into a decision.
3. Why the answer changes over time.

The four questions

And one that gets forgotten: does it have to work without internet? If so, only local.

The rule of thumb

  Can the data leave your machine?
          |
     no --+--> LOCAL (and design the task so a small model can handle it)
          |
     yes  v
  Is it a narrow task (classify, extract, short summary)
   and does your measurement say the local model gets it right?
          |
     yes -+--> LOCAL if volume is high; either one if it's low
          |
     no   v
  CLOUD (long reasoning, lots of context, tool chains)

Three ideas to take with you:

1. Measure with your task, not with anyone's opinion. What you did with measure.py is worth more than any table on the internet, because it's your request, on your machine.
2. Shrink the task so it fits the model. A small model performs well on narrow tasks with a clear format. Anything that's exact calculation or data lookup, give it as a tool.
3. It's not forever. Small models improve fast. What a 4B can't handle today, maybe it can in a few months: run your measurement again when a new one comes out. You only change the MODEL in providers.py.

Before you go: free up the memory

The model is still loaded in your RAM. If you're not going to keep practicing, take it out and turn off the server:

# Remove all loaded models from RAM
lms unload --all

# Turn off the local server
lms server stop

Or in the app: Eject next to the model and, in Developer, turn off the server. The model stays on your disk for next time.

Check your understanding

1. A clinic wants to classify patient notes into five categories, thousands of times a day. Local or cloud?

2. You want an assistant that reads long contracts and reasons about several clauses at once. The contracts aren't confidential. Local or cloud?

Summary

• The decision is per task: privacy and volume push toward local; long reasoning, a lot of context, and slow hardware push toward the cloud. Without internet, only local.
• Measure with your own task, shrink it so it fits in a small model, and put exact math in tools.
• When you finish, free up the memory: lms unload --all and lms server stop.
• To keep going, head to the After module: the NIEVA ecosystems at your own pace, or the bootcamps if you want something guided.

In the analogy: the restaurant is still better for the hard dishes. But for the simple orders that repeat a thousand times, or with ingredients you don't want anyone to see, your kitchen wins. And when you finish, send the cook back to the storeroom: otherwise it keeps taking up your kitchen.

Resources

• LM Studio: developer documentation
• LM Studio: OpenAI-compatible API
• LM Studio: tool use
• LM Studio: the lms command
• OpenRouter: free models