Ollama Tutorial: Run Your First Local LLM in Python
Originally published at https://practicaldevblog.blogspot.com/2026/11/ollama-tutorial-run-local-llm.html
Quick answer
Ollama is a free program that runs open language models on your own computer, no API key or cloud account needed. You install it, download a model with one command, and talk to it from a terminal or your own code through a local HTTP API on port 11434. This post installs Ollama on Linux, pulls a small 352 MB model, and sorts four real support tickets from Python.
Picture a small software company's support inbox on a Monday morning. Four tickets waited overnight: a plan that never activated after payment, an app crashing on the settings page, a misspelled name on an invoice, and a whole team locked out since breakfast. Sorting them by hand takes a morning; a cloud AI means uploading customer data to someone else's server and paying per message. A model on your own machine changes that trade: it sorts the tickets and sends nothing outside your computer. This post shows that setup working, with real timings.
What I would check before using it
My first check is whether the model fits the machine. A model file has to sit in memory with room to spare for the conversation. The model here, qwen2:0.5b, is a 352 MB download that runs on an ordinary laptop CPU. A 7B model needs around 8 GB of free RAM; a 13B model needs about 16 GB. Check free memory first, or the first run will crawl.
My second check is what the model is for. A small local model handles sorting, summarising, and drafting, but it is not as careful as the big cloud models. The demo below shows it: the 0.5B model called every ticket high urgency. For careful reasoning or long documents, a cloud model wins; local wins only when privacy matters more than polish.
My third check is whether the task repeats. Ollama earns its keep when the same job runs again and again: triage, summaries, drafted replies. One question a week does not justify an install; a repeated job makes the setup pay for itself.
My fourth check is how the model will be called. Ollama serves a local API, so a script, a service, or a tool like LangChain can call it by changing one URL. Keep the port on localhost, 127.0.0.1, so only your own machine can reach it.
My verdict, as judgement: use Ollama when the data should stay on your machine, the job repeats, and a small model fits your hardware. Start with the smallest model that works, and measure before sizing up. The recommendation flips the moment you need top-tier reasoning, long context, or a model bigger than your RAM.
Four tickets, one local model
The example sorts four support messages. Each ticket gets an urgency, a topic, and one short action, in one line. The script sends each ticket to qwen2:0.5b through the Ollama chat API, with temperature 0 and a 1024-token context. The first call deliberately asks for llama3.2, a model that was never pulled, to show the error. Here is the script, exactly as it ran:
"""Classify four support tickets with a local Ollama model.
Sends each ticket to qwen2:0.5b through the Ollama HTTP API on
http://localhost:11434 and prints the model's one-line verdict plus
timing. The first call deliberately uses a model name that has not
been pulled, to show the error Ollama returns in that case.
"""
import json
import time
import requests
API_URL = "http://localhost:11434/api/chat"
TICKETS = [
{
"id": "SUP-201",
"text": "I paid for the yearly plan yesterday but my account still shows the free plan.",
},
{
"id": "SUP-202",
"text": "The app crashes every time I open the settings page on my phone.",
},
{
"id": "SUP-203",
"text": "How do I change the name on my invoice? I typed it wrong when I signed up.",
},
{
"id": "SUP-204",
"text": "Our whole team cannot log in since this morning, we keep getting a server error.",
},
]
SYSTEM_PROMPT = (
"You triage support tickets for a small software company. "
"Reply in exactly one line: URGENCY: <low, medium, or high> | "
"TOPIC: <billing, technical, or account> | "
"ACTION: <one short action, under 10 words>."
)
def classify(model, ticket):
"""Send one ticket to the model, return the reply text and timings."""
payload = {
"model": model,
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Ticket: " + ticket["text"]},
],
"stream": False,
"options": {"temperature": 0, "num_ctx": 1024},
}
started = time.time()
response = requests.post(API_URL, json=payload, timeout=300)
wall = time.time() - started
if response.status_code != 200:
return None, response.status_code, response.text.strip(), wall
data = response.json()
return data["message"]["content"].strip(), response.status_code, data, wall
def main():
print("Step 1: ask for a model that was never pulled")
reply, status, detail, wall = classify("llama3.2", TICKETS[0])
print(f"HTTP status: {status}")
print(f"Response: {detail}")
print()
print("Step 2: same tickets, with the model that is actually on disk")
for ticket in TICKETS:
reply, status, data, wall = classify("qwen2:0.5b", ticket)
tokens = data.get("eval_count", 0)
eval_seconds = data.get("eval_duration", 0) / 1e9
rate = tokens / eval_seconds if eval_seconds else 0
print(f"{ticket['id']}: {reply}")
print(
f" {tokens} tokens in {eval_seconds:.1f}s "
f"({rate:.1f} tokens/sec), wall time {wall:.1f}s"
)
if __name__ == "__main__":
main()
Here is the real output from the run:
Step 1: ask for a model that was never pulled
HTTP status: 404
Response: {"error":"model 'llama3.2' not found"}
Step 2: same tickets, with the model that is actually on disk
SUP-201: URGENCY: High | TOPIC: Billing | ACTION: Contact customer support to resolve the issue.
21 tokens in 2.0s (10.4 tokens/sec), wall time 6.0s
SUP-202: URGENCY: High | TOPIC: App crash | ACTION: Close ticket with customer support.
20 tokens in 0.8s (25.7 tokens/sec), wall time 1.0s
SUP-203: URGENCY: High | TOPIC: Billing | ACTION: Contact customer support to resolve the issue.
21 tokens in 0.7s (29.6 tokens/sec), wall time 0.9s
SUP-204: URGENCY: High
TOPIC: Server Error
ACTION: Contact IT Support immediately.
19 tokens in 0.8s (24.1 tokens/sec), wall time 1.1s
The thing that went wrong is in step 1: the request asked for llama3.2, which was never downloaded, and the API answered HTTP 404 with {"error":"model 'llama3.2' not found"}. The fix is to check what is on disk first with ollama list, then pull the missing model. The first qwen2:0.5b call took 6.0 seconds of wall time against 2.0 seconds of generation, because the model was still loading; later calls took about a second each, at 24 to 30 tokens per second.
The verdicts teach a second lesson. The model called every ticket high urgency, invented a topic called "App crash" that was not in the allowed list, and split the last answer across three lines. A 0.5B model is a toy for learning the wiring, not a production classifier. Treat its output as a draft and check it with rules, the way the Pydantic post on this blog validates data first.
Common mistakes and how to fix them
First, asking for a model that was never pulled. The symptom is HTTP 404 with {"error":"model 'llama3.2' not found"}. The fix is to run ollama list, then ollama pull <model> before you call it, using the exact tag you pulled.
Second, calling the API before the server is running. The symptom in Python is a ConnectionError ending in [Errno 111] Connection refused. The fix is to start the server with ollama serve in a separate terminal, then check http://localhost:11434/api/tags first.
Third, trusting a tiny model's verdicts. The symptom is confident-looking output that is wrong on the details: everything high urgency, invented categories, broken formatting. The fix is to validate the reply against the allowed values in your own code, and test on ten known tickets first.
Fourth, pulling a model bigger than the machine. The symptom is a download that succeeds and a run that crawls or fails to load. The fix is to check the model size on the library page first and match it to free RAM, starting small.
Frequently Asked Questions
Do I need a GPU to run Ollama? No. Small models run on an ordinary CPU: qwen2:0.5b generated 24 to 30 tokens per second on a 4-core machine with no GPU.
Do I need the internet after the model is downloaded? No. Once the model file is on disk, Ollama runs fully offline, and the API works with the network off.
Is my data sent anywhere when I run a model locally? No, not for local models. Ollama's FAQ says it does not see your prompts or data when you run locally. Watch out for names ending in :cloud, which run on Ollama's servers instead.
Can I use Ollama with LangChain? Yes. LangChain can point its chat model at the local Ollama API instead of a cloud provider. The LangChain tutorial on this blog shows the chain pattern; swap the model line for a local one.
What to remember
- Ollama runs open models on your machine through a local API on port 11434, with no API key.
- Check
ollama listfirst, or a missing model answers with a 404. - The first call is slow while the model loads; later calls are much faster.
- A tiny model proves the wiring, but validate its verdicts in your own code.
Where this leaves you
You now have a local model answering real tickets from your own code, with timings you measured yourself, and you have seen both failure modes that catch beginners: a missing model and a small model overpromising. Pick one repeated job, a daily inbox sort or a meeting summary, wire it to the local API, and run it for a week. If the small model handles it, you have a private tool that costs nothing per message. If it does not, you will know exactly what a bigger model needs to fix.
Further reading
- Ollama, official site, https://ollama.com
- Ollama on GitHub, https://github.com/ollama/ollama
- Ollama model library, https://ollama.com/library
- LangChain Tutorial: Build Your First LLM App in Python
- PyTorch Explained: Machine Learning You Can Watch Learn
- TensorFlow: Machine Learning, From Tensors to Training
- Docker Explained: Containers Without the Confusion
- Pydantic: Catch Bad Data Before It Reaches Your Database
Waqas Shahid is a software engineer from Karachi with more than 12 years of experience, including 6+ years in engineering leadership. He writes The Practical Developer.
Comments
Join the discussionNo comments yet. Be the first to comment.

