How to Fine-Tune and Run LLMs Locally with Ollama in 2026

Why Run an LLM Locally

Local inference solves problems hosted APIs structurally can’t: data never leaves your machine, there’s no per-token bill at scale, and the model version you tested against is the model version you’ll be running next month, regardless of any upstream deprecation schedule. The tradeoff is quality and convenience — open-weight models in the 7B-70B range are good, but they generally trail the frontier hosted models on complex reasoning tasks.

Installing Ollama

Ollama ships native installers for macOS, Windows, and Linux, and wraps llama.cpp under the hood for efficient quantized inference.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download the installer from ollama.com/download

# Verify install
ollama --version

Hardware and Model Size

Model size in parameters roughly determines memory requirements once quantized. As a rule of thumb for 4-bit quantization (the Ollama default for most pulls):

• 3B-4B models — run smoothly on 8GB RAM, including most modern laptops with no discrete GPU.
• 7B-8B models — need roughly 8-12GB; a comfortable sweet spot for general-purpose chat and coding help.
• 13B-14B models — need 16-24GB for smooth performance.
• 70B-class models — need 48GB+ VRAM for good speed, or will run slowly split across system RAM and CPU.

Apple Silicon Macs punch above their weight here because unified memory lets the GPU access the full system RAM pool, making them surprisingly capable for 13B-34B models.

Your First Local Model

# Pull and run a general-purpose model
ollama run llama3.1:8b

# Pull a coding-focused model
ollama run qwen2.5-coder:7b

# List installed models
ollama list

# Remove a model to free disk space
ollama rm llama3.1:8b

The first run command downloads the model (several gigabytes) and drops you into an interactive chat prompt. Models are cached locally, so subsequent runs start instantly.

Customizing Behavior with a Modelfile

A Modelfile lets you layer a custom system prompt, temperature, and other parameters on top of a base model without touching its weights — conceptually similar to a Dockerfile for LLM configuration.

FROM qwen2.5-coder:7b

PARAMETER temperature 0.2
PARAMETER num_ctx 8192

SYSTEM """
You are a senior backend engineer. Answer concisely with working code.
Prefer Python and explain tradeoffs only when explicitly asked.
"""

ollama create backend-helper -f ./Modelfile
ollama run backend-helper

Calling Ollama from Python

Ollama exposes a local REST API on port 11434, with an official Python client on top of it — useful for wiring a local model into a script or backend service.

pip install ollama

import ollama

response = ollama.chat(model="qwen2.5-coder:7b", messages=[
    {"role": "user", "content": "Write a Python function that flattens a nested list."}
])
print(response["message"]["content"])

# Streaming
for chunk in ollama.chat(model="llama3.1:8b", messages=[
    {"role": "user", "content": "Explain TCP vs UDP in two sentences."}
], stream=True):
    print(chunk["message"]["content"], end="", flush=True)

Fine-Tuning with LoRA

Ollama itself is an inference runtime, not a training tool — for fine-tuning, you train a LoRA adapter with a library like Hugging Face’s peft or Unsloth, then either merge the adapter into the base weights or convert it to GGUF and load it through a Modelfile’s ADAPTER directive.

pip install unsloth peft transformers datasets trl

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=16,
    lora_dropout=0,
)

dataset = load_dataset("json", data_files="my_training_data.jsonl", split="train")

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=3,
        learning_rate=2e-4,
        output_dir="outputs",
    ),
)
trainer.train()
model.save_pretrained_gguf("my-finetuned-model", tokenizer, quantization_method="q4_k_m")

The 4-bit QLoRA approach (loading the base model in 4-bit, training only the small LoRA adapter matrices) is what makes this feasible on a single consumer GPU with 12-24GB VRAM for 7B-8B models. Once exported to GGUF, point a Modelfile’s FROM at the local file and load it into Ollama like any other model.

When Local Beats a Hosted API

Local models make the most sense for narrow, repetitive tasks where you’ve fine-tuned or prompted a smaller model to match a hosted model’s quality on that one task — classification, structured extraction, internal tooling with sensitive data, or offline/edge deployments. For open-ended reasoning, complex multi-step tool use, or anything where output quality directly affects the product, a frontier hosted model like Claude still wins; see our Claude API tutorial if you’re weighing that route for the same project.