How to Fine-Tune LLMs Locally

Fine-tuning a large language model locally used to require a rack of GPUs. In 2026, thanks to quantization and parameter-efficient methods like LoRA, you can fine-tune a 7-8B parameter model on a single consumer GPU with 12-24GB of VRAM — or even on Apple Silicon with unified memory. This guide walks through the full pipeline: preparing data, running LoRA fine-tuning, and merging the result into a model you can actually run.

Table of Contents

• Why Fine-Tune Instead of Prompting
• Hardware and Software Requirements
• Preparing a Training Dataset
• Fine-Tuning with LoRA
• Running the Training Loop
• Merging and Exporting the Model
• Running Your Fine-Tuned Model Locally
• Common Pitfalls
• FAQ

Why Fine-Tune Instead of Prompting

Prompt engineering and RAG solve most problems — try those first. Fine-tuning is worth the extra effort specifically when you need the model to consistently follow a narrow output format, adopt a specific tone across thousands of generations, or perform a task the base model handles inconsistently even with good prompts and examples. It bakes behavior into the weights instead of re-explaining it in every prompt.

Hardware and Software Requirements

For a 7-8B parameter base model with 4-bit quantization and LoRA, 12-16GB of VRAM is realistic on a single GPU. Larger 13-14B models push you toward 24GB cards. Install the core stack:

pip install torch transformers peft bitsandbytes accelerate datasets trl

peft (Parameter-Efficient Fine-Tuning) implements LoRA. bitsandbytes handles 4-bit quantization so the base model fits in memory. trl provides a training loop built for language model fine-tuning specifically.

Preparing a Training Dataset

Format matters more than volume for narrow tasks — 200-500 high-quality examples often outperform 5,000 noisy ones. Use a simple instruction format as JSONL:

{"instruction": "Summarize this support ticket in one sentence.", "input": "Customer reports login failures since the 2.3 update...", "output": "User cannot log in after updating to version 2.3."}
{"instruction": "Summarize this support ticket in one sentence.", "input": "Payment declined for annual plan renewal...", "output": "Annual plan renewal payment was declined."}

Load it with the datasets library:

from datasets import load_dataset

dataset = load_dataset("json", data_files="train.jsonl", split="train")

def format_example(example):
    return {
        "text": f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
    }

dataset = dataset.map(format_example)

Fine-Tuning with LoRA

LoRA freezes the base model’s weights and trains small low-rank adapter matrices injected into attention layers instead. This cuts trainable parameters from billions down to millions, which is why it fits on consumer hardware:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

model_name = "meta-llama/Meta-Llama-3-8B"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config, device_map="auto")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

That last line typically prints something like “trainable params: 0.06% of total” — a concrete reminder of why this fits on a single GPU where full fine-tuning wouldn’t.

Running the Training Loop

trl‘s SFTTrainer wraps Hugging Face’s Trainer with sensible defaults for instruction tuning:

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./lora-output",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    dataset_text_field="text",
)

trainer.train()

Gradient accumulation lets you simulate a larger batch size than your VRAM allows by accumulating gradients over several small batches before updating weights — set per_device_train_batch_size as high as memory allows, then increase gradient_accumulation_steps to reach an effective batch size of 8-16.

Merging and Exporting the Model

After training, merge the LoRA adapter weights back into the base model for a standalone checkpoint you can distribute or convert:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
merged_model = PeftModel.from_pretrained(base_model, "./lora-output/checkpoint-final")
merged_model = merged_model.merge_and_unload()

merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

Running Your Fine-Tuned Model Locally

Convert the merged model to GGUF format for fast local inference with llama.cpp or Ollama:

python convert_hf_to_gguf.py ./merged-model --outfile my-model.gguf --outtype q4_k_m

ollama create my-model -f Modelfile
ollama run my-model

The q4_k_m quantization keeps quality close to the full-precision model while cutting file size roughly 4x, which matters for running the model comfortably on a laptop.

Common Pitfalls

Overfitting on small datasets is the most common failure — if your loss drops close to zero within an epoch, you’re likely memorizing rather than generalizing, and validation loss should be watched separately from training loss. Learning rate is the second most common issue: 2e-4 works for most 7-8B LoRA setups, but going higher on small datasets destabilizes training fast.

Also watch prompt formatting consistency. If your training data uses a specific instruction template, your inference calls must use the exact same template — a mismatched format at inference time silently degrades output quality without any error being thrown.

For more on running models efficiently once fine-tuned, our home lab and DevOps guide covers self-hosting infrastructure, and if you’re building an API around your model, see our FastAPI REST tutorial.

FAQ

How much data do I actually need?
For narrow, well-defined tasks, 200-500 high-quality examples is often enough. Broader behavior changes (tone, multi-task instruction following) benefit from thousands of diverse examples.

Can I fine-tune on a MacBook?
Yes, using MLX or llama.cpp’s training support on Apple Silicon. Unified memory means a 32-64GB MacBook can handle 7-8B LoRA fine-tuning, though slower than a dedicated NVIDIA GPU.

What’s the difference between LoRA and QLoRA?
QLoRA adds 4-bit quantization of the frozen base model on top of LoRA, cutting memory requirements further with minimal quality loss — it’s what makes 8B models trainable on 12-16GB GPUs.

Do I need to fine-tune, or would RAG work?
If the problem is “the model doesn’t know this information,” use RAG. If the problem is “the model knows the information but won’t format or behave the way I need,” fine-tuning is the better tool.

How long does LoRA fine-tuning take?
For 500 examples and 3 epochs on a single consumer GPU, expect anywhere from 20 minutes to a couple of hours depending on sequence length and GPU class.

Ready to Fine-Tune?

Start with a small, clean dataset of 200-300 examples for a single narrow task before scaling up. The pipeline above — LoRA adapter, SFTTrainer, merge, GGUF export — is the same shape used for production fine-tunes, just at a scale that fits on hardware you probably already own.