How to Run a Local LLM on Your Own PC (Beginner's Guide)

You do not need a cloud API to use a large language model. Open-weight models can run entirely on your own computer: no data leaves your machine, there is no per-token bill, and it works offline. The trade-off is that local models are smaller than the largest cloud models, and speed depends on your hardware.

This guide explains what hardware you need, what "quantization" means, how to get a model running in minutes, and how to call it from your own code.

Why run a model locally?

• Privacy: prompts and documents never leave your computer — useful for sensitive code, notes or company data.
• No usage costs: after the hardware, every request is free.
• Offline use: works on a plane or with no internet.
• Control: pick the exact model and version, and it never changes underneath you.
• Learning: a great way to understand how LLMs actually work.

The downsides: smaller local models are less capable at complex reasoning than the largest cloud models, large models need expensive hardware, and you are responsible for setup and updates.

What hardware do you need?

The key resource is memory: the model's weights must fit in RAM, ideally in your graphics card's VRAM.

• GPU (NVIDIA, AMD, or Apple Silicon): much faster. If the model fits entirely in VRAM, responses stream quickly.
• Apple Silicon Macs use unified memory shared by CPU and GPU, so a Mac with lots of memory can run surprisingly large models.
• CPU only: works, but noticeably slower — fine for small models and experimentation.

Rough memory guide (4-bit quantized models)

Model size (parameters)	Approximate memory needed	Typical hardware
1–4 billion	2–4 GB	Almost any modern laptop
7–9 billion	5–8 GB	16 GB RAM laptop, or 8 GB GPU
12–14 billion	9–12 GB	12–16 GB GPU, or 32 GB Mac
30–35 billion	20–24 GB	24 GB GPU, or 32–64 GB Mac
70 billion	40+ GB	Multiple GPUs or a 64–128 GB Mac

These are approximations: longer context windows need extra memory on top of the weights. A 7–9B model is the sweet spot for most people starting out.

Quantization in plain English

Model weights are normally stored as 16-bit numbers. Quantization stores them with fewer bits — commonly 8, 5 or 4 — shrinking the model to roughly a half or a quarter of its size, with a modest loss in quality.

You will see names like Q4_K_M or Q8_0 in model files (from the GGUF format used by llama.cpp and Ollama). Rules of thumb:

• 4-bit (Q4): the usual default — good balance of size, speed and quality.
• 5–6-bit: slightly better quality if you have the memory.
• 8-bit: close to the original quality, about twice the size of 4-bit.
• Below 4-bit: noticeably worse; only when nothing else fits.

Option 1: Ollama (easiest from the command line)

Ollama downloads, manages and runs models with one command, and exposes a local API.

Install

• macOS and Windows: download the installer from the official site.
• Linux:

curl -fsSL https://ollama.com/install.sh | sh

Run a model

ollama run llama3.2

The first run downloads the model; then you get an interactive chat in the terminal. Type /bye to exit. Browse the model library on Ollama's website for other models and sizes — names and versions change frequently, so check the current list.

Useful commands:

ollama list              # downloaded models
ollama pull <model>      # download without running
ollama ps                # models currently loaded in memory
ollama rm <model>        # delete a model to free disk space

Option 2: A desktop app with a chat interface

If you prefer a graphical interface, desktop apps such as LM Studio or Jan let you search for models, download them and chat in a window, and they also offer a local API server. They use the same underlying model formats, so the hardware guidance above applies.

For maximum control and performance tuning, llama.cpp — the engine behind many of these tools — can be compiled and run directly.

Using a local model from your code

Ollama serves an HTTP API on http://localhost:11434. A minimal Python example using only the standard library:

import json
import urllib.request

def ask(prompt: str, model: str = "llama3.2") -> str:
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
    }).encode()
    req = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["message"]["content"]

print(ask("Explain what a Python generator is in two sentences."))

Ollama also offers an OpenAI-compatible endpoint at http://localhost:11434/v1, so many existing tools and SDKs work by changing only the base URL.

Ideas for local-model projects

• Summarize or tag your personal notes without uploading them anywhere.
• Classify or extract fields from documents in a private dataset.
• A coding helper for explaining unfamiliar code offline.
• A private search assistant over your own files using RAG.

Making it faster

• Choose a model that fits in VRAM. If part of the model spills into system RAM, speed drops sharply.
• Use a smaller model for simple tasks — classification and short summaries do not need the biggest model.
• Keep context short. Long prompts take longer to process and use more memory. See what tokens are.
• Close other GPU-heavy apps such as games or video editors.
• Update your GPU drivers and the runtime regularly; performance improvements are frequent.

Realistic expectations

• Small local models are good at summarizing, rewriting, simple coding help, classification and chat. They are weaker at long multi-step reasoning and niche facts.
• They hallucinate like any LLM — verify anything important.
• Check each model's license: "open weights" licenses vary, and some restrict commercial use.

FAQ

Can I run a local LLM without a GPU?

Yes. Small models (1–8B, 4-bit) run on a modern CPU with 8–16 GB of RAM. Expect slower responses than with a GPU.

Is a local LLM as good as cloud AI?

The largest cloud models are generally more capable, especially at complex reasoning. Good local models are more than enough for many everyday tasks, with complete privacy.

Is it safe to run local models?

Download models only from reputable sources and well-known tools. The GGUF format stores weights, not executable code, but as with any software, stick to trusted publishers.

Can I run a local LLM on a Raspberry Pi?

Only very small models, and slowly. A desktop or laptop is a much better experience — see self-hosting on a Raspberry Pi for what a Pi does well.

Further reading

• Ollama documentation
• llama.cpp on GitHub
• Related: RAG explained for developers