How to Run Local AI Models (LLMs) on Linux with Ollama

Disclosure: some links on this page are affiliate links. If you sign up through them we may earn a commission at no extra cost to you. It helps keep LinuxDistroFinder free.

Running a large language model (LLM) locally on Linux means your prompts never leave your machine, your data stays private, and you pay zero per-token API fees. Thanks to Ollama — a lightweight runtime that wraps llama.cpp and exposes a clean CLI plus a local REST API — getting a capable model like Llama 3, Mistral, or Gemma 3 running on Linux is now a single-command affair. This guide walks you through everything: installation, model management, hardware tuning, GPU acceleration, and a roundup of the best models to run locally in 2026.

Why Run an LLM Locally on Linux?

Cloud AI APIs are convenient, but they come with real trade-offs. Every prompt you send to a third-party API is logged, potentially used for training, and billed per token. For developers working with private codebases, sensitive documents, or regulated data, that matters enormously. Local inference solves all of it:

Privacy — prompts and completions never leave your hardware.
Zero ongoing cost — no per-token billing after the initial hardware outlay.
Offline capability — works on planes, in air-gapped environments, anywhere.
No rate limits — hammer the model as hard as your GPU (or CPU) allows.
Customisation — fine-tune, quantise, or swap models in seconds.

Linux is the ideal platform for this. Ollama has first-class Linux support, CUDA and ROCm drivers are mature, and the entire toolchain — from model weights to serving infrastructure — is open source.

Hardware Requirements: What Do You Actually Need?

The honest answer is: less than you think for small models, more than you think for large ones. LLM inference speed is almost entirely gated by memory bandwidth, not compute. Here are practical minimums and recommendations:

Model Size	Example Models	Min RAM (CPU-only)	Recommended GPU VRAM	Tokens/sec (CPU)	Tokens/sec (GPU)
~1–3 B params	Gemma 3 1B, Phi-3 Mini	4 GB RAM	2–4 GB VRAM	~18–30 t/s	~80–150 t/s
~7–8 B params	Llama 3.1 8B, Mistral 7B	8 GB RAM	6–8 GB VRAM	~6–12 t/s	~40–90 t/s
~13–14 B params	Llama 3.1 14B, Qwen2.5 14B	16 GB RAM	10–12 GB VRAM	~3–6 t/s	~25–55 t/s
~30–34 B params	Qwen2.5 32B, Phi-4	32 GB RAM	20–24 GB VRAM	~1–3 t/s	~15–30 t/s
~70 B params	Llama 3.3 70B	64 GB RAM	40–48 GB VRAM	<1 t/s	~8–18 t/s

Note: All figures assume 4-bit quantised (Q4_K_M) GGUF models, which is what Ollama downloads by default. Full-precision (FP16) models require roughly 2× the memory. Tokens per second measured on an AMD Ryzen 9 7950X (CPU) and an NVIDIA RTX 4090 (GPU).

A mid-range GPU from the last three years — an RTX 3060 (12 GB), RX 7600 (8 GB), or even an older GTX 1070 (8 GB) — will run 7B models comfortably. If you only have a CPU, modern multi-core processors like the Ryzen 7000 or Intel Core Ultra series with AVX-512 support are surprisingly capable for 7B models.

Installing Ollama on Linux

Ollama ships as a single self-contained binary. The official install script handles the binary, a systemd service, and shell completion in one go. Open a terminal and run:

# Official one-liner installer (works on Ubuntu, Fedora, Arch, Debian, etc.)
curl -fsSL https://ollama.com/install.sh | sh

The script detects your GPU drivers automatically. After it finishes, verify the service is running:

systemctl status ollama
# Should show: active (running)

ollama --version
# Example output: ollama version 0.3.14

Tip: If you prefer not to pipe scripts directly to your shell (good instinct!), you can download the binary manually from https://ollama.com/download/linux and move it to /usr/local/bin/ollama. Then create the systemd unit file yourself — Ollama's GitHub README has the exact unit file contents.

Manual Binary Install (No curl-pipe-bash)

# Download the binary directly
wget https://ollama.com/download/ollama-linux-amd64 -O ollama
chmod +x ollama
sudo mv ollama /usr/local/bin/

# Create a systemd service
sudo tee /etc/systemd/system/ollama.service <



Downloading and Running Your First Model
Once Ollama is running, pulling a model is identical to pulling a Docker image. Let's start with Llama 3.2 3B — a great first model because it fits in 4 GB of RAM and responds in real time even on CPU:

# Pull the model (downloads ~2 GB)
ollama pull llama3.2:3b

# Start an interactive chat session
ollama run llama3.2:3b

# You'll see a prompt — type anything and press Enter
>>> Explain quantum entanglement in simple terms.

To exit the interactive session, type /bye or press Ctrl+D.

Running a More Capable Model
For serious coding assistance or document analysis, pull the 8B variant of Llama 3.1 or Mistral 7B:

# Llama 3.1 8B — excellent for coding and reasoning
ollama pull llama3.1:8b

# Mistral 7B — fast, instruction-following, good for writing tasks
ollama pull mistral:7b

# List all downloaded models
ollama list

# Remove a model you no longer need
ollama rm mistral:7b

By default, Ollama stores all model weights in ~/.ollama/models. If your home partition is small, change the storage path by setting the OLLAMA_MODELS environment variable in the systemd unit file or your shell config.

Enabling GPU Acceleration

NVIDIA (CUDA)
Ollama's install script automatically detects NVIDIA GPUs if the proprietary driver and CUDA toolkit are present. Verify GPU offloading is active by checking the model's run output — you'll see a line mentioning layers loaded to GPU. You can also inspect with:

# Watch GPU utilisation while a model is running
watch -n 1 nvidia-smi

# Check which layers Ollama is offloading
OLLAMA_DEBUG=1 ollama run llama3.1:8b "Hello" 2>&1 | grep -i gpu

If CUDA isn't being used, install the NVIDIA driver and CUDA toolkit:

# On Ubuntu 22.04 / 24.04
sudo apt install -y nvidia-driver-550 nvidia-cuda-toolkit

# Reboot, then verify
nvidia-smi

AMD (ROCm)
ROCm support in Ollama is solid for RDNA 2 and RDNA 3 GPUs (RX 6000 and RX 7000 series). Install ROCm 6.x from AMD's official repository, then Ollama will pick it up automatically. Polaris (RX 480/580) and Vega cards have partial support via the HSA_OVERRIDE_GFX_VERSION environment variable trick.

# AMD ROCm install on Ubuntu (abbreviated)
wget https://repo.radeon.com/amdgpu-install/6.1.1/ubuntu/jammy/amdgpu-install_6.1.60101-1_all.deb
sudo dpkg -i amdgpu-install_6.1.60101-1_all.deb
sudo amdgpu-install --usecase=rocm
sudo usermod -aG render,video $USER

# For older GCN cards (RX 580 = gfx803), override the version:
export HSA_OVERRIDE_GFX_VERSION=10.3.0

Apple Silicon / Intel Arc
Ollama also supports Apple Metal (macOS only) and Intel Arc via the oneAPI backend — but this guide focuses on Linux x86. Intel Arc support on Linux is improving rapidly with the Arc A770 (16 GB) being a cost-effective option for 13B models.

Using Ollama's REST API
Ollama exposes a local HTTP API on http://localhost:11434, which means you can integrate it into scripts, applications, or even use it as an OpenAI-compatible backend (with the /v1/ prefix).

# Simple generation via curl
curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.1:8b",
    "prompt": "Write a Python function to reverse a linked list.",
    "stream": false
  }'

# OpenAI-compatible endpoint (works with many tools like Continue.dev)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "What is 17 * 14?"}]
  }'

Note: The OpenAI-compatible /v1/ endpoint lets you point tools like Continue (VS Code AI coding assistant), Open WebUI, Cursor (self-hosted), or any Python script using the openai library directly at your local Ollama instance — no code changes needed.

Installing Open WebUI (Browser Interface)
The CLI is great, but a web interface makes Ollama dramatically more accessible, especially for sharing with teammates. Open WebUI is the most polished option — it looks and works like ChatGPT but talks to your local Ollama:

# Requires Docker. Install Docker first if needed:
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

# Run Open WebUI (connects to Ollama on the host automatically)
docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

# Open in browser
xdg-open http://localhost:3000

Open WebUI gives you conversation history, system prompts, file uploads, RAG (Retrieval-Augmented Generation) with local documents, and multi-model switching — all in a browser tab, all 100% local.

Best Models to Run Locally in 2026


  
    #1
    Llama 3.1 8B / 3.3 70B
    Best All-Rounder
  
  Meta's Llama 3.x family remains the benchmark for open-weight models. The 8B variant runs fast on 8 GB VRAM and punches well above its weight on reasoning and coding. The 70B is the best locally runnable model for anything demanding a strong reasoning chain, if you have a 40+ GB VRAM setup or a beefy RAM machine.



  
    #2
    Mistral 7B / Mistral Nemo 12B
    Fast & Efficient
  
  Mistral 7B v0.3 remains one of the fastest models at its size class. Mistral Nemo 12B (a joint Mistral/NVIDIA release) is a significant step up in quality while still fitting in 10 GB VRAM. Both excel at instruction following, summarisation, and multilingual tasks.



  
    #3
    Qwen2.5 7B / 14B / 32B
    Best for Code
  
  Alibaba's Qwen2.5 series — especially the Coder variants — is exceptional for software development tasks. Qwen2.5-Coder 7B scores near GPT-4-level on HumanEval benchmarks and fits in 6 GB VRAM. The 32B instruct model is arguably the best sub-70B model available in 2026.



  
    #4
    Gemma 3 4B / 12B
    Best Small Model
  
  Google's Gemma 3 is remarkably capable for its size. The 4B model runs on just 4 GB of VRAM or even 6–8 GB of system RAM and supports multimodal inputs (image understanding) in the 12B variant. Great for embedded or resource-constrained setups.



  
    #5
    DeepSeek-R1 (distilled)
    Best Reasoning
  
  DeepSeek-R1's distilled 7B and 14B variants bring chain-of-thought reasoning capabilities to local hardware. The 7B distill (based on Qwen2.5) fits in 6 GB VRAM and outperforms much larger models on math, logic, and multi-step reasoning tasks. Pull it with ollama pull deepseek-r1:7b.


Useful Ollama Commands Cheat Sheet

# List available models on ollama.com
# (browse https://ollama.com/library in your browser)

# Pull a specific quantisation (e.g., Q8 for better quality)
ollama pull llama3.1:8b-instruct-q8_0

# Show model details and parameter count
ollama show llama3.1:8b

# Run with a custom system prompt
ollama run llama3.1:8b --system "You are a senior Linux kernel engineer."

# Pipe input from a file
cat mycode.py | ollama run qwen2.5-coder:7b "Review this code for bugs."

# Set number of GPU layers (useful if you're VRAM-limited)
OLLAMA_NUM_GPU=28 ollama run llama3.1:8b

# Set context window size (default is 2048, max depends on model)
ollama run llama3.1:8b --ctx-size 8192

# Check running model processes
ollama ps

Tip: Quantisation matters. Q4_K_M is the sweet spot — about 5% quality loss vs FP16 but uses half the VRAM. Q8_0 is near-lossless but uses ~2× the VRAM of Q4. For most users, stick with the default Q4_K_M unless you have headroom.

Running LLMs on a Remote Linux VPS
Not everyone has a powerful desktop GPU at home. Running Ollama on a cloud VPS with a GPU attachment is a great middle ground — you get the privacy benefits of self-hosting without the hardware cost, and you can access it from any device.


  🚀 Run Ollama on a GPU-Powered Linux VPS
  Vultr offers GPU cloud instances (NVIDIA A100, A16, L40S) with per-hour billing and full root access. Deploy Ubuntu 24.04, install Ollama in 60 seconds, and run 70B models you could never fit on a desktop GPU. New accounts get $100 free credit — enough to run a 70B model for weeks.
  Claim $100 Free Credit on Vultr →
  Need a cheaper always-on instance for smaller models? Hostinger VPS starts at ₹149/mo and runs 7B models on CPU just fine for low-volume personal use.


To expose your remote Ollama instance securely (without opening port 11434 to the internet), use an SSH tunnel:

# On your local machine — tunnel remote Ollama to localhost:11434
ssh -L 11434:localhost:11434 user@your-vps-ip -N

# Now Ollama on the VPS is accessible locally as if it were running on your machine
ollama run llama3.3:70b

Frequently Asked Questions

Can I run LLMs on Linux without a GPU?
Yes, absolutely. Ollama uses llama.cpp under the hood, which has highly optimised CPU inference using AVX2/AVX-512 SIMD instructions. On a modern 8-core CPU like the Ryzen 7 7700, you can expect 8–12 tokens per second with a 7B Q4 model — perfectly usable for non-interactive tasks like batch summarisation or code generation in scripts. It's slower than GPU inference but entirely viable.

Where does Ollama store downloaded model weights?
By default, in ~/.ollama/models/. You can change this by setting the OLLAMA_MODELS environment variable. For example, add Environment="OLLAMA_MODELS=/data/ollama/models" to your systemd unit file if your home partition is small but you have a large secondary drive mounted at /data.

What's the difference between Ollama's model tags (e.g., :7b vs :7b-instruct)?
The base tag (e.g., mistral:7b) is the base pre-trained model — it's a text completer, not an instruction follower. The -instruct or -chat variants have been fine-tuned with RLHF or DPO to follow instructions and engage in conversation. Always use the -instruct or -chat variant for chatbot-style use. Ollama often defaults to the instruct variant when no tag suffix is specified.

How do I update a model to the latest version?
Simply pull it again. Ollama checks the manifest and only downloads changed layers, similar to Docker. Run ollama pull llama3.1:8b and it will fetch only what has changed since your last pull.

Is Ollama the only way to run LLMs locally on Linux?
No — alternatives include llama.cpp (the underlying engine, usable directly via CLI), LM Studio (GUI app, Linux support is newer), Jan (cross-platform GUI), and vLLM (production-grade serving, better for multi-user deployments). Ollama is simply the most beginner-friendly with the best model library and tooling ecosystem as of 2026.

Can I use my local Ollama model with VS Code for AI coding assistance?
Yes. Install the Continue extension in VS Code, then point it at http://localhost:11434 using the Ollama provider. You can then use any locally installed model as your autocomplete or chat AI — completely free, completely private, and with no monthly subscription.

What Linux distro is best for running Ollama?
Any mainstream distro works. Ubuntu 22.04 and 24.04 LTS are the easiest due to well-maintained NVIDIA and AMD GPU driver packages. Fedora 40+ works great out of the box with AMD GPUs via the default open-source amdgpu driver. Arch Linux gives you the latest kernel and GPU driver versions fastest, which matters for bleeding-edge hardware like Intel Arc.

  ← Back to all guides