How to Run Llama 3.1 Locally on Your Mac or PC

Last Updated on 29 Jun, 2026 by Montel Anthony

Key Takeaways (TL;DR)

• Llama 3.1 is Meta’s current open-weight flagship, available in 8B, 70B, and 405B parameter sizes — with a massive 128K context window across all three.
• The 8B model (4.9 GB) is the practical choice for most Mac and PC users; the 70B (43 GB) needs a powerful GPU or multi-GPU setup; the 405B (243 GB) is for server-grade hardware.
• Ollama is still the fastest way to get Llama 3.1 running locally — one command on Mac, Windows, or Linux.
• llama.cpp (now maintained under ggml-org/llama.cpp on GitHub, with 118k+ stars) has a completely modern CLI: llama-cli and llama-server replace the old main binary.
• llama.cpp now installs via brew, winget, nix, or conda-forge — no manual build required for most users.
• Both tools are free, private, and work fully offline once the model is downloaded.

Llama 3.1 is Meta’s most capable openly available model family, released in mid-2024 and still the go-to choice for local AI inference in 2026. With three size options — 8B, 70B, and 405B parameters — and a 128,000-token context window on every variant, it is a significant leap beyond the Llama 2 model this guide originally covered.

Running Llama 3.1 locally means your prompts, files, and conversations never leave your machine. No API costs. No cloud dependency. No privacy trade-offs. This guide walks through two proven methods: Ollama (simplest, cross-platform, recommended for most users) and llama.cpp (more control, server mode, broader model support).

What Are the Best AI Tools Right Now? A Curated List

What Is Llama 3.1?

Llama 3.1 is Meta’s third-generation open-weight language model, available in three sizes:

Compared to Llama 2, the 3.1 generation brings multilingual support, significantly stronger reasoning and coding, native tool use (function calling), and a context window that is 16x longer. The 405B model is the first openly available model that competes with GPT-4o and Claude 3.5 Sonnet on general benchmarks, according to Meta’s evaluations across 150+ benchmark datasets.

The most practical starting point for personal hardware is the 8B model. It fits in under 8 GB of RAM and runs at a usable speed even on CPU-only machines.

Method 1: Ollama (Recommended — Mac, Windows, Linux)

Ollama is the fastest path to running Llama 3.1 locally. It handles model downloading, quantization selection, and the inference server automatically, all behind a single CLI command. It supports Mac (Apple Silicon and Intel), Windows, and Linux.

Step 1: Install Ollama

Download and install Ollama from ollama.com. On macOS, drag the app to your Applications folder and launch it. On Windows, run the installer. On Linux, use the install script:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, Ollama runs as a background service.

Step 2: Pull and Run Llama 3.1

Open your terminal and run:

# Pull and run the 8B model (4.9 GB — recommended for most users)
ollama run llama3.1

# Or pull the 70B model (43 GB — needs 48 GB+ RAM or a high-VRAM GPU)
ollama run llama3.1:70b

# Or pull the 405B model (243 GB — server-grade hardware required)
ollama run llama3.1:405b

The first run downloads the model. Subsequent runs start instantly from the local cache. Once the model loads, you are dropped into an interactive chat prompt — type your message and press Enter.

Example session:

>>> Explain how a transformer neural network works in simple terms.
A transformer is like a very attentive reader that can look at every word in a sentence simultaneously,
rather than reading left to right one word at a time. It uses a mechanism called "attention" to decide
which words are most relevant to each other when generating a response...

Step 3: Use Ollama via API (Optional)

Ollama also exposes an OpenAI-compatible REST API at http://localhost:11434, so you can connect local apps, scripts, or coding environments directly:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1",
    "messages": [{"role": "user", "content": "Write a Python function to reverse a string"}]
  }'

You can also launch Llama 3.1 from Ollama’s supported app ecosystem including Claude Code, Codex App, OpenCode, and Hermes Agent — all accessible directly via ollama launch.

Method 2: llama.cpp (Advanced — Full Control, Server Mode, Broader Model Support)

llama.cpp is a C/C++ inference engine that supports a wide range of quantized model formats. It is now maintained under ggml-org/llama.cpp (previously ggerganov/llama.cpp) and has grown to 118,000+ GitHub stars. Apple Silicon gets first-class Metal GPU acceleration. NVIDIA GPUs are supported via CUDA. AMD via HIP.

The old main binary is gone. The current CLI tools are llama-cli (interactive chat) and llama-server (OpenAI-compatible HTTP server).

Step 1: Install llama.cpp

The easy way — package managers (recommended):

# macOS
brew install llama.cpp

# Windows (via winget)
winget install llama.cpp

# Linux / macOS (via conda-forge)
conda install llama.cpp -c conda-forge

From pre-built binaries:
Download the latest release directly from the llama.cpp releases page. Extract and run — no compilation needed.

Build from source (for GPU support or custom builds):
See the official build guide on GitHub.

Step 2: Download a Llama 3.1 GGUF Model

llama.cpp uses the GGUF model format. You can download Llama 3.1 GGUF files directly from Hugging Face using the built-in -hf flag:

# Download and run the Llama 3.1 8B model directly from Hugging Face
llama-cli -hf ggml-org/Llama-3.1-8B-Instruct-GGUF

Or download manually from Hugging Face and point to the local file:

llama-cli -m /path/to/llama-3.1-8b-instruct-q4_k_m.gguf

Recommended quantization for local use:

• Q4_K_M — best balance of speed, quality, and file size for most hardware
• Q5_K_M — slightly better quality, slightly larger
• Q8_0 — near full-quality, needs more VRAM/RAM

Step 3: Run llama-cli (Interactive Chat)

# Basic interactive chat (models with built-in chat templates auto-enable conversation mode)
llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf

# Manually enable conversation mode with a chat template
llama-cli -m llama-3.1-8b-instruct-q4_k_m.gguf -cnv --chat-template llama3

Once running, type your message and press Enter. The model replies in the terminal.

Step 4: Run llama-server (OpenAI-Compatible API)

For a local API endpoint compatible with any OpenAI SDK:

# Start server on port 8080 (default)
llama-server -m llama-3.1-8b-instruct-q4_k_m.gguf --port 8080

# Web UI accessible at: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions

You can then connect any app that supports the OpenAI API — just point it at http://localhost:8080 instead of https://api.openai.com.

Multi-user / parallel decoding:

# Up to 4 concurrent users, 4096 tokens each
llama-server -m llama-3.1-8b-instruct-q4_k_m.gguf -c 16384 -np 4

System Requirements

For Ollama + Llama 3.1 8B (most users)

For llama.cpp + Llama 3.1 8B

• Mac: Apple M1/M2/M3/M4 (Metal GPU acceleration built-in), or Intel Mac with 16 GB RAM
• Windows/Linux: x64 CPU with 16 GB RAM; NVIDIA GPU with 8 GB+ VRAM for GPU acceleration
• Apple Silicon advantage: Metal framework gives Apple Silicon Macs GPU acceleration out of the box — no separate GPU required

For Llama 3.1 70B

You need approximately 48 GB of combined RAM/VRAM. This works on a Mac with 64 GB unified memory (M2/M3 Ultra, M4 Max), or a workstation with two NVIDIA RTX 4090s.

Llama 3.1 vs. Llama 2 — What Changed

Troubleshooting Common Issues

Ollama: “Error: model not found”

Run ollama list to see what you have downloaded. If llama3.1 is not listed, run ollama pull llama3.1 to fetch it first.

llama.cpp: “command not found: llama-cli”

If you built from source, make sure the build output folder is in your PATH, or run the binary directly from the build directory. If you used brew, run brew link llama.cpp.

Model runs very slowly

On CPU-only setups, expect 2–8 tokens per second for the 8B model — usable but slow. For GPU acceleration on macOS, Metal is enabled automatically. On Windows/Linux with an NVIDIA GPU, ensure CUDA is installed and rebuild llama.cpp with CUDA support, or use Ollama which handles this automatically.

Out of memory crash

Switch to a lower quantization (e.g., Q4_K_M instead of Q8_0), or use a smaller model. On Mac, close other apps to free unified memory.

Ollama port conflict

If port 11434 is already in use, set OLLAMA_HOST=127.0.0.1:11435 before starting Ollama.

Privacy and Offline Use

Once the model is downloaded, both Ollama and llama.cpp run entirely offline. No data is sent to Meta, Ollama, or any third party. Your prompts, documents, and conversations stay on your local machine. This makes local Llama 3.1 a strong choice for:

• Drafting sensitive documents or code
• Experimenting with AI on air-gapped machines
• Avoiding per-token API costs on high-volume tasks

For a broader look at open-source productivity tools that respect your privacy, see our roundup of Top 20 Free Open-Source Tools for Maximum Productivity.

FAQs

Can I run Llama 3.1 on a Mac with Apple Silicon?

Yes, and it is one of the best platforms for it. Ollama and llama.cpp both use Apple’s Metal framework to accelerate inference on M1/M2/M3/M4 chips. The 8B model runs comfortably on a base MacBook Air with 8 GB unified memory.

What is the difference between Ollama and llama.cpp?

Ollama is a user-friendly wrapper that automates model management, downloading, and serving. llama.cpp is the underlying inference engine that gives you more granular control over quantization, server configuration, and advanced features like speculative decoding. Ollama actually uses llama.cpp under the hood.

Do I need a GPU?

No. Both tools run on CPU. GPU acceleration (Metal on Mac, CUDA on NVIDIA, HIP on AMD) makes inference significantly faster, but the 8B model is usable on a modern CPU at 2–8 tokens per second.

Is Llama 3.1 free to use?

Yes. Meta releases Llama 3.1 under a custom community license that allows use, redistribution, and fine-tuning. Commercial use is allowed for most applications. The full license is available at Meta’s Llama page.

Can I use the local model as an API for my apps?

Yes. Both ollama serve and llama-server expose an OpenAI-compatible REST API. Point any app that supports the OpenAI SDK at your local endpoint and it works without code changes.

Where can I find more Llama 3.1 GGUF models?

Browse Hugging Face GGUF models filtered by trending. Search for llama 3.1 gguf to find the quantized variants.

What happened to MLC LLM?

MLC LLM (the mobile app approach covered in the original version of this guide) has evolved significantly. For mobile use, check the current MLC LLM GitHub for the latest iOS and Android builds. However, for Mac and PC use, Ollama and llama.cpp are now far simpler and more capable options.

Conclusion

Running Llama 3.1 locally in 2026 is faster, simpler, and more capable than running Llama 2 was when this guide was first published. The model is better across every metric — longer context, stronger reasoning, native tool use, and multilingual support — while the tooling has matured to the point where getting started takes a single terminal command.

Start with ollama run llama3.1 for the fastest path. Move to llama.cpp’s llama-server when you need a persistent local API, multi-user support, or fine-grained quantization control. Either way, you get a genuinely capable AI assistant running entirely on your hardware, with no API costs and no data leaving your machine.