2026-07-18 · Tutorial

llama.cpp tensor-split: running one model across multiple GPUs

You have more than one GPU and a model that won't fit on either alone. Or you have a big card and a small card and want to use both without one of them sitting idle. That's what --tensor-split is for — but the flag is under-documented, the name is slightly misleading, and there's a second flag, --split-mode, that changes what it means. Here's the whole picture, with the output you should expect to see.

The one-sentence version

--tensor-split tells llama.cpp how to divide a model's weights across the GPUs you have. --split-mode tells it how to divide: by whole layers (the default) or by slicing individual tensors. Get those two straight and everything else is detail.

--split-mode: the flag that actually decides the strategy

There are three modes (-sm for short):

layer (default) — give each GPU a contiguous block of the model's layers. GPU 0 holds layers 0–39, GPU 1 holds 40–79, and a token flows through GPU 0 then GPU 1. This is pipeline-style. Interconnect traffic is tiny (just the activations handed off at the boundary), so it works fine over a plain PCIe slot — or even across machines over the network.
row — split individual weight tensors across GPUs by rows, so every GPU does part of every layer and their partial results get combined. This is the one people mean when they ask about tensor parallelism. It moves a lot more data between GPUs each token, so it only pays off with a fast link between them (NVLink, or at least a fat PCIe bus).
none — don't split; put the whole model on one GPU (--main-gpu). Use this when the model fits and you just want to pick which card.

The practical takeaway most people need: on consumer hardware without NVLink, stick with layer. It's simpler, it's usually just as fast for a single chat stream, and it's the only mode that makes sense once GPUs live in different machines. Reach for row only when you have the interconnect to feed it and you've measured that it's actually faster for your workload.

--tensor-split: the proportions

This is a comma-separated list of relative weights, one per GPU, in device order:

# two identical 24 GB cards — split the model evenly
./build/bin/llama-server \
  -m ~/models/qwen2.5-72b-instruct.Q4_K_M.gguf \
  -ngl 99 --split-mode layer --tensor-split 1,1 \
  --host 127.0.0.1 --port 8080

The numbers are ratios, not gigabytes. 1,1 is a 50/50 split; 3,1 puts three-quarters on the first GPU. So for a 24 GB card paired with an 8 GB card:

--tensor-split 3,1        # 24GB:8GB is roughly a 3:1 ratio

Size these to real free memory, not raw spec — leave headroom for the KV cache, which grows with context length and lives on the GPUs too. If you set the ratios wrong, the fuller GPU runs out during load and the whole thing dies with an out-of-memory error even though the other card had room to spare.

Leave --tensor-split off entirely and llama.cpp splits evenly across all visible GPUs. That's fine when your cards match and wrong the moment they don't.

Reading the split in the log

After load, llama.cpp prints exactly where the weights landed:

load_tensors: offloaded 81/81 layers to GPU
load_tensors:  CUDA0 model buffer size = 21440.19 MiB
load_tensors:  CUDA1 model buffer size =  7128.55 MiB
load_tensors:   CPU  model buffer size =   410.98 MiB

Two GPUs carrying the weights in a ~3:1 ratio, with a sliver left on CPU (the parts that always stay host-side). If a card is emptier than you intended, your ratios or your -ngl are off. -ngl 99 just means "offload as many layers as will fit" — llama.cpp stops when memory runs out, so a too-small number, not a too-big one, is what leaves layers stranded on the CPU.

"Does llama.cpp support tensor parallelism?"

This is the most-searched version of the question, so here's a straight answer: partly. --split-mode row is genuine row-wise tensor splitting, which is a form of tensor parallelism — each GPU computes a slice of every matmul and the results are reduced together. But it is not the heavily-optimized, overlap-everything tensor parallelism that vLLM or TensorRT-LLM implement for datacenter GPUs. llama.cpp's design goal is running models on hardware you already own, not squeezing maximum tokens-per-second out of an 8×H100 node.

What that means in practice:

Want to fit a model across multiple GPUs? llama.cpp does that well — use layer mode. Job done.
Want the last drop of throughput from a machine full of fast, NVLinked GPUs serving many users? That's the case where a dedicated tensor-parallel engine will clearly beat llama.cpp, and you should use one.

For almost everyone reading this — a couple of GPUs, or a couple of machines, running models at home — layer split is the right tool and the tensor-parallelism question is a distraction.

Beyond one machine

Here's the part that surprises people: --tensor-split doesn't only apply to local GPUs. When you attach remote workers with the RPC backend, each one shows up as another device in the same list, and the same proportions apply to them. So a split across a local GPU and two networked machines is just:

--rpc 192.168.1.10:50052,192.168.1.11:50052 \
--tensor-split 4,3,3

Local GPU gets 4/10ths, each remote machine gets 3/10ths. The mechanics of the RPC side — building the workers, the security caveats, the failure modes — are covered in the RPC backend post. Tensor-split is simply how you steer the proportions once the devices exist, local or remote.

Quick gotchas

KV cache lives on the GPUs. Long contexts eat VRAM that isn't in the model-weight numbers. Leave headroom or cap context with -c.
Order matters. The ratios map to devices in the order llama.cpp enumerates them. Check the log's CUDA0 / CUDA1 lines to confirm which physical card is which.
row mode without a fast link is a trap. It'll run, it'll just be slower than layer while you wonder why. Measure before committing to it.
Running several different models at once is a separate thing from splitting one — that's just multiple llama-server processes on different ports, no tensor-split involved.

Where this goes next

Tensor-split is the local, manual version of a bigger idea: pool whatever compute you can reach and treat it as one machine. Two GPUs in a case is the easy end. Ten machines across a house or an office is where you want something planning the split for you and re-planning when hardware comes and goes — which is what SharedLLM is: an open-source coordinator that does the device discovery, memory accounting, and split-sizing automatically, then runs the same llama.cpp under the hood. Start with the flags here; graduate to that when the hand-tuning gets old.