2026-07-18 · Tutorial

llama.cpp RPC backend: distributed inference across multiple machines

If you search for how the llama.cpp RPC backend works, you land on a one-paragraph README section and a handful of GitHub issues. That's it. There's no real manual — which is strange, because it's one of the most useful things in the project: it lets you take several ordinary computers and run a single model across all of them, as if their memory were pooled into one machine.

This is that manual. It covers the general case — any mix of machines, any backend (Metal, CUDA, CPU) — and more than two workers. If you specifically have two Apple Silicon Macs and want the copy-paste version, read Splitting Llama across two MacBook Pros first; this post is the wider reference behind it.

What the RPC backend is (and isn't)

Inside llama.cpp, ggml is the tensor library, and it talks to hardware through pluggable backends: CPU, Metal, CUDA, Vulkan, SYCL, and one called RPC. Most backends run math on local silicon. The RPC backend doesn't. It serializes each tensor operation, ships it over a TCP socket to a rpc-server process on another machine, that machine runs the op on its local backend, and the result comes back.

So from the main process's point of view, a remote machine is just another device with some memory and the ability to run ops. You can hand it a slice of the model's layers exactly like you'd hand layers to a second GPU. The scheduler doesn't care that the "GPU" happens to be a Mac Studio in the next room.

What it is not is a speed-up. Splitting a model across machines does not make tokens come out faster than running the whole thing on one machine that could fit it. Every remote op pays a network round-trip. The reason to do this is memory: you get to run a 70B model when no single machine you own could hold a 70B model. You trade latency for capacity. Be honest with yourself about which one you're short on before you build a cluster.

Rule of thumb, restated for N machines: RPC turns "I can't run this model at all" into "I can run this model, slowly." It does not turn "slow" into "fast."

Step 1 — Build with GGML_RPC=ON on every machine

Every machine in the cluster needs a llama.cpp built with the RPC backend compiled in, and every one needs to be the same version. The GGML wire format changes between releases; mismatched builds hang at the handshake or crash mid-inference. Pick a tag and pin all machines to it.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b4404          # pin the SAME tag on every machine

# macOS (Metal worker)
cmake -B build -DGGML_RPC=ON -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release

# Linux + NVIDIA (CUDA worker)
cmake -B build -DGGML_RPC=ON -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release

# CPU-only worker (still useful — RAM is RAM)
cmake -B build -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release

cmake --build build --target rpc-server llama-server -j

You want two binaries out of this: rpc-server (runs on the workers) and llama-server (runs on the one machine you'll actually talk to — call it the primary). The primary needs RPC compiled in too, because that's the machine loading the RPC backend to reach out to the others.

Step 2 — Start rpc-server on each worker

On every worker, start the server and tell it how much memory it's allowed to lend to the network:

# worker A (a CUDA box)
./build/bin/rpc-server -H 0.0.0.0 -p 50052 -m 20000

# worker B (a Mac)
./build/bin/rpc-server -H 0.0.0.0 -p 50052 -m 12000

Each worker prints its endpoint and backing memory and then waits. Leave them all running. A worker holds no state between runs — if one dies, you restart it and reload the model on the primary.

Step 3 — Point llama-server at all of them

This is the part the README glosses over. --rpc takes a comma-separated list of workers. List every one:

./build/bin/llama-server \
    -m ~/models/llama-3.3-70b-instruct.Q4_K_M.gguf \
    --rpc 192.168.1.10:50052,192.168.1.11:50052 \
    --host 127.0.0.1 --port 8080 \
    -ngl 99

llama.cpp now sees a device list that includes the primary's own backend (Metal or CUDA), any local GPUs, and one RPC device per worker. With -ngl 99 it offloads as many layers as it can and spreads them across that whole device list. In the startup log you'll see the split spelled out:

load_tensors: offloaded 81/81 layers to GPU
load_tensors:  RPC[192.168.1.10:50052] model buffer size = 18022.14 MiB
load_tensors:  RPC[192.168.1.11:50052] model buffer size = 11498.30 MiB
load_tensors:              Metal model buffer size =  9640.05 MiB

Three model buffer size lines, three machines each holding a chunk of the weights. That's a 70B model resident across hardware where no single box could have held it.

Controlling who holds how much

By default llama.cpp splits layers roughly evenly, which is wrong when your machines are uneven — a 24 GB card and an 8 GB laptop should not get equal shares. Use --tensor-split to set the proportions, in the same order your devices appear:

--tensor-split 20,11,9

Those are relative weights, not gigabytes — here the first device gets 20/40ths of the layers, and so on. Match them to each machine's real free memory and you'll fit a bigger model without any single worker running out. (This flag has a second, more interesting use for multiple GPUs in one box — that's the subject of the tensor-split post.)

Step 4 — Confirm it's really distributed

Send a completion and watch a worker's stdout:

curl -s http://127.0.0.1:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"List three prime numbers.","n_predict":32}'

On each worker you should see a steady stream of remote_graph_compute lines — the primary handing ops out and pulling results back. If a worker is silent, its layers never got assigned: usually the primary had enough local memory to take everything, or --rpc couldn't connect. The blunt way to force a real split is to pick a model bigger than the primary can hold alone.

The failures you'll actually hit

Everything hangs at load, no error

Version skew. One machine is on a different llama.cpp build than the others. Re-check that every machine is on the exact same tag, rebuild the odd one out, retry. This is the single most common cause of "it just sits there."

GGML_ASSERT(tensor->ne[0] % 512 == 0) on a worker

A real RPC limitation: some quantized tensor shapes whose row count isn't a multiple of 512 assert-fail when serialized. It shows up on certain Q4_K / Q5_K model shapes. Switch that model to Q8_0, or use a model whose hidden dimension is a clean multiple of 512. (SmolLM2-360M, for example, is not 512-aligned and won't offload over RPC; Llama 3, Qwen, Phi-3 are fine.)

It works but crawls

Two usual suspects. First, WiFi — move to wired ethernet or a Thunderbolt bridge; I've watched throughput fall by 10× just from switching a worker onto WiFi. Second, one slow machine drags the whole chain, because tokens are generated in sequence across the layer split. A cluster runs at the speed of its weakest hop, not the average.

Anyone on the network can use your worker

This is the one that should worry you. rpc-server has no authentication. Bound to 0.0.0.0 on a network you don't fully trust, anyone who can reach the port can run arbitrary tensor graphs on your machine and read whatever's in its memory pool. On a home LAN behind a router that's usually fine. Anywhere else, put the workers on a private network — Tailscale or WireGuard is the easy button — and never expose the port to the open internet.

RPC, tensor-split, or pipeline — which one?

People conflate these. Quick map:

When you outgrow hand-wiring

Two or three machines you can point at each other by hand. Past that it gets fiddly fast: you're tracking IPs, restarting dead workers, editing --rpc lists, and hoping nobody unplugs a laptop mid-generation. That coordination is exactly the gap SharedLLM fills — a coordinator that discovers workers, tracks each one's free memory, builds the split for you, re-plans when a machine drops, and wraps the raw RPC socket in an authenticated (HMAC-SHA256) proxy so you're not leaving an open port on the network. It's the RPC backend you just set up, minus the babysitting, under AGPL-3.0 so it can't be fenced off later.

But you don't need any of that to start. Two machines, one --rpc flag, and a model that was too big an hour ago. Go.

Related: llama.cpp tensor-split: one model across multiple GPUs → · Two MacBook Pros, one model →