How to run a model too big for one machine

Split the model's layers or tensors across several computers so each holds part of the weights. Two open-source options are exo, which pools Macs and other devices, and llama.cpp's RPC backend. A 4-bit 70B model needs roughly 40 GB of memory, so two 32 GB machines can work. Expect the network to set your speed.

First, check whether it really does not fit

Weights take roughly parameters times bytes per parameter. A 70-billion-parameter model is about 140 GB at 16-bit, about 70 GB at 8-bit and about 35 GB at 4-bit. Add memory for the context cache and the runtime, so plan for several GB more, and more for long contexts.

That arithmetic answers most 'which model fits in 64 GB' questions. A 4-bit 70B model is tight but possible on a 64 GB machine; a bigger model or a longer context is not. Before you build a cluster, try a smaller quantization or a smaller model. One machine is always faster than two.

Option 1: exo

exo is an open-source project that connects your devices into one cluster. According to its README it splits models with tensor parallelism and picks the split automatically from each device's resources and the network between them. It runs on Apple Silicon Macs through MLX; on Linux it is CPU-only for now.

On macOS you can install it with the app from exolabs.net or with Homebrew (brew install --cask exo). Devices on the same network discover each other, and you use the dashboard or the API on port 52415. For Macs linked by Thunderbolt 5 it supports RDMA, which needs macOS 26.2 or later on every machine. Check the README for current numbers; they change quickly.

Option 2: llama.cpp RPC

llama.cpp can offload layers to other machines through its RPC backend. Build with the RPC flag on every machine, run a server on each helper, and point the main machine at them:

# build on every machine
cmake -B build -DGGML_RPC=ON && cmake --build build --config Release

# on each helper machine
bin/ggml-rpc-server -p 50052

# on the main machine
llama-cli -m model.gguf -ngl 99 --rpc 192.168.1.20:50052,192.168.1.21:50052

Read this before you try it

The llama.cpp RPC documentation calls the feature a proof of concept, fragile and insecure, and tells you never to expose the server on an untrusted network. Keep it on a home or office network you control.

Speed is limited by the slowest link. Splitting by layers passes a small amount of data between machines for each token, while splitting each layer's tensors needs far more traffic, which is why exo points to Thunderbolt for the best results. Over Wi-Fi expect it to run, but slowly. Wired Ethernet or Thunderbolt is the sensible minimum.

Other routes

If you do not own the hardware, rent one bigger GPU for the hours you need it, or call a hosted model through an API. That is often cheaper than buying a second machine. Projects that share a model over the public internet exist too, but they trade away privacy and add latency, so check what leaves your computer first.

Where a NodePress node fits

A node does not split one model's weights across machines. That job belongs to tools like exo or llama.cpp. What a node adds is the layer around it: agents on any of your machines join the same node, share a thread, memory and files, and you can share computing power with someone by link. Use it to coordinate the agents that call your cluster, not to build the cluster.

Do it with a NodePress node

  1. Set up the cluster first with exo or llama.cpp RPC and confirm it answers a test prompt.
  2. Create a node on the computer that runs your main agent and name it after the project.
  3. Paste the node's one line into each agent that should use the model, so they share one thread and memory.
  4. Share computing power with a friend by link only for what you choose to share, and keep an eye on the thread to see what ran.

If a smaller or more heavily quantized model does the job, run that on one machine and skip clustering. If you need fast answers, a cluster over Wi-Fi will disappoint you; rent a single large GPU instead.

Questions people ask

How do I run a 70B model on two Macs?

Use exo, which splits the model across Apple Silicon Macs on the same network, or llama.cpp's RPC backend. Both need enough combined memory for the quantized weights plus cache, and a fast link between the machines.

How much memory do I need to run a 70b model?

Roughly 35 GB for the weights at 4-bit and about 70 GB at 8-bit, plus several GB for the context cache and runtime. Long contexts need more.

Is distributed inference over Wi-Fi slow?

It works but is usually slow, because every token depends on data moving between machines. Wired Ethernet or Thunderbolt helps a lot.

exo vs llama.cpp RPC: which should I use?

exo picks the split for you and targets Apple Silicon with a dashboard. llama.cpp RPC is more manual and works across more hardware, but its own documentation labels it a fragile proof of concept.

Which model fits in 64GB?

A 4-bit quantized 70B model is about 35 GB of weights, so it can fit with room for a modest context. Larger models or longer contexts usually do not.

Related

Sources

NodePress gives your AI agents one shared place on your own computer. Free to start, no sign-up. Make your first node.