The GPU Is Too Small. Now What?

There are exactly two reasons you end up on the “multi-GPU” thread of an LLM serving build-out, and they are not the same problem:

• Fitting. Your model’s weights plus its KV cache do not fit on one card. A 70B model in bf16 is ~140 GB of weights alone — that’s two 80 GB GPUs with almost nothing left for the KV cache. You must spread the model across cards.
• Throughput. The model fits on one GPU, but one GPU cannot serve your request rate. You need more cards to push total tokens/sec up.

Conflating the two is the most common mistake in this space. The fix for “doesn’t fit” and the fix for “too slow” are different techniques, and applying the wrong one makes things worse, not just not-better. This post lays out the three parallelization strategies — tensor parallelism, pipeline parallelism, and replicas — when to use each, and the math that decides.

The Three Ways to Use More Than One GPU

Every multi-GPU scheme is a way of slicing a workload that is really one big matrix multiply + attention + a lot of KV memory. There are three fundamental arrangements:

The key rule that drives everything: tensor parallelism trades communication for memory; replicas trade memory for throughput; pipeline parallelism trades idle time for depth. Once you know which resource you’re short on, the choice is almost mechanical.

Tensor Parallelism: Fitting the Model Across Cards

Tensor parallelism splits a single layer’s weight matrices across GPUs. For a transformer’s attention and MLP, each GPU holds a slice of the weight matrix, computes a partial result on its slice, and the GPUs all-reduce to produce the full activation before moving to the next layer.

from vllm import LLM, SamplingParams

# TP=4 splits weights AND KV cache across 4 GPUs.
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    dtype="bfloat16",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    enable_prefix_caching=True,
)

params = SamplingParams(max_tokens=256, temperature=0.8)
out = llm.generate(
    "Explain tensor parallelism's communication overhead in one paragraph:",
    params,
)
print(out[0].outputs[0].text)

Why TP exists: the per-GPU memory for a layer is (weights + KV) / TP, so turning TP up makes the model fit on smaller per-card memory. In vLLM this is one parameter, tensor_parallel_size.

The catch is communication. Every layer boundary requires an all-reduce across the TP group. If GPUs are behind a fast interconnect (NVLink, ~900 GB/s), that’s cheap. If they’re on different PCIe buses or worse, different machines in the node, the all-reduce becomes the bottleneck and adding GPUs increases latency. This is why TP is a within-node, tightly-interconnected strategy.

The latency math: per token, each GPU does (layer compute) / TP work, then a constant-cost all-reduce. So per-token latency drops as TP rises only while the all-reduce is small relative to the compute. Over-provision TP — say TP=8 for a model that already fits at TP=2 — and the constant communication cost dominates; you’ve added GPUs and lost speed.

The scaling itself is a strong signal. Near-linear scaling in TP is possible up to a point (attention heads and the matrix kernels limit it). Watch the real TPOT (time per output token) as you raise TP; the moment it plateaus or climbs, you’ve hit the communication wall.

Pipeline Parallelism: Slicing Layers, Not Matrices

Pipeline parallelism assigns contiguous groups of layers to different GPUs: GPU 0 runs layers 0–15, GPU 1 runs layers 16–31, and activations flow forward between stages. It’s a fit-the-model strategy like TP, but it does not require the GPUs to be tightly tied — communication is one activation vector per stage boundary, not a per-layer all-reduce.

from vllm import LLM, SamplingParams

# pipeline_parallel_size stages the model's layers across devices.
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=2,          # within a "megatron" cell
    pipeline_parallel_size=3,        # three groups of layers
    dtype="bfloat16",
    max_model_len=8192,
    gpu_memory_utilization=0.90,
)

The fundamental problem with naive pipeline parallelism is the pipeline bubble: while GPU 1 is busy with the deep layers of sequence A, GPU 0 is idle waiting for its next chunk. Utilization drops, and the bubble grows with the number of stages. Modern serving stacks use interleaved and chunked schedules (micro-batching many small jobs through the pipeline back-to-back) to keep every stage busy, which turns PP from a latency loser into a workable throughput+fit strategy at scale.

In practice, most serving teams use hybrid: a small TP group inside a node or cell, and PP across stages to extend the total model size. TP handles the weights that must stay tightly coupled; PP stretches the model deeper across more devices without paying NVLink costs between every layer.

Replicas: Throughput, at the Cost of Redundant Memory

Replicas are boring and correct. You run several independent vLLM servers, each with its own full copy of the model, each on one or more GPUs, and put them behind a load balancer. There is zero inter-worker communication. Adding a replica adds its full weight+KV memory footprint but directly adds throughput.

# Not one process with TP — several independent processes behind a load balancer.
# Worker 0..N, each: vllm serve meta-llama/...-8B-Instruct --tensor-parallel-size 1

             ┌───────────────────────────────────────┐
             │   Load balancer (nginx / Ray Serve)   │
             └─────┬──────────────┬──────────────┬───┘
                   │              │              │
          ┌────────▼─────┐ ┌──────▼──────┐ ┌─────▼──────┐
          │ Worker 0     │ │ Worker 1    │ │ Worker 2   │
          │ TP=1 (GPU0)  │ │ TP=1 (GPU1) │ │ TP=1 (GPU2)│
          │ full model   │ │ full model  │ │ full model │
          └──────────────┘ └─────────────┘ └────────────┘

The rule of thumb that resolves most design arguments: if the model fits on one card and you’re out of throughput, add replicas, never TP. Turning up TP on a single-request-per-second model that fits on one GPU just adds communication and buys nothing. Replicas scale out RPS linearly, with memory as the only exchange.

Replicas also divide the KV cache, which matters for two things:

• Prefix caching locality. Each replica has its own prefix cache. And a hash-routed or affinity-routed load balancer keeps the same prompt families hitting the same replica, so the prefix cache actually warms — the same lesson as giving the KV cache a stable route key in a microservice architecture.
• Isolation. If you need per-tenant isolation (separate prefixes, separate KV pools, no cross-tenant attention state), a replica-per-tenant pattern is the cleanest enforcement: the tenant’s traffic never touches another tenant’s worker.

When the Model Doesn’t Split: Offloading and Quantization First

Before you reach for more GPUs, ask whether the model genuinely needs them. Quantization often dissolves the “doesn’t fit” problem entirely. A 70B in bf16 needs ~140 GB raw; cut the weights to INT4 and it’s ~35 GB — one 80 GB card, with room for a healthy KV cache. That single change can turn a forced multi-GPU problem into a single-card problem, which is strictly cheaper to operate.

The decision order that an engineer should follow:

1. Quantize first (weights, then KV cache to fp8). If the model fits on one card now, you’re done — no multi-GPU complexity.
2. If it still doesn’t fit, tensor-parallelize within a node (small TP, enough to fit).
3. Extend with pipeline parallelism only if the model is too deep/large for practical TP.
4. Once fitted, replicate for throughput.

Putting It Together: The Hybrid Pattern

The realistic production layout combines all three. Fit the model with a small TP group, stretch it with PP if needed, then replicate the whole arrangement behind a router for RPS:

                 ┌─────────────────────────────────────────┐
                 │            Load balancer                │
                 └───────┬─────────────────┬───────────────┘
                         │                 │
                ┌────────▼───────┐ ┌───────▼────────┐
                │ Worker 0       │ │ Worker 1       │
                │ TP=2, PP=2     │ │ TP=2, PP=2     │
                │ (4 GPUs)       │ │ (4 GPUs)       │
                └────────────────┘ └────────────────┘

Each worker is a vLLM instance with its weights split TP × PP ways; each worker group is replicated for throughput. This is the pattern that shows up in practice far more than any single pure strategy.

import os
from vllm import LLM, SamplingParams

TP = int(os.getenv("TP", "2"))
PP = int(os.getenv("PP", "2"))

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=TP,
    pipeline_parallel_size=PP,
    dtype="bfloat16",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    enable_prefix_caching=True,
)

Keeping the Fleet Healthy Under All Three

Whatever mix you choose, three operational concerns cut across every strategy:

• Communication-bound vs. compute-bound. Profile TPOT. If raising TP doesn’t improve TPOT, you’re communication-bound and should add replicas instead. If raising replicas doesn’t improve RPS, you’re hitting a shared bottleneck upstream (load balancer, tokenizer, or the KV pool) rather than the GPUs.
• KV cache locality across replicas. Route traffic by a stable prompt-family key so replicas keep warm prefix caches. A round-robin that scatters the same conversation across different replicas throws away prefix-cache hits on every call.
• Metrics that decide the choice. Watch TTFT (prefill/queueing) separately from TPOT (generation). A climbing TTFT points at prefill or queueing — fix with prefix caching or prefill capacity. A climbing TPOT as concurrency rises points at generation — fix with replicas.

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

def probe(prompt: str, max_tokens: int = 128):
    t0 = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )
    first, ttft = True, None
    for chunk in stream:
        if first:
            ttft = time.perf_counter() - t0
            first = False
    tpot = (time.perf_counter() - t0 - ttft) / max_tokens
    return {"ttft": ttft, "tpot": tpot}

Summary

• Two different problems hide under “multi-GPU”: fitting and throughput. They need different fixes.
• Tensor parallelism fits a model by splitting layers across tightly-interconnected GPUs; its cost is per-layer all-reduce, so it dies on slow interconnects and over-provisioning.
• Pipeline parallelism splits layers into stages to stretch model size; its cost is pipeline bubbles, mitigated by micro-batching.
• Replicas give throughput by duplicating the model behind a load balancer; their cost is redundant memory.
• The decision order: quantize first, then TP to fit, PP to extend, replicas for RPS.
• Use affinity routing so replica-local prefix caches stay warm, and separate TTFT from TPOT so you know which bottleneck to attack.

If the model doesn’t fit and the interconnect is fast, tensor-parallelize. If it fits but the fleet is slow, replicate — never over-TP. The GPU wasn’t too small; you were slicing the wrong dimension.