Your Model’s Weights Are the Easy Part. Serving Them Is Not.

Anybody can load a 70B model and generate a sentence. The hard problem is doing it 10,000 times a second, for a mix of 500-token chat responses and 30k-token document summaries, on a GPU budget you actually have, without the p95 latency blowing past 1.2 seconds.

Most LLM latency is not the forward pass over the prompt. It is the autoregressive generation loop: every output token re-reads the entire attention context. A 2,048-token prompt generates each new token with a full attention score over all 2,048 keys. The naive math is brutal, and it is why naïve LLM serving collapses under load.

This post is about the four levers I actually use to operate and scale LLM inference infrastructure:

1. vLLM — PagedAttention-based serving that stops the KV cache from fragmenting all over the GPU.
2. Quantization — cutting weights (and KV cache) down so models fit on fewer, cheaper GPUs.
3. Prefix caching — reusing computed attention state instead of recomputing the same prompt prefix every request.
4. Multi-GPU serving — tensor parallelism to fit models that don’t fit on one card, and higher throughput once they do.

I’ll back each one with runnable Python, not just theory.

Why Normal PyTorch Serving Dies Under Load

A naive serving loop looks like this and it works, until about three concurrent users:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model.eval().to("cuda")

def generate(prompt: str, max_new_tokens: int = 128) -> str:
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

    past_key_values = None
    for _ in range(max_new_tokens):
        with torch.no_grad():
            out = model(input_ids=input_ids, past_key_values=past_key_values)
        next_id = out.logits[:, -1].argmax(dim=-1).unsqueeze(0)
        past_key_values = out.past_key_values

        if next_id.item() == tokenizer.eos_token_id:
            break
        input_ids = next_id

    return tokenizer.decode(input_ids[0])

Each iteration recomputes attention over the whole KV history and allocates fresh large-tensor copies of past_key_values on every step. Under concurrency, requests interleave in a way that makes the GPU memory allocator thrash and leaves paged memory so fragmented that a second request can’t get a contiguous block even though the card has free capacity overall.

Memory is the real constraint. For a model with n_layers layers, n_kv_heads KV heads, dim hidden size, head_dim per head, you need roughly:

kv_bytes_per_token ≈ 2 × n_layers × n_kv_heads × head_dim × 2_bytes_per_fp16

For a 32-layer model with 4 KV heads and a 128-dim head, that’s about 65 KB of KV cache per token. A 4,096-token sequence eats ~273 MB of KV cache. On an 80 GB GPU with ragged, concurrent requests, unmanaged allocation fragments that space into unusable chunks.

Levers 1 & 2: vLLM + Quantization, Working Together

The single highest-leverage change is moving from the manual loop above to vLLM, which is built around PagedAttention — the KV cache is split into fixed-size pages and tracked in a block table, exactly like virtual memory pages in an OS. No fragmentation, no reallocation blowups, and multiple concurrent sequences can share allocated pages.

Install and serve:

pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,        # 1 for a single GPU; see multi-GPU section
    max_model_len=8192,            # cap sequence length to bound KV cache usage
    gpu_memory_utilization=0.90,   # how much VRAM vLLM is allowed to take
    dtype="bfloat16",
    enable_prefix_caching=True,    # turn on shared-prefix reuse (below)
    kv_cache_dtype="auto",         # can be fp8_e5m2 to shrink KV cache
)

params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=256,
)

prompts = [
    "Explain paged attention in two sentences:",
    "Give three ways to cut LLM serving latency:",
]

outputs = llm.generate(prompts, params)
for out in outputs:
    print(out.prompt, "=>", out.outputs[0].text)

vLLM handles dynamic batching (continuous batching): while one generation is in flight, other requests with empty GPU slots get added to the same batch. That alone pushes utilization from “one request at a time” to “the GPU is nearly always doing useful work.”

That’s the weights-handling half.

Quantization: Fitting a Big Model onto Fewer, Cheaper Cards

Quantization reduces memory so a model fits on a smaller GPU, and for many setups it actually speeds generation because you fetch fewer bytes per parameter. The classic trade-off is precision vs. output quality, but modern 4-bit and 8-bit schemes keep degradation minimal.

Two choices matter:

• Which weight format. Group-wise quantized INT4/INT8 schemes like AWQ and GPTQ keep a small per-group scale so the quantization error stays local. Down to fp8/bf8 you generally don’t need tuning at all.
• Which KV cache format. KV cache is the biggest per-token memory hog, so quantizing it down to fp8 (kv_cache_dtype="fp8_e5m2") is often the cheapest memory win because it doesn’t touch model quality.

# Download an AWQ-quantized 70B instead of the full-precision one
# e.g. TheBloke/Llama-2-70B-Chat-AWQ
pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-AWQ-INT4",
    quantization="awq",            # weight quant: 4-bit, group-wise
    dtype="float16",               # compute still happens in fp16
    max_model_len=8192,
    gpu_memory_utilization=0.95,
    kv_cache_dtype="fp8_e5m2",     # KV cache quantized to fp8
    enable_prefix_caching=True,
)

params = SamplingParams(max_tokens=128)
out = llm.generate("Summarize the economics of quantized serving:", params)
print(out[0].outputs[0].text)

Memory math explains why this matters. A full-precision 70B model needs ~140 GB of weights in fp16 — that’s two A100s (80 GB each) with almost nothing left for KV cache. At INT4, weights drop to ~35 GB, so the same model fits on one 80 GB card and still leaves room for a healthy KV cache. One GPU instead of two, plus FP8 KV cache means far more concurrent sequences per card. That is the entire serving cost equation changing.

Use the --quantization path on the CLI for quick experiments:

vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
    --quantization awq \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.95 \
    --kv-cache-dtype fp8_e5m2 \
    --enable-prefix-caching

Lever 3: Prefix Caching — Stop Recomputing What You Already Know

Attention over a fixed prompt prefix is a deterministic computation: the same prefix yields the same keys and values, every single time. If thousands of requests share a system prompt, or share a long instruction such as “You are a maritime domain assistant. Use only the JSON schema below…”, a naive server recomputes those KV tensors for each and every request. That is pure waste.

Prefix caching keeps the computed KV blocks of recently seen prefix chunks in the cache. When a new request arrives with the same prefix up to the last token, vLLM loads the cached KV blocks from disk or GPU memory and only computes attention for the new part of the prefix. The speedup scales with how much of the prompt is shared:

from vllm import LLM, SamplingParams
from datetime import datetime

SYSTEM_PROMPT = (
    "You are a logistics analyst. Answer only from the provided shipping schema. "
    "Always cite the vessel MMSI field. Keep answers under 60 words. "
    "If a field is missing, reply 'insufficient data'."
)

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,    # the key flag
    max_model_len=8192,
    gpu_memory_utilization=0.90,
)

params = SamplingParams(max_tokens=96, temperature=0.3)

user_queries = [
    "Where is vessel 374651000 right now?",
    "What cargo type is on vessel 257465900?",
    "Is vessel 636011111 approaching port limits?",
]

def build_messages(q: str) -> str:
    return f"{SYSTEM_PROMPT}\n\nUser: {q}\n\nAssistant:"

requests = [build_messages(q) for q in user_queries]
requests.append(build_messages(user_queries[0]) + " Reword only.")

# First call warms the cache for the shared system prompt.
t0 = datetime.now()
outputs = llm.generate(requests, params)
t1 = datetime.now()

for out in outputs:
    print(out.prompt[-40:], "=>", out.outputs[0].text.strip()[:40])

print(f"wall time with prefix caching: {(t1 - t0).total_seconds():.2f}s")

The first request pays full prefill. Every request that shares the system prefix pays nearly zero prefill for that shared tail. In systems where 500 truncated prompts or a giant few-shot block is shared, prefix caching routinely cuts total request latency by 40–70% because prefill (one pass over every prompt token) is often the dominant phase in high-context workloads.

For the same idea across a cluster of workers, precompute and store prompt embeddings / prefix caches and load them at startup or share them from a cache index (e.g., a Redis-backed key on a stable hash of the prefix). The model-level knob is one flag; the operational win is measuring how much of your traffic shares a prefix.

Lever 4: Multi-GPU Serving — Fit It, Then Parallelize It

When a model’s weights plus KV cache don’t fit on one card, you split the model itself across GPUs using tensor parallelism. Every layer is sliced across TP GPUs, shards exchange activations between layers, and the KV cache is spread across cards, too.

The correct serving setup depends on the constraint:

from vllm import LLM, SamplingParams

# Tens parallelism: 4 × H100 splits the weights and KV across 4 GPUs.
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,        # split model across 4 GPUs
    dtype="bfloat16",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    enable_prefix_caching=True,
)

params = SamplingParams(max_tokens=256, temperature=0.8)
out = llm.generate(
    "Explain how tensor parallelism works in one paragraph:",
    params,
)
print(out[0].outputs[0].text)

Two very different motivations live under “multi-GPU”:

• Fitting. The model is larger than one GPU. Tensor parallelism is mandatory. Latency per request stays roughly the same as a single-GPU model of the same size would have, because each shard only does its slice of work.
• Throughput. The model fits on one card, but you have idle GPUs. You can serve multiple independent vLLM workers behind a load balancer (data parallelism), pushing total requests/sec (RPS) far past what a single card does.

A practical hybrid pattern: use tensor parallelism within a node to fit the model, then replicate the whole worker group behind a round-robin ingress for RPS:

             ┌─────────────────────────────────────────┐
             │   Load balancer (nginx / Ray Serve / ALB) │
             └─────┬────────────────┬─────────────────┬─┘
                   │                │                 │
          ┌────────▼───────┐ ┌──────▼────────┐ ┌─────▼────────┐
          │ Worker 0       │ │ Worker 1       │ │ Worker 2      │
          │ TP=2 (2 GPUs)  │ │ TP=2 (2 GPUs)  │ │ TP=2 (2 GPUs) │
          │ vLLM serve     │ │ vLLM serve     │ │ vLLM serve    │
          └────────┬───────┘ └──────┬────────┘ └─────┬────────┘
                   │                │                 │
          (GPU 0,1)          (GPU 2,3)         (GPU 4,5)

The math on why tensor parallelism lowers latency: per token, the workload on each GPU is (per-token compute) / TP. All-reduce between GPUs adds a constant, so the per-token latency scales down, but only when TP is the right granularity. Over-provisioning tensor parallelism (TP=8 for a model that fits in TP=2) adds communication and usually worsens latency. For batch serving, replicate workers rather than over-TP.

When to choose which, in one rule:

• Model doesn’t fit → tensor parallelism (tensor_parallel_size), minimal TP needed (2, 4) and add replication only if RPS demands it.
• Model fits but GPUs idle → more replicas, never more tensor parallelism.

Measuring What a Good Serving Setup Actually Is

You cannot tune what you do not measure. vLLM exposes a Prometheus endpoint (--enable-metrics) that surfaces exactly the numbers you need:

• TPOT (Time Per Output Token) — the per-token speed of generation. This is the number users actually feel.
• TTFT (Time To First Token) — prefill latency. This is where bad setups look terrible for long prompts.
• Throughput (tokens/sec) — how much aggregate generation the system sustains.
• GPU KV cache hit rate — the share of KV blocks served from cache. This is your prefix-caching health check.
• Request queue time and batch size — how well continuous batching is absorbing bursts.

A quick probe against the OpenAI-compatible endpoint:

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

def time_ttft_and_tpot(prompt: str, max_tokens: int = 128):
    t0 = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,                      # stream so we can measure TTFT separately
    )

    first = True
    t_ttft = None
    for chunk in stream:
        if first:
            t_ttft = time.perf_counter() - t0
            first = False

    t_total = time.perf_counter() - t0
    tpot = (t_total - t_ttft) / max_tokens
    return {"ttft": t_ttft, "tpot": tpot, "total": t_total}

print(time_ttft_and_tpot("Explain continuous batching:", 96))

Watch the ratio: if tpot climbs as concurrency rises, generation is the bottleneck and you add replica workers. If ttft climbs instead, prefill / queueing is the bottleneck and prefix caching, tighter max_model_len, or higher prefill capacity is the fix.

Putting It Together: A Small Serving Entrypoint

A minimal but production-shaped config-driven entrypoint:

pip install vllm prometheus-client

import os
from dataclasses import dataclass
from vllm import LLM, SamplingParams

@dataclass
class ServeConfig:
    model: str = os.getenv("MODEL", "meta-llama/Meta-Llama-3.1-8B-Instruct")
    tensor_parallel_size: int = int(os.getenv("TP_SIZE", "1"))
    max_model_len: int = int(os.getenv("MAX_MODEL_LEN", "8192"))
    gpu_util: float = float(os.getenv("GPU_UTIL", "0.92"))
    kv_cache_dtype: str = os.getenv("KV_CACHE_DTYPE", "auto")   # or fp8_e5m2
    quant: str | None = os.getenv("QUANTIZATION")               # awq / gptq / None
    prefix_cache: bool = os.getenv("PREFIX_CACHE", "1") == "1"

config = ServeConfig()

llm = LLM(
    model=config.model,
    tensor_parallel_size=config.tensor_parallel_size,
    max_model_len=config.max_model_len,
    gpu_memory_utilization=config.gpu_util,
    kv_cache_dtype=config.kv_cache_dtype,
    quantization=config.quant,
    enable_prefix_caching=config.prefix_cache,
)

params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=int(os.getenv("MAX_TOKENS", "256")),
)

def serve_batch(prompts: list[str]) -> list[str]:
    return [out.outputs[0].text for out in llm.generate(prompts, params)]

if __name__ == "__main__":
    for text in serve_batch(
        ["Write a haiku about KVS caches:", "Explain paged attention in one line:"]
    ):
        print("-", text)

That single config block encodes all four levers so the deployment can be tuned without code changes.

The Levers to Pull, in Order

If you’re starting from a naive transformers loop, the order of wins is usually:

1. Move to vLLM (PagedAttention + continuous batching) — the biggest single win, no accuracy cost.
2. Turn on prefix caching when traffic shares long prompts — free latency win.
3. Quantize (weights to INT4/AWQ, KV cache to FP8) when memory is the limiter — enables fitting bigger models on fewer cards and more concurrency per card.
4. Multi-GPU: tensor parallelism to fit, then replicas for throughput — only when the others aren’t enough.

Each lever attacks a specific bottleneck: fragmentation, redundant prefill, weight/KV memory, and single-card capacity. Pull them in that order, measure TTFT and TPOT along the way, and you go from “the server falls over at five concurrent requests” to “the server sustains the full production batch.”

The model weights were never the hard part. The serving infrastructure is.