Honest Math for LLM Serving

Ask a free GPU calculator how much traffic your model can serve and it hands you a confident number. The number is usually a lie, because it counts completions per second a fast small model can never sustain, so it reports headroom that evaporates the instant real users arrive and the KV cache fills. I built Backpressure to report the honest version instead: the batch slots the cache can really hold, the point where the queue stops absorbing load and starts shedding it, the request rate where p99 breaks the SLO.

Backpressure is a load simulator you drive from a canvas. You wire a system out of catalog parts (a load source, a serving tier, a model, a GPU, a queue, a database), push simulated concurrent traffic through it, and watch each node hold or fall over on live health bars: queue depth, time to first token, tokens per second, p99, error budget. At the bottom it names the bottleneck in one line. The numbers have to be real enough to trust. That was the hard part.

The number free calculators fake

A serving rig does not have a requests-per-second capacity. It has a number of concurrent slots. Weights and framework overhead come off the VRAM first, whatever is left is the KV budget, and each in-flight request rents a slice of that budget for the length of its context. Work it through for a 24 GB card running a 7B model at an 8k context:

weights      = 7B x 4.89 bits       = 4.3 GB
KV budget    = 24 - 4.3 - 2 (fw)    = 17.7 GB
KV / request = 0.017 MB x 7 x 8192  = 0.95 GB
slots        = floor(17.7 / 0.95)   = 18

Eighteen. Not thousands. Two calibration constants decide that number. The first is the KV cost per token: 0.017 MB per billion parameters, the value for grouped-query attention. The old textbook 0.07 came from the multi-head attention that GQA replaced, and it overstates capacity fourfold. The second is the quant: a GGUF “4-bit” model really costs about 4.89 bits per weight, since each block stores a scale and a minimum on top of the weights. Cost it at 4.0 and you undercount VRAM by 6 to 18 percent, then hand back slots the card cannot hold.

The first version of Backpressure made this exact mistake. It reported capacity as a completion rate, and in one run 5000 users sat at 76 percent utilization and “survived” a box that was really dropping them. I keep the audit of that collapse in the repo. The fix was to stop reporting completions per second and report slot occupancy, the thing the hardware is actually rationing.

What backpressure actually is

Once offered load passes the slots, the excess has to go somewhere, and choosing where is the whole game. Each node runs one of three queue disciplines:

• shed drops the overflow immediately and fails fast. This is the default.
• buffer holds up to a max depth and abandons anything older than its max age.
• block refuses new work at the intake, so the pressure travels back up the graph. This is the mechanic the tool is named after.

Whatever a node cannot serve this tick becomes backlog for the next one. Requests that wait too long are dropped, and the time they spent waiting is added to their latency. Under heavy overload the queue serves the newest requests first, an idea from CoDel, the algorithm built to fight bloated network queues. The oldest requests have usually missed their deadline already, so serving them would waste capacity a fresh one could use. I have written before about reading queue depth from the router’s seat. This is the same number from the other side, the queue as something you design on purpose.

Two ceilings, not one

A model with slots to spare can still crawl, because there are two ceilings and you hit the lower one. The first is the KV batch cap from earlier: how many requests fit in memory at once. The second is decode speed, and it is limited by memory bandwidth. Every token the model generates has to read the active weights out of VRAM, so one stream runs about as fast as the memory bandwidth divided by the size of those weights. Your real capacity is whichever ceiling is lower.

Mixture-of-Experts shows how far the two can drift apart. The VRAM you need is set by the total parameter count. Decode speed is set by the active count. Mixtral 8x7B has 47B parameters sitting on the card but only 13B active per token, so it needs the memory of a large model and runs at the speed of a small one. A calculator that models one ceiling and not the other will overpromise.

Making it break honestly

A simulator that only reports steady state is a calculator with animations. The failure curve is the interesting part. Latency does not rise in a straight line with load. It follows the M/M/1 queue curve, where wait time scales as one over one-minus-utilization. At 95 percent utilization that factor is twenty, so the rig runs about twenty times slower than it does idle. This is why “we’re only at 95 percent” is not the reassurance it sounds like.

Utilization is not the only thing that can fail a design. TTFT is held to a hard 1500 ms budget, the point where an interface stops feeling live. Duration is its own axis: a node pinned at 85 percent for two minutes fails a soak test even if it never overloads, because sustained heat breaks things a brief spike does not.

The model is also honest about what it cannot do exactly. When traffic is bursty, a code comment admits the shape is only an approximation of real self-similar bursts, not true fractional Brownian motion. And where the math is least sure of itself, it widens the error band instead of faking precision: plus or minus 20 percent for a mainstream card serving a 7-to-34B model, out to plus or minus 50 percent when the model barely fits or runs on CPU.

Determinism is the product

The engine has no randomness it does not control. One seed pins an entire run to byte-identical ticks, and the only randomness comes from a small seeded generator, mulberry32. That makes a simulation a reproducible artifact: you can save it, share it, and reopen it to the exact same result.

That reproducibility pays for the trust model almost for free. When a ranked scenario or a leaderboard entry has to be believed, the server re-runs the exact same simulate function and compares, rather than trusting whatever the client reported. Recomputing is verifying. A frozen golden run, a load into an app into Postgres that collapses at 1500 rps, guards against floating-point drift between the JavaScript engines (V8, JSC, and Bun), so a quiet recalibration that would change a verdict cannot slip through.

The limits I keep on purpose

Backpressure does not simulate closed models. No public weights, no honest math, and a confident number with nothing under it is worse than no number at all. The calibration I can defend (GQA KV cost, GGUF effective bits, MoE active parameters, unified memory on Apple silicon) is exactly the calibration the free tools get wrong. That gap is the product, not a footnote.

One tension I have not resolved, and will not pretend I have. The design doc still says the simulation should never ship to a client. Today it runs in the browser anyway, because an instant canvas is worth more than the principle. Whether to add a server-authoritative path later is a decision I have left open on purpose.

Every serving calculator hands you a number. Backpressure hands you the one that matters: the request rate where it breaks, before real users find it for you.

Build a rig and watch it break →