Latency is a feature, not a tax

Placeholder post — replace with real copy. Kept here to verify the post template and list view render correctly.

When I first benchmarked an inference stack, I did what everyone does: I maximized throughput. Tokens per second, batched to the gills, GPU utilization pinned near 100%. The number went up and I felt good about it.

The number was measuring the wrong thing.

A user waiting on a chat response does not experience your aggregate throughput. They experience time to first token, and then the steadiness of the stream after that. A stack that does 4,000 tok/s aggregate but makes one person wait 900ms to see anything feels slower than a stack at half the throughput with a 180ms first token.

The number I track now

Time to first token at p95, under realistic concurrency — not the median on an idle box. It’s a worse marketing number and a much better product one. Once I started optimizing for it, a few “obvious” wins (bigger batches, longer scheduling windows) turned out to be actively hurting the experience I was supposedly building.