LLM Inference Benchmark
completeA head-to-head benchmark of two LLM inference platforms, same model and same GPU, plus a log of everything that broke on the way there.
Background
I recently got the chance to interview at one of the best ML inferencing platforms in the world (Simplismart) and the experience made me very curious about the inference stack that is responsible for how the world uses AI on a daily basis.
Ultimately I ended up doing a LLM inferencing benchmarking exercise across the top inference providers as part of the interview process, and the experience gave me many insights which I wanted to share here.
ML inferencing is ultimately about designing and using systems that vend AI output to downstream consumers. As we think about it more, it becomes clear that inferencing analytics is a very in demand vertical of the AI stack right now as we see companies scurrying across the board to maximise their token efficiencies and costs.
Companies like Simplismart and Fireworks are at the forefront of this race. MLOps has become a very common use case for all businesses and these companies promise to abstract the details for this problem while providing maximum reliability and customisability with all the latest and popular open source models.
So what separates one LLM inference platform from another once you get past the pricing page? Everybody quotes tokens per second but what does that even mean or feel like, and is that the only way to measure your inference performance and how it trickles down to the customer experience?
To pick all this apart, I used the free credits given by Simplismart and Fireworks AI, put the same model (Gemma 3 4B Instruct) on the same class of GPU (a dedicated H100 endpoint) on both, pointed the same client code at them, and ran ten scenarios in one sitting: 130 requests across different prompt types, sizes and situations with everything torn down inside the hour.
What the numbers said
Mean time to first token
The gap sits entirely before the first token
- Simplismart
- Fireworks AI
1 request in flight E01 / E02 vs E06 / E07
5 in flight E03 / E04 vs E08 / E09
10 in flight E05 vs E10
Gemma 3 4B Instruct · 1× dedicated H100 on each platform · 130 requests, 100% success
The headliner of the analysis is a 7× gap in time to first token (TTFT) (for a single request) where Simplismart blows Fireworks out of the water. The time per output token was basically identical between both platforms, which makes sense as the same weights alongside the same chip will result in the same steady-state generation from GPU. TTFT is thus a great indicator on the specific performance of the host inference engine.
Mean time per output token
Once the stream is running, it's the same machine
- Simplismart
- Fireworks AI
Simplismart
4.19–4.97ms
Fireworks AI
4.59–6.16ms
ms per token
Ten scenarios, five per platform, every one between 4.19 and 6.16 ms. For scale: the time-to-first-token gap at one request in flight is 879 ms which about 126 times this entire axis, off the right edge.
As you can see above, once the tokens start flowing for the output, there’s not much separating the two platforms.
So what could this difference in TTFT mean for an end consumer? Run the widget below to visually feel the difference. I encourage you to also try playing around with the numbers below to see how a bad inferencing setup can manifest into consumer frustration:
Platform A 210 tok/s
Platform B 210 tok/s
The second finding was that I’d have got the ranking wrong if I’d stopped at one concurrency level.
Output tokens per second
The ranking flips once the endpoint is busy
- Simplismart
- Fireworks AI
1
5
10
Concurrent requests in flight
At 10 concurrent requests with 200-token generations, Fireworks overtakes. The only scenario in the run where it leads on raw throughput.
Simplismart wins comfortably at one and five requests in flight. At ten, with longer generations, Fireworks overtakes it, batching more efficiently once its fixed overhead is amortised across more simultaneous requests. That makes things more complicated than before. Since most output token requests are batched in real life scenarios, Fireworks now makes a strong case for choosing it over Simplismart.
Third: on dedicated endpoints, the per-token pricing doesn’t matter as much as the time you choose to reserve GPUs for yourself. Fireworks costs twice as much per token and my total bill came out within 2% of Simplismart’s because GPU-hours dominate everything at this scale and I ran good teardown scripts which stopped provisioning GPUs for me on both platforms the moment I finished my test. So having good reserving logic seems to pay huge dividends when designing how you want to do your inferencing.
Cost
2× on the rate card, 2% on the bill
What you compare
Per million output tokens
Fireworks costs 2× as much
What you pay
Total spend, this benchmark
The bills land within 2%
Both endpoints were a dedicated H100, live for well under an hour each. At this scale the GPU hours are the bill: the token volume costs a fraction of a dollar either way. The two panels use separate scales.
The things I learnt along the way
I tried setting up a purely agentic pipeline to perform and collect results for this experiment, so doing that and encountering the roadblocks in the process turned out to be a key learning area as well. My agent surfed the web, scraped web pages and scoured the platforms’ documentation on how to proceed and set this all up.
I also instructed it to create logs of all friction points and document areas where it could not proceed on its own to understand what is missing from these platforms and their documentation.
Thirteen of those friction points went into a log, and the pattern across them is very interesting as well:
- Agentic SEO is a real asset for any company which wishes to be AI native. Adding linked markdown files and good XML skeletons really help agents discover, use and suggest your platform to unknown customers.
- The agentic experience uncovered gaps in the Simplismart’s documentation where they did not explicitly mention the GPU types they have or how to search for them so that they can be used for other key SDK functions.
- Even though I had free credits from registering on their platform, Fireworks had gated custom GPU deployments behind a “Add Credit Card” paywall. That didn’t make any sense and it cost me some churn because “we have credits, why is this failing” isn’t a question an agent knows to ask or answer unless it has been explicitly trained. I had to go look at the billing page.
None of these are bugs exactly. They’re cases where the documented surface and the real surface have drifted apart, and you only find them by using their services and APIs from scratch. Ironically, using agents exclusively for this project gave me insights on how these companies can improve the human facing product experience.
What you should take away from this
- It’s worthwhile to learn the ins and outs of inferencing in today’s world. That includes knowing the tradeoffs involved in using different kinds of GPUs to use, MLOps frameworks like vLLM and optimisation techniques like quantisation.
- You should seriously consider setting up your inference analytics stack and being able to adapt it rapidly. Ever decision after that becomes so much easier for any use case (for example: How well are we using our enterprise budget when comparing token quality vs quantity? Will building this feature for our AI product kill our speed/budget? Is the Ops pipeline for our AI product setup effectively?)
- Everybody knows data analytics and SQL but the same will eventually apply for inference analytics in a few years. It won’t be surprising to see data and business intelligence engineers pick this up as part of their job (or have new job roles open up for this).
- Conducting experiments researching such things has never been easier. I did all this using claude code and I learnt much more than I would have than if I did it manually reading documentation.
- Open source models are actually very cool and versatile. We all use ChatGPT and Claude, but you should consider purchasing your next machine (or setting up your current one) so that it can atleast run ~10B parameter model via tools like Openclaw, Odysseus and Ollama.
Stack
Python, asyncio, the OpenAI streaming client, pandas and matplotlib for the report, the Simplismart SDK and the Fireworks REST API for deployment. Built with Claude Code, with a full write-up, a friction log, and a second independent run to check the first one’s accuracy.
Check it out!
You can reproduce the results for yourself by following the reproduction guide given in the repo link at the top of this page.
Happy to hear thoughts and critical feedback on this. Please feel free to write to athreya.shreyas@gmail.com.