Projects

Simplismart inference benchmark

complete

A head-to-head inference benchmark of Simplismart vs Fireworks AI — where the friction, not the numbers, was the finding.

A benchmark comparing inference performance and the agentic developer experience of Simplismart and Fireworks AI, built as an Associate PM hiring task. Same model (Gemma 3 4B Instruct), same GPU class (dedicated H100 80GB), the benchmark code identical across both with only a base URL and key swapped.

The numbers were never the point

The task looked like a performance benchmark. The actual product question was: how well do these platforms support developers building agentic, autonomous workflows? So the real output isn’t the latency tables — it’s AGENTIC_LOG.md, a real-time record of every friction point hit while trying to deploy and benchmark both platforms entirely programmatically. What could be scripted, what forced a manual UI step, what failed with no useful error.

The sharpest finding: the advertised set of valid GPUs didn’t match the actually-provisionable set. A100 80GB was a valid accelerator_type in Simplismart’s SDK enum and its deployment UI — then compilation failed with HTTP 400: "gpu a100: need 1 but only 0.0 available". The account had zero A100 quota, with no way to discover that short of attempting the deploy. That mismatch between SDK enums / pricing pages / UI dropdowns and what an account can really provision — surfaced as a first-class friction point — is what actually determined the final H100-on-Gemma-3-4B configuration. Not a search for the cheapest card.

Why Gemma 3 4B, deliberately

I explicitly avoided Llama 3.1 8B — Simplismart has a step-by-step deployment blog for it, and using it would have turned an independent product evaluation into a tutorial playthrough that hid exactly the discovery experience the task was meant to probe. Gemma 3 4B was the first model confirmed deployable on H100 on both platforms, cheap enough to stay under a $5 credit ceiling, and small enough (~8GB on an 80GB card) that GPU choice has effectively no bearing on the latency numbers.

Full write-up, tables, charts, and a run-to-run validation against an independent redeployment are in the project’s report/REPORT.md.