# Speed — what a real agentic call costs in wall-clock

Measured live 2026-07-27 · model gemma-4-31b · point-in-time, under that evening's load.
HTML version: [https://inferencecanary.com/speed/](https://inferencecanary.com/speed/) · matrix: [https://inferencecanary.com/index.md](https://inferencecanary.com/index.md)

What a real agentic call — 20,000 tokens of context in, 256 tokens out — costs in wall-clock,
per provider. Two things dominate it: whether the provider **reuses context it has already
seen**, and how fast it **reads uncached context**. The generation speed providers advertise
barely matters at agentic context sizes; the spread between working providers is roughly 13×.

| Provider | Caching | Floor (ms) | Floor high-water (ms) | Read (tok/s) | Write (tok/s) | TTFT @20K | Turn, cold | Turn, hit |
|---|---|---|---|---|---|---|---|---|
| [Cerebras](https://inferencecanary.com/cerebras/gemma-4-31b/index.md) | Yes (3/3 hits) |197 | 197 | 13.9K | 1855 | 1.6s | 1.8s | 0.7s |
| [Lightning](https://inferencecanary.com/lightning/gemma-4-31b/index.md) | Yes (3/3 hits) |186 | 186 | 21.5K | 339 | 1.1s | 1.9s | 1.2s |
| [AWS Bedrock (Chat Completions)](https://inferencecanary.com/aws/gemma-4-31b/index.md) | No (0/8 hits) |385 | 406 | 12.2K | 198 | 2.0s | 3.3s | — |
| [AWS Bedrock (Responses)](https://inferencecanary.com/aws-responses/gemma-4-31b/index.md) | No (0/8 hits) |363 | 371 | 6.8K | — | 3.3s | — | — |
| [SambaNova](https://inferencecanary.com/sambanova/gemma-4-31b/index.md) | No (0/8 hits) |1255 | 1681 | 8.6K | 189 | 3.6s | 4.9s | — |
| [Together AI](https://inferencecanary.com/togetherai/gemma-4-31b/index.md) | Sometimes (2/3 hits) |284 | 284 | 6.2K | 83 | 3.5s | 6.6s | 3.4s |
| [DeepInfra](https://inferencecanary.com/deepinfra/gemma-4-31b/index.md) | Sometimes (1/3 hits) |418 | 15,071 | 3.7K | 11 | 5.8s | 29.7s | 24.8s |
| [Parasail](https://inferencecanary.com/parasail/gemma-4-31b/index.md) | Sometimes (1/3 hits) |289 | 289 | 2.5K | 15 | 8.3s | 25.1s | 17.4s |
| [Novita](https://inferencecanary.com/novita/gemma-4-31b/index.md) | No (0/8 hits) |439 | 70,035 | 3.5–6K | 9–12 | ~4–6s | ~30s | — |
| [Google (native)](https://inferencecanary.com/google/gemma-4-31b/index.md) | Fail |917 | 917 | 22.9K | 5.7 | 1.8s | 47s | — |
| [Google (OpenAI-compat)](https://inferencecanary.com/google-openai/gemma-4-31b/index.md) | Fail |804 | 804 | 25.8K | 5.6 | 1.6s | 48s | — |

## What the columns mean

The measurement: a fresh ~19K-token agentic transcript is read cold, then again warm one
second later. A unique marker makes the cold read provably unseen, so a warm answer arriving
under 3× the provider's floor can only be cache reuse. TTFT = time to first token.

- **Caching** — is already-seen context reused? Yes: every trial. Sometimes: some trials.
  No: never.
- **Floor** — TTFT of a tiny request. Pure overhead: network, queue, scheduling.
- **Floor high-water** — the worst floor ever observed. How long a trivial request can pend
  under load.
- **Read** — how fast uncached context is ingested.
- **Write** — how fast new tokens are generated.
- **TTFT @20K** — estimated wait before the first token over 20,000 uncached context tokens.
  Floor + 20,000 ÷ read.
- **Turn, cold** — TTFT @20K plus 256 written tokens. One full uncached call.
- **Turn, hit** — the same call when the cache hits. Empty where there is no cache to hit.

## Notes

- **AWS Bedrock (Responses)** — Write speed unmeasured: the streamed usage report on this API carried no completion-token count. The sibling AWS chat endpoint writes 198 tok/s on the same backend.
- **Novita** — Least trustworthy estimates on the sheet: within single runs, cold reads of the identical context ranged 9.7–32.6s and one warm call pended 40.1s while tiny requests answered in 0.4s — large calls queue when small ones don't.
- **Google (native)** — The fastest read on the sheet feeds the slowest write: 20K tokens prefill in ~0.9s, then 256 output tokens take ~45s at 5.7 tok/s. And a second 20K submission waited 45.9s in quota 429s before being served (measured 2026-07-28, 14 attempts) — gemma-4 has no paid tier, so consecutive agentic turns pay that wait with no upgrade path. Any cache reuse is unobservable behind it; no cached-token count is disclosed.
- **Google (OpenAI-compat)** — Same backend and quota as the native lane: ~0.8s to prefill 20K tokens, ~46s to write 256 at 5.6 tok/s, and a repeat 20K submission waited 49.2s in quota 429s before being served (measured 2026-07-28, 15 attempts). gemma-4 has no paid tier, so there is no way to buy that wait down.

Speeds are point-in-time under real load, not a benchmark under ideal conditions — read speed
on one provider moved from 7.0K to 12.2K tok/s between same-night runs. The caching class per
provider also appears on the grade matrix: [https://inferencecanary.com/index.md](https://inferencecanary.com/index.md).