inferencecanary: Conformance eval suite for inference providers

All cells probed live · latest probe 2026-08-29 · model gemma-4-31b · how grading works

A provider has one job: surface the model's capabilities to application developers. The model's capabilities are defined by its chat template. Everything the template defines and the provider doesn't deliver - under the best dialect they offer - is documented here, with a reproducible probe behind every grade.

gemma-4-31b

Read findings →

How grading works

Grades are computed, not opined. Every flag is a documented defect with live evidence and a stated repro; the letter is arithmetic over the flags. This section is the rubric; each finding documents the probe that produced it.

Axes

• Inference - does the provider's serving layer deliver what the chat template defines? Reasoning carried across tool-call turns, images in tool results, tool_choice actually enforced, tool results paired by id, arguments delivered byte-exact, honest finish_reason. Letter grade, see ladder below.
• Billing - do reported token counts match what was sent and received? Binary: ✓ or Fail. Metering errors in the customer's favor are footnoted on the provider's card, not flagged.
• Caching - does the provider reuse context it has already seen, instead of re-reading (and re-billing) it on every call? Measured by timing a cold read of a fresh ~19K-token agentic transcript against a warm repeat one second later. Three answers: Yes (reused every trial), Sometimes (some trials), No (never). A Fail - a high-severity issue - where the repeat submission itself waits out a per-minute token quota before it is served and no paid tier exists to raise it - context reuse is unobservable behind the wait, and there is no quota to buy.
• 20K-token agentic turn - what one measured call - 20,000 context tokens in, 256 out - costs. Speed is that call in seconds under best caching conditions; where caching is Sometimes, both bounds: cache-hit (lower) to no-hit (upper). Full speed measurements on the speed page. Cost - the same call in cents at list price, derived the same way: where caching is Sometimes and the provider prices cached reads cheaper, both bounds. A cache hit only changes the price where a cached-read rate exists - a hit billed as uncached costs full price. TBD where no price is on record. A Fail - a high-severity issue - where the provider sells no paid tier for the model at all - there is no price at which the turn can be bought.

The inference ladder

Every provider starts at A. Each high-severity flag - a defect that breaks or corrupts a real application - subtracts a full letter. Each medium-severity flag - a defect that degrades an application or violates the dialect contract with a workaround - subtracts half a step.

Read findings →