Google (native)reference

generateContent — Google's own protocol · serves gemma-4-31b-it as gemma-4-31b · probed live 2026-07-28 · back to the matrix

This is Google's native API. Google's OpenAI-compat shim is graded separately as its own provider: google-openai.

Harness

B−

A − 1 red − 1 yellow, per the ladder; flags below.

Billing

✓

Reported usage consistent with bytes on the wire.

Caching

Fail

A repeat 20K-token submission waits out the free tier's per-minute token quota (~46s of 429 retries) before it is served — and gemma-4 has no paid tier, so there is no quota to buy. Whether context is reused is unobservable behind the wait. Speed numbers →

Faithfulness

reference

This provider is the reference implementation the battery compares against.

Flags

red pricing Google sells no paid tier for gemma-4: every request is used to improve their products, and quota gates agentic work
Google's own price sheet lists gemma-4's paid tier as "Not available" on every line — the free tier is the only tier — and carries the row "Used to improve our products: Yes" with no paid alternative. The Gemini API...
yellow harness Tool results replayed out of order are mis-paired on most providers
The chat dialect's contract is that a tool message is matched to its call by tool_call_id, in any order. After two parallel calls, replaying the results reversed swaps the data between them on these providers — the...

Every flag links to its finding — evidence, repro, and disclosure records live there; the deepest findings have full writeup pages.