SambaNova bills identical requests inflated token counts

red · billing · probed live 2026-07-26 · affects SambaNova · run the repro yourself

The same request body, sent repeatedly, is not billed the same number of prompt tokens — and the errors overwhelmingly point up. Across three independent 40-call runs, SambaNova charged 9,176 tokens for 4,200 actually sent: 2.18× on average. The inflated figure is carried in total_tokens — the field an invoice sees — so this is what a customer pays, not a cosmetic reporting glitch.

What was measured

One request body — a 35-token prompt, byte-identical on every send — was sent 40 times in sequence, three separate times on 2026-07-26. A correct meter bills identical bytes identically, every time. Instead:

RunBilled for 1,400 sentOverchargeWrong values seen
12,8102.01×27, 67, 221, 256, 459, 590
21,9271.38×27, 578
34,4393.17×27, 647, 2478

13 of the 120 calls (11%) were billed a wrong value; ten distinct values appeared, from 27 to 2,478 tokens, for one unchanging 35-token request.

The money is extra, not shuffled

An inflated count could in principle be another request's count, swapped — annoying but zero-sum. It is not. Proof by a closed pair: two requests that are both ours, fired together, so every token entering and leaving the set is known. A 74-token request co-fired with a 1,197-token request was billed 1,197 on both — the small request took the large one's count while the large one kept its own. The pair was billed 2,394 tokens for 1,271 sent: an excess of 1,123 tokens that nobody sent. Totals do not conserve; the overcharge is genuine extra billing.

Mechanism — our working hypothesis

Cross-batch attribution: SambaNova serves requests in batches, and one batch member's token count gets stamped onto another member's bill. Supporting evidence: (a) the wrong value can be chosen in advance — co-fire a small request with a large one of known size and the small one is billed the large one's exact count; (b) co-fired responses return bit-identical time_to_first_token and end_time — direct proof that batch-level values are stamped onto every member; (c) the occasional under-bill is what being co-batched with a smaller neighbor looks like; (d) the wrong values in sequential runs (27…2,478) look like a sample of other customers' prompt sizes. The limit of the claim: foreign traffic is unobservable, so for sequential runs the neighbor's identity is inferred from consistency with the controlled co-fire experiment, not directly observed.

Why it matters

Spend on this provider cannot be predicted, reconciled, or attributed. A cost-control layer that meters against reported usage will mis-bill by multiples — and because the wrong values track other traffic, the error grows with how busy the endpoint is, so it is largest exactly when spend is highest.

Run it yourself

The repro is a standalone script with no dependencies beyond Node: one loop that sends the same request 40 times and prints what each call was billed. Save it as sambanova-overcharge-repro.ts and run SAMBANOVA_API_KEY=<key> npx tsx sambanova-overcharge-repro.ts.

sambanova-overcharge-repro.ts
/**
 * SambaNova over-billing repro.
 *
 * Sends the exact same request N times, one after another, and prints the
 * number of prompt tokens SambaNova billed for each call. A correct meter
 * bills identical bytes identically, every time. Seeing more than one value
 * below proves the meter is broken; the totals at the end show the overcharge.
 *
 * Run: SAMBANOVA_API_KEY=<key> npx tsx sambanova-overcharge-repro.ts
 */

const API_KEY = process.env.SAMBANOVA_API_KEY;
if (!API_KEY) {
  console.error("Set SAMBANOVA_API_KEY first.");
  process.exit(1);
}

const N = 40;

// This exact body is sent every time. Nothing about it ever changes.
const REQUEST_BODY = JSON.stringify({
  model: "gemma-4-31B-it",
  messages: [{ role: "user", content: "What is 6 times 7? Reply with just the number." }],
  temperature: 0,
  max_tokens: 5,
  chat_template_kwargs: { enable_thinking: false },
});

const billed: number[] = [];

for (let i = 1; i <= N; i++) {
  const res = await fetch("https://api.sambanova.ai/v1/chat/completions", {
    method: "POST",
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" },
    body: REQUEST_BODY,
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
  const json = (await res.json()) as { usage: { prompt_tokens: number; total_tokens: number } };
  billed.push(json.usage.prompt_tokens);
  console.log(`call ${String(i).padStart(2)}/${N}: billed prompt_tokens = ${json.usage.prompt_tokens} (total_tokens = ${json.usage.total_tokens})`);
  await new Promise((r) => setTimeout(r, 1200));
}

// The request's real size is whatever value the meter reports most often.
const counts = new Map<number, number>();
for (const b of billed) counts.set(b, (counts.get(b) ?? 0) + 1);
const trueCount = [...counts.entries()].sort((a, b) => b[1] - a[1])[0]![0];

const distinct = [...counts.keys()].sort((a, b) => a - b);
const wrongCalls = billed.filter((b) => b !== trueCount).length;
const totalBilled = billed.reduce((a, b) => a + b, 0);
const totalSent = trueCount * N;

console.log("------------------------------------------------------------");
console.log(`the same request was sent ${N} times`);
console.log(`distinct billed values: ${distinct.join(", ")} (a correct meter shows exactly one)`);
console.log(`the request's real size: ${trueCount} tokens`);
console.log(`calls billed a wrong value: ${wrongCalls} of ${N}`);
console.log(`tokens billed in total: ${totalBilled} — tokens actually sent: ${totalSent}`);
console.log(`overcharge: ${(totalBilled / totalSent).toFixed(2)}x`);

A captured run — the 3.17× one from the table above, complete:

output · 2026-07-26
call  1/40: billed prompt_tokens = 35 (total_tokens = 37)
call  2/40: billed prompt_tokens = 35 (total_tokens = 37)
call  3/40: billed prompt_tokens = 35 (total_tokens = 37)
call  4/40: billed prompt_tokens = 35 (total_tokens = 37)
call  5/40: billed prompt_tokens = 35 (total_tokens = 37)
call  6/40: billed prompt_tokens = 35 (total_tokens = 37)
call  7/40: billed prompt_tokens = 35 (total_tokens = 37)
call  8/40: billed prompt_tokens = 35 (total_tokens = 37)
call  9/40: billed prompt_tokens = 35 (total_tokens = 37)
call 10/40: billed prompt_tokens = 35 (total_tokens = 37)
call 11/40: billed prompt_tokens = 647 (total_tokens = 649)
call 12/40: billed prompt_tokens = 35 (total_tokens = 37)
call 13/40: billed prompt_tokens = 35 (total_tokens = 37)
call 14/40: billed prompt_tokens = 35 (total_tokens = 37)
call 15/40: billed prompt_tokens = 35 (total_tokens = 37)
call 16/40: billed prompt_tokens = 35 (total_tokens = 37)
call 17/40: billed prompt_tokens = 35 (total_tokens = 37)
call 18/40: billed prompt_tokens = 35 (total_tokens = 37)
call 19/40: billed prompt_tokens = 35 (total_tokens = 37)
call 20/40: billed prompt_tokens = 35 (total_tokens = 37)
call 21/40: billed prompt_tokens = 35 (total_tokens = 37)
call 22/40: billed prompt_tokens = 27 (total_tokens = 29)
call 23/40: billed prompt_tokens = 35 (total_tokens = 37)
call 24/40: billed prompt_tokens = 35 (total_tokens = 37)
call 25/40: billed prompt_tokens = 35 (total_tokens = 37)
call 26/40: billed prompt_tokens = 27 (total_tokens = 29)
call 27/40: billed prompt_tokens = 35 (total_tokens = 37)
call 28/40: billed prompt_tokens = 35 (total_tokens = 37)
call 29/40: billed prompt_tokens = 35 (total_tokens = 37)
call 30/40: billed prompt_tokens = 35 (total_tokens = 37)
call 31/40: billed prompt_tokens = 35 (total_tokens = 37)
call 32/40: billed prompt_tokens = 35 (total_tokens = 37)
call 33/40: billed prompt_tokens = 35 (total_tokens = 37)
call 34/40: billed prompt_tokens = 35 (total_tokens = 37)
call 35/40: billed prompt_tokens = 35 (total_tokens = 37)
call 36/40: billed prompt_tokens = 35 (total_tokens = 37)
call 37/40: billed prompt_tokens = 2478 (total_tokens = 2480)
call 38/40: billed prompt_tokens = 35 (total_tokens = 37)
call 39/40: billed prompt_tokens = 35 (total_tokens = 37)
call 40/40: billed prompt_tokens = 35 (total_tokens = 37)
------------------------------------------------------------
the same request was sent 40 times
distinct billed values: 27, 35, 647, 2478 (a correct meter shows exactly one)
the request's real size: 35 tokens
calls billed a wrong value: 4 of 40
tokens billed in total: 4439 — tokens actually sent: 1400
overcharge: 3.17x

Provider response

Reported to SambaNova (community thread) — The response did not address the defect. Reproducible as of 2026-07-26.

Have a response or a correction? [email protected] — it becomes part of this record.