SambaNova ships raw chain-of-thought as the answer when truncation lands mid-thought

medium · inference · probed live 2026-08-02 · affects SambaNova

With thinking enabled, the model emits its reasoning first, closes the thought channel with a template marker, then writes the answer. SambaNova only separates the two when that closing marker actually arrives: if max_tokens cuts generation off mid-thought, the entire raw thought channel — opening marker included — is returned in content, the field applications display as the answer, while the reasoning field comes back empty. The answer itself never arrives.

What was sent

An arithmetic prompt with thinking on, temperature: 0, and a max_tokens budget of 60 — small enough that the cut lands inside the thought (download):

{
  "model": "gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": "A shop sells pens at 17 cents each. I buy 23 pens and pay with a $5 bill. How many cents do I get back? Answer with the number, then explain the arithmetic in one or two sentences."
    }
  ],
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "temperature": 0,
  "max_tokens": 60
}

What came back

HTTP 200, finish_reason: "length" — and this choices[0], verbatim:

{
  "finish_reason": "length",
  "index": 0,
  "logprobs": null,
  "message": {
    "content": "<|channel>thought\n*   Item: Pens\n    *   Price per pen: 17 cents\n    *   Quantity bought: 23 pens\n    *   Payment: $5 bill\n    *   Goal: Calculate change in cents.\n\n    *   Total cost = Price per",
    "role": "assistant"
  }
}

content opens with the literal template marker <|channel>thought followed by the unfinished chain of thought; there is no reasoning field on the message at all. Three identical runs returned this byte-identically, 3 of 3.

The contrast that isolates it

The same request with max_tokens: 1200 completes and splits cleanly: 641 characters of reasoning in the reasoning field, and "109 — 23 pens at 17 cents each cost 391 cents…" as the answer. The same request at max_tokens 300 and 289 — budgets where the cut lands mid-answer, past the thought — also splits cleanly, 2 of 2: full reasoning in its field, a truncated answer in content, no marker anywhere. Truncation as such is handled correctly. Only the cut that lands inside the thought leaks, and it leaks every time.

How this breaks expectations

content must carry answer text only; template control markers belong to the provider's parser. SambaNova's own clean handling of mid-answer truncation proves the split is implemented — it just isn't applied until the closing marker arrives, so a thought cut short is never recognized as a thought.

Why it matters

Running against a token budget is routine in production, and reasoning inflates output length unpredictably — so this fires under ordinary load, deterministically. When it does, the user-visible answer becomes the model's raw scratchpad: internal deliberation an application may deliberately hide, prefixed with template markup, with no answer following it. Any caller that renders or parses content gets it wholesale.

Run it yourself

curl "https://api.sambanova.ai/v1/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @think-truncation.sambanova.json

As shipped (max_tokens: 60) the response leads with <|channel>thought in content. Raise max_tokens to 300 — the cut lands mid-answer and the split is clean; raise it to 1200 — the response completes with reasoning and answer in their proper fields.