Lightning leaks template reasoning markers into content

medium · inference · probed live 2026-07-26 · affects Lightning

A chat template's control markers — the delimiters that separate the model's reasoning channel from its answer — belong to the provider's parser, never to the content field an application renders. On Lightning, on an image-bearing tool turn, the parse intermittently fails in both directions at once: the reasoning field comes back empty and marker text sits at the front of the answer.

What was sent

The tool-result-image conversation — a magenta test image returned by get_image, max_tokens: 300 (download; the same conversation as the tool-result images finding, which Lightning otherwise passes):

{
  "model": "lightning-ai/gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "get_image",
            "arguments": "{}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_image",
        "description": "Returns an image.",
        "parameters": {
          "type": "object",
          "properties": {}
        }
      }
    }
  ],
  "temperature": 0,
  "max_tokens": 300,
  "stream": true,
  "stream_options": {
    "include_usage": true
  }
}

What came back

HTTP 200; content, exact bytes:

thought
<channel|>The dominant color of the image is magenta.

The reasoning field is the empty string; the answer itself is correct. The response used 13 of its 300 tokens — nowhere near truncation. Two of four identical runs leak, byte-identically; the other two return the clean "The dominant color of the image is magenta." Same bytes sent, same budget.

The contrast that isolates it

Same provider, same session: a text-returning tool call at a 400-token budget parses cleanly — reasoning text lands in the reasoning field, content is clean. The user-message-image control at the same 300-token budget is clean every time. Only the image-bearing tool turn leaks, and only sometimes.

How this breaks expectations

content must carry answer text only. Lightning's own clean responses prove the marker-stripping normally works; this is a nondeterministic channel-parse failure on one turn shape.

Why it matters

Any caller that renders or string-matches content intermittently gets a stray thought line and a literal <channel|> in its output — the worst kind of bug to chase, because the identical request usually comes back clean.

Run it yourself

curl "https://lightning.ai/api/v1/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @tool-image.lightning.json

POST the body a few times; some responses will lead with the marker text shown.