Images in tool results: rejected, 500'd, or silently blinded on most providers

high · inference · probed live 2026-07-26 · affects Cerebras, Novita, SambaNova, DeepInfra, AWS Bedrock (Chat Completions), AWS Bedrock (Responses), Google (OpenAI-compat)

Tools return images — screenshots, charts, photos — and the model reads them. Google's native API proves it: given a solid magenta test image inside a tool result, the model names the color. Together AI, Lightning and Parasail prove the standard OpenAI-compatible shape (an image_url content part in a role:"tool" message) works too. Most providers fail anyway — on schema choice, not model limits.

What was sent

One conversation, everywhere: the model is asked to call get_image and name the dominant color; the tool returns a 64×64 solid magenta PNG (133 bytes, embedded as a data URI). The OpenAI-compatible body — set model to your provider's gemma-4-31b id (download):

tool-image.chat.json

{
  "model": "gemma-4-31b",
  "messages": [
    {
      "role": "user",
      "content": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "get_image",
            "arguments": "{}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_image",
        "description": "Returns an image.",
        "parameters": {
          "type": "object",
          "properties": {}
        }
      }
    }
  ],
  "temperature": 0,
  "max_tokens": 300,
  "stream": true,
  "stream_options": {
    "include_usage": true
  }
}

What came back

The expected answer is the reference lane's, verbatim: "The dominant color of the image is magenta." Instead:

SambaNova — HTTP 500, deterministic, 7 of 7 attempts: {"error":"unexpected_error","error_code":null,"error_param":null,"error_type":"unexpected_error"}
Google (OpenAI-compat) — HTTP 400: "Invalid content part type: image_url" — Google's own shim rejecting a capability Google's native protocol serves the same model.
Cerebras — HTTP 400: messages.2.tool.content…image_url: property … is unsupported — the tool-message schema types content as text-only.
DeepInfra — HTTP 422: "Input should be a valid string" — tool content must be a plain string.
Novita — HTTP 422: "invalid request error trace_id: cab746ba…" — no reason given at all.
AWS Bedrock (Chat Completions) — HTTP 400: "Invalid 'content': value did not match any expected variant"
AWS Bedrock (Responses) — HTTP 200, and the model is blind. See below.

The worst class: 200-but-blind

The Bedrock Responses lane has its own shape — the image rides as input_image inside function_call_output.output (download):

tool-image.responses.json

{
  "model": "google.gemma-4-31b",
  "input": [
    {
      "role": "user",
      "content": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
    },
    {
      "type": "function_call",
      "call_id": "call_1",
      "name": "get_image",
      "arguments": "{}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_1",
      "output": [
        {
          "type": "input_image",
          "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
        }
      ]
    }
  ],
  "store": false,
  "tools": [
    {
      "type": "function",
      "name": "get_image",
      "description": "Returns an image.",
      "parameters": {
        "type": "object",
        "properties": {}
      }
    }
  ],
  "max_output_tokens": 300
}

It is accepted without error. The answer, verbatim:

response · HTTP 200

The dominant color of the image is blue.

The image is magenta. A control with the same image in a user message passes on this lane, so the model can see images here; the tool-result image is silently dropped before the model, and the model answers confidently about content it never saw.

The reference behavior

The identical conversation and identical image bytes on Google's native protocol (download) return HTTP 200 and "The dominant color of the image is magenta."

tool-image.google-native.json

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "get_image",
            "args": {}
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_image",
            "response": {
              "result": ""
            },
            "parts": [
              {
                "inlineData": {
                  "mimeType": "image/png",
                  "data": "iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
                }
              }
            ]
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "get_image",
          "description": "Returns an image.",
          "parameters": {
            "type": "OBJECT",
            "properties": {}
          }
        }
      ]
    }
  ],
  "generationConfig": {
    "temperature": 0,
    "maxOutputTokens": 300
  }
}

How this breaks expectations

Delivery, not status codes, is the contract: the reference lane and several OpenAI-compatible providers name magenta from identical bytes. Several providers reject the request, one 500s, and AWS Bedrock (Responses) is the worst class — success status, wrong answer, no error for the application to catch.

Why it matters

Agentic multimodal — any tool that returns a screenshot, chart, or photo — is impossible on these providers, and on the Bedrock Responses lane it fails by inventing.

Run it yourself

curl

curl "$BASE_URL/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @tool-image.chat.json

Magenta in the answer = delivered; an error or any other color = this finding. The Bedrock Responses variant POSTs to $BASE_URL/responses; the Google-native variant POSTs to …/models/gemma-4-31b-it:generateContent with x-goog-api-key.

Provider response

Reported to SambaNova (community thread) — No substantive response. Reproducible as of 2026-07-26.

Have a response or a correction? [email protected] — it becomes part of this record.