# Images in tool results: rejected, 500'd, or silently blinded on most providers

high · inference · probed live 2026-07-26 · affects Cerebras, Novita, SambaNova, DeepInfra,
AWS Bedrock (Chat Completions), AWS Bedrock (Responses), Google (OpenAI-compat) ·
HTML version: [https://inferencecanary.com/tool-result-images](https://inferencecanary.com/tool-result-images)

Tools return images — screenshots, charts, photos — and the model reads them. Google's native
API proves it: given a solid magenta test image inside a tool result, the model names the
color. Together AI, Lightning and Parasail prove the standard OpenAI-compatible shape (an
`image_url` content part in a `role:"tool"` message) works too. Most providers fail anyway —
on schema choice, not model limits.

## What was sent

One conversation, everywhere: the model is asked to call `get_image` and name the dominant
color; the tool returns a 64×64 solid magenta PNG (133 bytes, embedded as a data URI). The
OpenAI-compatible body — set `model` to your provider's gemma-4-31b id
([https://inferencecanary.com/assets/repro/tool-image.chat.json](https://inferencecanary.com/assets/repro/tool-image.chat.json)):

```json
{
  "model": "gemma-4-31b",
  "messages": [
    {
      "role": "user",
      "content": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "get_image",
            "arguments": "{}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_image",
        "description": "Returns an image.",
        "parameters": {
          "type": "object",
          "properties": {}
        }
      }
    }
  ],
  "temperature": 0,
  "max_tokens": 300,
  "stream": true,
  "stream_options": {
    "include_usage": true
  }
}
```

## What came back

The expected answer is the reference lane's, verbatim: *"The dominant color of the image is
magenta."* Instead:

- **SambaNova** — HTTP **500**, deterministic, 7 of 7 attempts:
  `{"error":"unexpected_error","error_code":null,"error_param":null,"error_type":"unexpected_error"}`
- **Google (OpenAI-compat)** — HTTP **400**: `"Invalid content part type: image_url"` —
  Google's own shim rejecting a capability Google's native protocol serves the same model.
- **Cerebras** — HTTP **400**: `messages.2.tool.content…image_url: property … is unsupported` —
  the tool-message schema types content as text-only.
- **DeepInfra** — HTTP **422**: `"Input should be a valid string"` — tool content must be a
  plain string.
- **Novita** — HTTP **422**: `"invalid request error trace_id: cab746ba…"` — no reason given
  at all.
- **AWS Bedrock (Chat Completions)** — HTTP **400**: `"Invalid 'content': value did not match
  any expected variant"`
- **AWS Bedrock (Responses)** — HTTP **200, and the model is blind.** See below.

## The worst class: 200-but-blind

The Bedrock Responses lane has its own shape — the image rides as `input_image` inside
`function_call_output.output`
([https://inferencecanary.com/assets/repro/tool-image.responses.json](https://inferencecanary.com/assets/repro/tool-image.responses.json)):

```json
{
  "model": "google.gemma-4-31b",
  "input": [
    {
      "role": "user",
      "content": "Call the `get_image` tool. In one short sentence: what is the dominant color of the image?"
    },
    {
      "type": "function_call",
      "call_id": "call_1",
      "name": "get_image",
      "arguments": "{}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_1",
      "output": [
        {
          "type": "input_image",
          "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAATElEQVR42u3PMQkAAAwDsPo33UnoPQjEQNL0tQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgILAdBp+HSRtACMAAAAABJRU5ErkJggg=="
        }
      ]
    }
  ],
  "store": false,
  "tools": [
    {
      "type": "function",
      "name": "get_image",
      "description": "Returns an image.",
      "parameters": {
        "type": "object",
        "properties": {}
      }
    }
  ],
  "max_output_tokens": 300
}
```

It is accepted without error. The answer, verbatim:

```
The dominant color of the image is blue.
```

The image is magenta. A control with the same image in a *user* message passes on this lane,
so the model can see images here; the tool-result image is silently dropped before the model,
and the model answers confidently about content it never saw.

## The reference behavior

The identical conversation and identical image bytes on Google's native protocol
([https://inferencecanary.com/assets/repro/tool-image.google-native.json](https://inferencecanary.com/assets/repro/tool-image.google-native.json))
return HTTP 200 and *"The dominant color of the image is magenta."*

## How this breaks expectations

Delivery, not status codes, is the contract: the reference lane and three OpenAI-compatible
providers name magenta from identical bytes. Several providers reject the request, one 500s, and
AWS Bedrock (Responses) is the worst class — success status, wrong answer, no error for the
application to catch.

## Why it matters

Agentic multimodal — any tool that returns a screenshot, chart, or photo — is impossible on
these providers, and on the Bedrock Responses lane it fails by inventing.

## Run it yourself

```
curl "$BASE_URL/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @tool-image.chat.json
```

Magenta in the answer = delivered; an error or any other color = this finding. The Bedrock
Responses variant POSTs to `$BASE_URL/responses`; the Google-native variant POSTs to
`…/models/gemma-4-31b-it:generateContent` with `x-goog-api-key`.

## Provider response

[Reported to SambaNova (community thread)](https://community.sambanova.ai/t/gemma-4-image-in-tool-call-response/1677) — No substantive response. Reproducible as of 2026-07-26.