SambaNova enforces tool_choice by post-hoc 400, not constrained decoding

high · inference · probed live 2026-07-26 · affects SambaNova

tool_choice is the Chat Completions field an application uses to constrain tool calling: "required" guarantees the reply contains at least one tool call, and naming a function forces a call to exactly that function. The guarantee is the server's to keep — normally by constrained decoding, where the model is only allowed to generate output satisfying the constraint. SambaNova accepts the field, lets the model generate freely, and when the output happens to violate the constraint, fails the request with a 400 error. Whether your request succeeds depends on what the model felt like emitting.

What was measured

Three arms at temperature 0, thinking disabled: tool_choice: "none", "required", and forced-specific. "none" passes — there is nothing to violate. The other two arms, request and response verbatim:

"required" with a prompt that invites prose (download):

tool-choice-required.sambanova.json

{
  "model": "gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": "Just say hello."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "log_event",
        "description": "Logs an event message.",
        "parameters": {
          "type": "object",
          "properties": {
            "message": {
              "type": "string"
            }
          }
        }
      }
    }
  ],
  "tool_choice": "required",
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "temperature": 0,
  "max_tokens": 200
}

Received — HTTP 400, after 2.45 s (generation ran to completion before the rejection):

response · HTTP 400

{
  "error": "Specified tool_choice: \"required\", but the model output contains zero valid function call.",
  "error_code": null,
  "error_model_output": "Hello!",
  "error_param": null,
  "error_type": "Invalid function calling output."
}

Forcing get_time by name on a weather prompt, with get_weather, get_news and get_time offered (download):

tool-choice-specific.sambanova.json

{
  "model": "gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": "What's the weather in Paris right now?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string"
            }
          }
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "get_news",
        "description": "Latest news on a topic.",
        "parameters": {
          "type": "object",
          "properties": {
            "topic": {
              "type": "string"
            }
          }
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "get_time",
        "description": "Current time in a timezone.",
        "parameters": {
          "type": "object",
          "properties": {
            "timezone": {
              "type": "string"
            }
          }
        }
      }
    }
  ],
  "tool_choice": {
    "type": "function",
    "function": {
      "name": "get_time"
    }
  },
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "temperature": 0,
  "max_tokens": 200
}

Received — HTTP 400:

response · HTTP 400

{
  "error": "Specified tool_choice with function name \"get_time\", but the model output contains function call to another tool with function name \"get_weather\".",
  "error_code": null,
  "error_model_output": "<|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>}<tool_call|>",
  "error_param": null,
  "error_type": "Invalid function calling output."
}

How this breaks the contract

A valid request with a valid constraint must return 200 with a conforming tool_calls array; a 4xx is for malformed requests. Here the error reports a model output failure on a well-formed request — and the error_model_output field is raw template text, not a tool_calls structure an application could repair from. The model did emit a valid get_weather call in the second case; the provider discarded it and returned an error instead.

Why it matters

Forced-tool workflows that pass in testing throw unrecoverable runtime errors in production, timed to whenever the model's free-run output happens to disagree with the constraint. There is nothing to catch and retry against except the same coin-flip.

Run it yourself

Save either body above and POST it with your key:

curl

curl "https://api.sambanova.ai/v1/chat/completions" -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d @tool-choice-required.sambanova.json

A conforming provider returns 200 with a log_event (or get_time) call. SambaNova returns the 400 shown whenever the model's unconstrained output violates the constraint.