Forensic transcript formats

A forensic evaluation takes as input a JSON file containing a conversations array. Each entry is a conversation transcript in one of two formats, with data nested under a single top-level container key that identifies it:

Format	Container key	Use it when
Chat Completions	messages	The conversation is plain back-and-forth text, with no tool use.
Responses API	items	The agent called tools (function calling, retrieval, MCP, code execution) and you want that behavior judged.

tip

A single conversations array can contain both Chat Completions and Responses API transcripts, in any order.

Chat Completions transcripts

Chat Completions transcripts follow the OpenAI Chat Completions API, the same format the API modality uses. A transcript is an object with a messages array, where each message has a role and a string content:

Chat Completions transcript

{
  "messages": [
    { "role": "user",      "content": "My order #48231 never arrived. Can I get a refund?" },
    { "role": "assistant", "content": "I'm sorry to hear that. I've issued a full refund to your original payment method. It should arrive within 3 to 5 business days." }
  ]
}

Field	Description
messages	Non-empty array of turns, in order.
messages[].role	Either user or assistant. No other roles are accepted in this format.
messages[].content	The turn's text. Must be a non-empty string.

A transcript must contain at least one assistant turn: that is the output Spectral's judges score.

Responses API transcripts

Use this format when your agent calls tools and you want Spectral to evaluate them as well. It carries the full sequence of tool calls, their outputs, and the agent's final reply as an ordered list of items, following the OpenAI Responses API. A transcript is an object with two fields:

{
  items: ResponseItem[];   // the ordered turn-by-turn stream, required, non-empty
  tools?: ToolDef[];       // the agent's tool catalog, optional
}

Items

items is the ordered event stream of the conversation. Each item is an object with a string type. Spectral recognizes the standard Responses item types; the ones that matter for a forensic transcript are:

type	Meaning	Key fields
message	A user or assistant turn.	role (user | assistant | system), content
function_call	A tool call the agent made.	call_id, name, arguments
function_call_output	The result returned to the agent for a call.	call_id, output
reasoning	The agent's private reasoning summary, when exposed.	summary

Built-in and MCP tool items (web_search_call, file_search_call, mcp_call, mcp_list_tools, …) are also accepted and normalized. Any item type Spectral does not recognize is preserved but otherwise ignored, so forwarding a raw transcript is safe.

At least one item must be an assistant message: the reply the judges score.

Messages

A message item carries a role and content. Assistant content is the Responses output structure, an array of parts; Spectral reads the reply text from the output_text parts. User and system messages may use a plain string or the same array structure.

{
  "type": "message",
  "role": "assistant",
  "content": [
    { "type": "output_text", "text": "I've issued a full refund of $79.90." }
  ]
}

Tool calls and outputs

Each tool call is a function_call item paired with a function_call_output item carrying its result. The two are matched by a shared call_id.

Field	On	Description
call_id	both	Correlates a call with its output. Must match between the pair.
name	function_call	The tool the agent invoked.
arguments	function_call	The call arguments as a JSON-encoded string (as the Responses API emits them).
output	function_call_output	The tool's result. A string (often JSON) or a structured value. An error key in a JSON result marks the call as failed.

{ "type": "function_call",        "call_id": "call_lookup_1", "name": "lookup_order",
  "arguments": "{\"order_id\": \"48231\"}" }
{ "type": "function_call_output", "call_id": "call_lookup_1",
  "output": "{\"status\": \"in_transit\", \"refund_eligible\": true}" }

Tool catalog

The optional top-level tools array declares the tools available to the agent, using the flat Responses API tool format. Declaring the full catalog does two things. It gives Spectral each tool's JSON Schema, so it can check that call arguments are well-formed. More importantly, it lets Spectral judge the agent's tool selection against every tool it could have called, not just the ones it did: whether the calls it made were the right ones, and whether it overlooked a tool it should have used. Without the complete set, that selection cannot be scored.

Field	Description
type	"function" for a function tool. Built-in tool entries (e.g. { "type": "web_search" }) are accepted and skipped for schema checking.
name	The function's name. Must match the name on the corresponding function_call items.
description	What the tool does. Optional but recommended.
parameters	The tool's JSON Schema. Used to validate call arguments.

{
  "type": "function",
  "name": "issue_refund",
  "description": "Issue a refund for an order to the customer's original payment method.",
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" },
      "amount":   { "type": "number" },
      "reason":   { "type": "string" }
    },
    "required": ["order_id", "amount"]
  }
}

Declaring tools is optional. Spectral also discovers tools from any mcp_list_tools items in items. A declared tool takes precedence over a discovered one with the same name.

Automatic tool evaluation

When a Responses API transcript contains tool calls, Spectral evaluates them automatically, with no extra setup: whether each call was well-chosen with justified arguments, whether its result was useful, and whether the agent's answer faithfully used it. A Chat Completions transcript, or a Responses API transcript with no tool calls, is judged on its output alone, with no tool findings.

Full example

A complete Responses API transcript (one entry of the conversations array) with the user's message, a reasoning step, two tool calls and their outputs, the assistant's reply, and the tool catalog:

Responses API transcript

{
  "items": [
    {
      "type": "message",
      "role": "user",
      "content": "My order #48231 never arrived and the tracking hasn't updated in five days. Can I get a refund?"
    },
    {
      "type": "reasoning",
      "summary": [
        {
          "type": "summary_text",
          "text": "The customer is reporting a missing order. I should look up its status and refund eligibility before deciding what to offer."
        }
      ]
    },
    {
      "type": "function_call",
      "call_id": "call_lookup_1",
      "name": "lookup_order",
      "arguments": "{\"order_id\": \"48231\"}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_lookup_1",
      "output": "{\"order_id\": \"48231\", \"status\": \"in_transit\", \"last_scan\": \"2026-05-27\", \"carrier\": \"UPS\", \"refund_eligible\": true, \"amount\": 79.90}"
    },
    {
      "type": "function_call",
      "call_id": "call_refund_1",
      "name": "issue_refund",
      "arguments": "{\"order_id\": \"48231\", \"amount\": 79.90, \"reason\": \"lost_in_transit\"}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_refund_1",
      "output": "{\"refund_id\": \"re_9f2a3c\", \"status\": \"succeeded\", \"amount\": 79.90}"
    },
    {
      "type": "message",
      "role": "assistant",
      "content": [
        { "type": "output_text", "text": "I'm sorry your order didn't arrive. I've confirmed it was lost in transit and issued a full refund of $79.90 to your original payment method. It should appear within 3 to 5 business days." }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "name": "lookup_order",
      "description": "Look up the delivery status and refund eligibility of a customer order.",
      "parameters": {
        "type": "object",
        "properties": { "order_id": { "type": "string" } },
        "required": ["order_id"]
      }
    },
    {
      "type": "function",
      "name": "issue_refund",
      "description": "Issue a refund for an order to the customer's original payment method.",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": { "type": "string" },
          "amount": { "type": "number" },
          "reason": { "type": "string" }
        },
        "required": ["order_id", "amount"]
      }
    }
  ]
}

Drop this object straight into the conversations array of a forensic request.

Validation and limits

Spectral checks every conversation before creating the evaluation; a violation rejects the whole request with 422 and nothing is queued.

• Format: each conversation must match the Chat Completions (messages) or Responses API (items) format; one matching neither is rejected.
• Judgeable output: every conversation must contain at least one assistant turn (Chat Completions) or assistant message item (Responses API).
• Chat Completions rules: non-empty messages; each role is user or assistant; each content is a non-empty string.
• Responses API rules: non-empty items; each item is an object with a non-empty string type; tools, if present, is an array of objects.
• Request size: a single request body may not exceed 30 MB. Split larger sets across multiple requests (each produces its own evaluation).
• Not re-runnable: forensic evaluations replay fixed transcripts, so they cannot be re-run.

Launch a forensic evaluationThe endpoint, request body, and how to track the run to completion.→