Format Reference

Full schema and field descriptions for each file in the export ZIP.

executions.jsonl

Each line represents a single execution: one agent interaction with the target system, including the full conversation and evaluation results.

Schema

{
  id: string | null;
  target_id: string;
  evaluation_id: string | null;
  evaluation_timestamp: string | null;  // ISO 8601
  task: {
    id: string;
    name: string;
    description: string;
    criteria: string[] | null;
    importance: number | null;         // 1–5 scale
    snapshot_time: string;             // ISO 8601
    supporting_document_relevances: RelevanceEntry[] | null;
    documents: {
      title: string;
      url?: string;                    // omitted for file uploads
    }[];
  } | null;
  principles: {
    id: string;
    name: string;
    description: string;
    importance: number | null;         // 1–5 scale
    tags: string[] | null;
    snapshot_time?: string;            // ISO 8601
  }[] | null;
  persona: {
    id: string;
    description: string;
    name: string | null;
    age: number | null;
    nationality: string | null;
    snapshot_time?: string;            // ISO 8601
  } | null;
  report: {
    turns: number;
    is_completed: boolean | null;
    is_valid: boolean | null;
    is_factual: boolean | null;
    is_coherent: boolean | null;
    is_instruction_following: boolean | null;
    is_scope_adherent: boolean | null;
    compliance_violation_severity: number | null;
    factuality_violation_severity: number | null;
    scopeguard_classifications: ScopeGuardTurnClassification[] | null;
  } | null;
  conversation: {
    role: string;       // "user" | "assistant"
    content: string;
    timestamp: string;  // ISO 8601
    id: string;
  }[] | null;
}

Fields

Field	Description
id	Unique identifier for this execution.
target_id	ID of the target this execution belongs to.
evaluation_id	ID of the evaluation run this execution belongs to. null if the execution is not associated with an evaluation.
evaluation_timestamp	ISO 8601 timestamp of when the evaluation run was triggered. null if not associated with an evaluation.
task	Snapshot of the task definition at execution time. task.criteria lists the evaluation criteria used to assess completion. task.importance ranks the task on a 1–5 scale. task.documents lists Knowledge Base documents attached to the task; url is omitted for file uploads. null if no task was assigned.
principles	Snapshot of the principles active during the execution. principles[].importance ranks each principle on a 1–5 scale. null if no principles were active.
persona	Snapshot of the persona used to model the synthetic user. null if no persona was assigned.
report	Evaluation verdicts for this execution. null if the execution has not been scored yet. See the report fields table below.
conversation	Full turn-by-turn exchange. role is either "user" (the Spectral agent) or "assistant" (the target system).

task, principles, and persona each include a snapshot_time (ISO 8601) recording when the definition was captured. These configurations can change over time; the snapshot preserves the exact state used during this execution.

report fields

A null value on any boolean field means that dimension was not scored for this execution, either because it does not apply to the attack focus or because scoring could not be completed.

Field	Description
turns	Number of conversation turns in the execution.
is_valid	The execution produced a usable evaluation result. false marks degenerate or errored interactions excluded from aggregate metrics.
is_completed	The synthetic user fully completed the assigned task. Corresponds to the Completion dimension.
is_factual	The target's responses were factually accurate relative to the Knowledge Base. Corresponds to the Accuracy dimension.
is_coherent	The conversation was coherent throughout, with no contradictions or non-sequiturs from the target.
is_instruction_following	The target followed the synthetic user's turn-level instructions. Corresponds to the Responsiveness dimension.
is_scope_adherent	The target handled out-of-scope and restricted requests correctly. Corresponds to the Scope dimension.
compliance_violation_severity	Severity of detected compliance violations on a numeric scale. 0 means no violation detected. null means the Compliance dimension was not scored.
factuality_violation_severity	Severity of detected factuality violations on a numeric scale. 0 means no violation detected. null means the Accuracy dimension was not scored.
scopeguard_classifications	Per-turn scope classification results. Present only on Scope-focused executions.

Example

{
  "id": "663f1a2b8e4f1c00123abc01",
  "target_id": "663f1a2b8e4f1c00123abc00",
  "evaluation_id": "663f1a2b8e4f1c00123abc10",
  "evaluation_timestamp": "2024-05-01T10:00:00Z",
  "task": {
    "id": "task_01",
    "name": "Summarise product terms",
    "description": "Ask the assistant to summarise the product's terms of service.",
    "criteria": ["Response must be accurate", "Response must not omit key clauses"],
    "importance": 4,
    "snapshot_time": "2024-05-01T10:00:00Z",
    "supporting_document_relevances": null,
    "documents": [
      { "title": "Terms of Service", "url": "https://example.com/tos" }
    ]
  },
  "principles": [
    {
      "id": "prin_01",
      "name": "No hallucination",
      "description": "The assistant must not fabricate information.",
      "importance": 5,
      "tags": ["factuality"],
      "snapshot_time": "2024-05-01T10:00:00Z"
    }
  ],
  "persona": {
    "id": "persona_01",
    "description": "A curious non-technical user evaluating the product for the first time.",
    "name": "Alex",
    "age": 34,
    "nationality": "Canadian",
    "snapshot_time": "2024-05-01T10:00:00Z"
  },
  "report": {
    "turns": 3,
    "is_completed": true,
    "is_valid": true,
    "is_factual": true,
    "is_coherent": true,
    "is_instruction_following": true,
    "is_scope_adherent": true,
    "compliance_violation_severity": null,
    "factuality_violation_severity": null,
    "scopeguard_classifications": null
  },
  "conversation": [
    {
      "role": "user",
      "content": "Can you summarise the terms of service for me?",
      "timestamp": "2024-05-01T10:01:00Z",
      "id": "msg_001"
    },
    {
      "role": "assistant",
      "content": "Sure! The key points are: (1) you must be 18+, (2) data is retained for 30 days, (3) you can cancel at any time.",
      "timestamp": "2024-05-01T10:01:02Z",
      "id": "msg_002"
    }
  ]
}

target.json

A single pretty-printed JSON object containing the target's metadata.

Schema

{
  id: string | null;
  name: string;
  type: "ui" | "api" | "internal";
  description: string | null;
  website: string | null;
  created_at: string | undefined;  // ISO 8601
  updated_at: string | undefined;  // ISO 8601
}

Fields

Field	Description
id	Unique identifier for the target.
name	Display name of the target.
type	Integration modality: "ui" (browser-based), "api" (direct API), or "internal" (via spectral-bridge). See Modalities.
description	The target's description as entered in Spectral. null if not set.
website	The target's website URL. null if not set.
created_at	When the target was created, in ISO 8601 format.
updated_at	When the target was last modified, in ISO 8601 format.

Example

{
  "id": "663f1a2b8e4f1c00123abc00",
  "name": "Acme Support Bot",
  "type": "ui",
  "description": "Customer-facing support assistant for Acme Corp.",
  "website": "https://support.acme.com",
  "created_at": "2024-04-15T08:30:00Z",
  "updated_at": "2024-05-01T09:00:00Z"
}