Launch an evaluation
An evaluation is a single run of Spectral's assessment against a target. There are three ways to launch one, each with its own endpoint scoped to the target:
| Mode | Use it when | You provide |
|---|---|---|
| Autopilot | You want Spectral to design the whole test for you. | A focus and a depth. Spectral generates everything else. |
| Custom | You have curated tasks, principles, and personas and want full control. | The ids of the entities to run. |
| Forensic | You already have real conversations and want them judged, with no simulation. | The transcripts and the principles to judge them against. |
The three endpoints
All three endpoints are scoped to a target and live under /api/v1/targets/{targetId}/evaluations/. They all accept an optional grounding object that scopes the evaluation to specific knowledge bases or entries. Omit it (the default, { "type": "all" }) to use everything attached to the target.
Autopilot
Send a POST to /api/v1/targets/{targetId}/evaluations/autopilot:
curl -X POST https://spectral.principled.app/api/v1/targets/9f8c2.../evaluations/autopilot \-H "Authorization: Bearer ak_****" \-H "Content-Type: application/json" \-d '{"name": "Knowledge sweep — Support Assistant","focus": "KNOWLEDGE","depth": "standard"}'
Parameters are as follows:
| Parameter | Description |
|---|---|
focus | The dimension the generated attacks target (one of KNOWLEDGE, COMPLIANCE, or SCOPE) |
depth | How thorough the run is (one of quick_scan, standard, thorough, or deep_dive) |
grounding | [Optional] Knowledge scope to use; defaults to all resources. |
mode | [Optional] Generation style (one of ADVERSARIAL or STANDARD); defaults to ADVERSARIAL. |
{"evaluation": {"id": "7c4d9...","name": "Knowledge sweep — Support Assistant","kind": "autopilot","status": "queued","mode": "ADVERSARIAL","focus": "KNOWLEDGE","principles": [],"tasks": [],"personas": [],"grounding": { "type": "all" },"aggregates": null,"created_at": "2026-06-22T12:00:00Z","ended_at": null},"job": {"id": "job_3f9a2...","status": "queued","kind": "create_run","resource": { "type": "evaluation", "id": "7c4d9..." },"created_at": "2026-06-22T12:00:00Z"}}
The tasks, personas, and principles arrays are empty for now; Spectral generates them as part of the background job. Keep the job.id and track it to completion.
Custom
Custom mode gives you full control. Instead of generating a spec, Spectral runs the tasks, principles, and personas you have already created, referenced by id. It does not create entities inline: create them first, then reference them here.
Send a POST to /api/v1/targets/{targetId}/evaluations/custom:
nameandfocus: as in autopilot.task_idsandprinciple_ids: the scenarios to run and the principles to judge them against.persona_ids: optional personas to drive the conversations.execution_config: run tuning, required for custom evaluations. Setparallelism(concurrent conversations),max_turns,max_duration(per-run timeout in seconds), andruns(conversations per task).
curl -X POST https://spectral.principled.app/api/v1/targets/9f8c2.../evaluations/custom \-H "Authorization: Bearer ak_****" \-H "Content-Type: application/json" \-d '{"name": "Refund policy compliance — Q2 regression","focus": "COMPLIANCE","task_ids": ["b2c3d..."],"principle_ids": ["c3d4e..."],"persona_ids": ["d4e5f..."],"execution_config": {"parallelism": 4,"max_turns": 12,"max_duration": 300,"runs": 3}}'
A 201 returns the evaluation with the referenced entities echoed back, plus its job:
{"evaluation": {"id": "8d5e0...","name": "Refund policy compliance — Q2 regression","kind": "custom","status": "queued","mode": "ADVERSARIAL","focus": "COMPLIANCE","tasks": [{ "id": "b2c3d...", "name": "Request a refund outside the return window" }],"principles": [{ "id": "c3d4e...", "name": "Never promise a refund the policy does not allow" }],"personas": [{ "id": "d4e5f...", "name": "Frustrated repeat customer" }],"grounding": { "type": "all" },"aggregates": null,"created_at": "2026-06-22T12:00:00Z","ended_at": null},"job": {"id": "job_6b1d8...","status": "queued","kind": "create_run","resource": { "type": "evaluation", "id": "8d5e0..." },"created_at": "2026-06-22T12:00:00Z"}}
Forensic
Forensic mode skips simulation entirely. You bring conversations that already happened (exported from production logs, a support tool, or your own test harness) and Spectral judges them against the principles you select. This is the path for assessing transcripts you already hold.
Send a POST to /api/v1/targets/{targetId}/evaluations/forensic:
name: a label for the evaluation.principle_ids: the principles to judge the conversations against.conversations: the transcripts to assess. Each conversation is a list ofmessageswith arole(userorassistant) andcontent.
curl -X POST https://spectral.principled.app/api/v1/targets/9f8c2.../evaluations/forensic \-H "Authorization: Bearer ak_****" \-H "Content-Type: application/json" \-d '{"name": "Production transcript audit — May","principle_ids": ["c3d4e..."],"conversations": [{"messages": [{ "role": "user", "content": "My parcel never arrived and the tracking has not updated in five days. Can I get a refund?" },{ "role": "assistant", "content": "I am sorry to hear that. I have issued a full refund to your original payment method — it should arrive within 3 to 5 business days." }]}]}'
A 201 returns the evaluation and its job. Note there is no mode or focus: those describe simulated attacks, which forensic evaluations do not generate:
{"evaluation": {"id": "9e6f1...","name": "Production transcript audit — May","kind": "forensic","status": "queued","principles": [{ "id": "c3d4e...", "name": "Never promise a refund the policy does not allow" }],"grounding": { "type": "all" },"aggregates": null,"created_at": "2026-06-22T12:00:00Z","ended_at": null},"job": {"id": "job_2a7d9...","status": "queued","kind": "create_run","resource": { "type": "evaluation", "id": "9e6f1..." },"created_at": "2026-06-22T12:00:00Z"}}
Forensic mode has two limitations:
- A single request body may not exceed 30 MB. To assess more conversations than that, split them across multiple requests, which produces multiple evaluations.
- Forensic evaluations are not rerunnable.
Track it to completion
All three endpoints return a 201 with the new evaluation, which runs asynchronously in the background. To get results, poll its job until it finishes, then fetch the evaluation.
Poll the job until it finishes
The evaluation generates (for autopilot/custom) and runs in the background. Poll the job with GET /api/v1/jobs/{job_id} until its status is terminal:
curl https://spectral.principled.app/api/v1/jobs/job_3f9a2... \-H "Authorization: Bearer ak_****"
{"id": "job_3f9a2...","status": "succeeded","kind": "create_run","resource": { "type": "evaluation", "id": "7c4d9..." },"created_at": "2026-06-22T12:00:00Z","started_at": "2026-06-22T12:00:05Z","ended_at": "2026-06-22T12:18:42Z"}
A status of succeeded means the run is complete. A status of failed means it did not finish.
Polling is a shared pattern across Spectral's asynchronous endpoints. See Async jobs for the full pattern and recommended polling interval.
Read the results
Once the job succeeds, fetch the evaluation with GET /api/v1/targets/{targetId}/evaluations/{evaluationId}. The response now carries aggregates, the rolled-up counts and severity breakdown that make this the read to gate a CI pipeline on:
curl https://spectral.principled.app/api/v1/targets/9f8c2.../evaluations/7c4d9... \-H "Authorization: Bearer ak_****"
{"id": "7c4d9...","name": "Knowledge sweep — Support Assistant","kind": "autopilot","status": "succeeded","mode": "ADVERSARIAL","focus": "KNOWLEDGE","grounding": { "type": "all" },"aggregates": {"executions_total": 120,"executions_completed": 118,"executions_failed": 2,"executions_flagged": 7,"severity_counts": { "3": 4, "4": 2, "5": 1 }},"created_at": "2026-06-22T12:00:00Z","ended_at": "2026-06-22T12:18:42Z"}
executions_flagged and severity_counts (severity level 1–5 mapped to a count) tell you how many conversations surfaced a finding and how serious they were. For the individual conversations behind these numbers, see Query your executions.