Simulation Testing
How often should I run simulations?
It depends on where you are in your testing maturity. Three patterns are common, each requiring more workflow investment than the last but yielding progressively stronger coverage.
| Pattern | Description | Where it fits |
|---|---|---|
| One-off assessment | A single evaluation to establish a baseline before launch or to satisfy a compliance or due-diligence requirement. | Simplest starting point |
| On every change | Run an evaluation whenever you update your system prompt, retrain your model, or modify your knowledge base. Catches regressions early and gives confidence that each change ships safely. | Natural next step |
| Recurring schedule | Run evaluations periodically regardless of whether your system has changed. The risk landscape evolves even when your code does not: new user behaviors emerge, regulatory expectations shift, and adversarial techniques improve. | Recommended for active production |
If you are just getting started, a one-off assessment is the right first step. From there, integrating Spectral into your change process is the logical progression.
What interfaces do you support?
Spectral connects through one of three modalities:
- API (recommended): Spectral sends conversations directly to a publicly accessible OpenAI-compatible endpoint. Use Chat Completions for plain conversational systems, or the Responses API for agents that call tools.
- UI: Spectral drives your public web interface with browser automation, navigating and typing just as a real user would. No API access required.
- Internal: Spectral reaches private or local systems through spectral-bridge, a secure relay.
Not sure which fits your setup? The supported modalities overview has a decision tree.
What if my chatbot is private?
Use the Internal modality. spectral-bridge runs on your own infrastructure and opens an outbound-only connection to Spectral, so you can test private or local systems without exposing anything publicly.
What if my chatbot is behind authentication?
Not a problem for UI targets behind a login. Two methods are available:
- Credential login: you provide a username and password, and Spectral submits them before each run.
- SSO login: you sign in yourself once through single sign-on (including identity-provider redirects and 2FA), and Spectral reuses that session for every evaluation.