Simulation Testing
How often should I run simulations?
It depends on where you are in your testing maturity. Three patterns are common, each requiring more workflow investment than the last but yielding progressively stronger coverage.
| Pattern | Description | Where it fits |
|---|---|---|
| One-off assessment | A single evaluation to establish a baseline before launch or to satisfy a compliance or due-diligence requirement. | Simplest starting point |
| On every change | Run an evaluation whenever you update your system prompt, retrain your model, or modify your knowledge base. Catches regressions early and gives confidence that each change ships safely. | Natural next step |
| Recurring schedule | Run evaluations periodically regardless of whether your system has changed. The risk landscape evolves even when your code does not: new user behaviors emerge, regulatory expectations shift, and adversarial techniques improve. | Recommended for active production |
If you are just getting started, a one-off assessment is the right first step. From there, integrating Spectral into your change process is the logical progression.
What interfaces do you support?
If your chatbot has a public OpenAI-compatible /v1/chat/completions endpoint or a public web interface,
Spectral can test it directly with no special integration.
No special integration is required. If your system exposes an OpenAI-compatible /v1/chat/completions endpoint, Spectral can connect to it directly for the smoothest experience. If not, Spectral can interact through your system's standard web interface, navigating and typing just as a real user would. You do not need to expose any internal model or infrastructure.
What if my chatbot is private?
This is exactly why we are working on spectral-bridge at Principled Intelligence.
It connects Spectral to private or local systems through a secure relay,
giving you the freedom to test anything without exposing it publicly.