This feature is in public preview.
How to Create a Simulationpublic
Last verified 13 Oct 2026
DigitalOcean Evaluations measure model, Inference Router, and agent performance using dataset evaluations and simulations.
Simulations test an agent through generated, multi-turn conversations. Use simulations to test agent behavior across realistic scenarios based on a goal you provide.
A simulation is built from the following:
- Scenario: Describes a situation for a simulated user to act out, including the user’s goal, persona, expected outcome, stopping criteria, and turn limit.
- Agent under test: The agent that the scenario runs against. This is a managed agent, or an agent that the simulation creates from a model or Inference Router.
- Episode: One scenario played against one agent under test. Each episode produces a trajectory, which is the full conversation, and the judge model scores it.
- Simulation run: A group of episodes. A run plays every scenario in the set against the agent you chose, as many times as you set in Episodes per scenario.
DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.
Before you create a simulation, you need a scenario set. You can generate scenarios from a goal description using Scenario Builder, upload scenarios as a JSONL file, or create scenarios from production traces using DigitalOcean Signals, which is in public preview. For details, see How to Generate and Manage a Scenario Set.
Create a Simulation
To create a simulation, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, and then click New Evaluation / Simulation to open the Create Evaluation page.
In the How do you want to evaluate? section, select Simulations.
If needed, click Change to select a different evaluation method before running the evaluation.
Select a Preset
If you have saved a simulation preset, the Presets section appears at the top of the page. To populate the simulation settings from a preset, click the Select a preset dropdown menu, and then select a preset. The dropdown lists only simulation presets. To configure the simulation manually instead, click Configure without a preset.
The Load preset? window lists the values that the preset applies. Loading a preset replaces your current configuration. To apply it, click Load preset. To keep your current configuration, click Cancel.
If you have no simulation presets, the Presets section does not appear. You can save the configuration as a preset at the end of the setup.
Choose an Agent
Select the agent that participates in the simulated conversations. This is the agent under test: the agent that every scenario in the run plays against.
In the Choose agent section, under Agent source, choose one of the following:
- Managed agent to evaluate an existing deployed agent, including its configured tools, instructions, and other agent settings.
- Agent from a model to evaluate a model or Inference Router without deploying a managed agent first. The simulation creates an agent configuration that uses the selected model. The agent configuration remains on your team after the run.
Use a managed agent to evaluate an agent you have already deployed. The simulation holds its generated conversations with the selected managed agent.
Under the Agent source sub-section, click Managed agent, and then click the Select a managed agent dropdown menu then select an existing agent.
If you need to create one, see Create a Managed Agent.
Use an agent from a model to have the simulation create an agent backed by a model or Inference Router. The simulation holds its generated conversations with this agent.
Under the Agent source sub-section, click Agent from a model, then in the Model sub-section, click the Select a model type dropdown menu, and then select one of the following types:
- Serverless Inference: Use when you want to evaluate a foundation model without managing inference infrastructure. Serverless Inference automatically scales and is suited for variable or unpredictable workloads.
- Third Party: Use when you want to evaluate a commercial model from a third-party provider, such as OpenAI or Anthropic, using a partner model provider key.
- Dedicated Inference: Use when you want to evaluate a model running on an existing dedicated inference deployment, such as a model configured for steady, high-throughput workloads or specific GPU resources.
- Model Router: Use when you want to evaluate an Inference Router configuration that routes requests among multiple models based on the routing rules you configured.
Then, click the Select a model dropdown menu, and then select one of the options available for that type. If your team has access to only one model type, the Select a model type dropdown menu does not appear.
When you start the simulation run, DigitalOcean creates the following resources on your team:
- A model access key named
<run-name>-key. The key can query all models available to your team. If you do not have permission to create keys, paste an existing model access key in the Model access key field instead. - An agent configuration that uses the selected model and the model access key. Its name starts with
sim-, followed by the model name and the first eight characters of the run’s UUID.
Both resources remain on your team after the run finishes, is cancelled, or is deleted. The agent configuration appears in the Managed agent list, so you can select it in later runs. If you delete the model access key, the agent configuration can no longer call the model.
Choose Scenarios
Scenarios define the situations that the user simulator and agent act out during the evaluation. You can select an existing scenario set or generate a new one from a goal description.
To select an existing scenario set, expand the Choose simulation scenarios section, and then select a scenario set from the Scenario set dropdown menu. You can build new scenarios from this form by clicking Scenario Builder. For details on how to use Scenario Builder or import scenarios, see How to Generate and Manage a Scenario Set. For guidance on designing scenarios and scenario sets for simulations, see Scenario Set Best Practices.
After you choose a scenario set, the page displays the number of scenarios in the set and an overview of each scenario.
Choose a User Simulator Model
The user simulator model plays the end user across every scenario in the simulation.
In the Configure models & metrics section, expand the section, and then click the User simulator model dropdown menu to select the model you want to use.
By default, DeepSeek V4 Flash 0731 is selected. The dropdown menu displays the input and output token pricing for each model.
Choose a model based on the complexity of the simulated conversations and the cost you want to incur. For less complex scenarios, a lower-cost model is often enough. For scenarios that require nuanced, varied, or complex conversations, use a more capable model. When comparing simulation runs, use the same user simulator model to keep the simulated user behavior consistent between runs.
Configure Your Judge
The judge model scores each simulated episode using the evaluation metrics you select.
In the Configure models & metrics section, in the Judge model sub-section, click the Select a judge model dropdown menu, and then select the model you want to use.
The dropdown menu displays the input and output token pricing for each model.
Choose a judge model based on the complexity of the behavior you are evaluating and the cost you want to incur. For less complex evaluation criteria, a lower-cost model is often enough. For evaluations that require nuanced interpretation of long conversations, complex instructions, or subjective criteria, use a more capable model. When comparing simulation runs, use the same judge model to keep scoring consistent between runs.
Choose Metrics
Choose the metrics you want the judge model to use to score each simulated conversation. By default, User goal completion is selected.
Expand the Configure models & metrics section, and then in the Evaluation metrics section, select the checkbox next to each conversational metric you want to use. To select all available metrics, click Select all. You can also add custom metrics with your own scoring criteria.
You can select from the following metrics:
| Metric | Description | Recommendation |
|---|---|---|
| User goal completion | Measures whether the agent achieved the user’s overall goal by the end of the conversation. The judge scores against the scenario’s expected outcome or, if there is none, its stopping criteria. If the scenario has neither, the judge infers the goal from the conversation. | Use this to evaluate whether the conversation ultimately accomplishes what the simulated user requested. |
| Knowledge Retention | Measures whether the agent remembers and correctly uses information from earlier turns. | Use this when later responses depend on information provided earlier in the conversation. |
| Conversation Completeness | Measures whether the agent addressed all of the user’s requests across the conversation. | Use this to identify conversations where requests or follow-up questions were left unresolved. |
| Turn Relevancy | Measures whether each agent response stays relevant to the conversation so far. | Use this to identify responses that drift from the current topic or conversation context. |
| Tool Use Accuracy | Measures whether the agent selected the correct tools and called them with the correct arguments across the conversation. | Use this when the agent uses tools or external actions to complete the simulated user’s goal. |
| Agent Policy Adherence | Measures whether the agent’s behavior satisfied the expected-outcome criteria, or guardrails, defined for the conversation. | Use this when the simulation has specific behavioral or outcome requirements the agent must follow. |
Add a Custom Metric
Custom metrics let you define your own criteria for scoring each simulated conversation. Use custom metrics when the built-in metrics do not cover the behavior, policy, or outcome you want to evaluate. For guidance on designing custom metrics, see Dataset Evaluation Best Practices.
To use an existing custom metric, under Custom metrics, click the Custom metrics dropdown menu and select the metric.
To create a metric, click Add metric to open the Create metric window.
In the Metric name field, enter a name for the metric.
Optionally, in the Description (optional) field, describe what the metric measures.
In the Scoring prompt field, enter instructions that tell the judge model how to score the conversation.
Then, click Create. The new metric appears in the Custom metrics section with its checkbox selected.
To change a custom metric, click … next to the metric, and then click Edit metric. To take the metric off this simulation without deleting it, click Remove metric.
To delete the metric from your team, click … next to the metric, and then click Delete to open the Delete metric window. If any saved presets use the metric, the window lists them, and deleting the metric also deletes those presets. To confirm, enter the name of the metric, and then click Delete metric. Deleting a custom metric permanently removes it from future evaluations and simulations. This action cannot be undone.
Name Your Simulation
The run name identifies the simulation in the evaluation list. If you leave the name blank, we generate one automatically.
To name the simulation, expand the Optional settings section, and then in the Run name field, enter a name for the simulation run.
Set the Turn Limit per Scenario
The turn limit per scenario determines the maximum number of user-assistant exchanges that can occur before a scenario ends. The default is 5.
To change the limit, expand the Optional settings section, and then in the Turn limit per scenario field, enter the maximum number of turns.
Set Episodes per Scenario
Episodes per scenario determines how many times the simulation replays each scenario. Because the user simulator can take a different path each time, more episodes produce more trajectories for the same scenario, but also increase the amount of simulation work performed.
The number of episodes is set to 1 by default.
To change the number of episodes, expand the Optional settings section, and then in the Episodes per scenario field, enter the number of times you want each scenario to run.
Save the Configuration as a Preset
To reuse this configuration in later simulations, in the Save as preset section, select Save this configuration as a preset.
In the Preset name field, enter a name for the preset.
In the Choose what to save section, either click Select all sections, or select the individual sections you want the preset to retain.
The preset is saved when you click Run simulation.
Run the Simulation
After configuring the simulation, click Run simulation. When the run starts, the simulation results page opens.
To cancel a simulation run that is pending, running, or evaluating, go to the Evals & Simulation Runs tab of the Evaluations & Simulations page, click … next to the simulation’s evaluation, and then click Cancel Evaluation. Cancelling the evaluation also cancels the simulation run.
The results page displays the simulation status, number of scenarios and episodes, duration, configuration, and a breakdown of the episodes for each scenario.
For additional actions you can take on evaluations, see Manage Evaluations.
Monitor Simulation and Episode Status
A simulation run moves through the following statuses:
| Status | Description |
|---|---|
| Pending | The run is queued and has not started. |
| Running | Episodes are running as sessions on the agent under test. |
| Evaluating | All episodes finished, and the selected evaluation metrics are scoring the trajectories. |
| Succeeded | All episodes finished successfully. |
| Partially successful | Some episodes succeeded and some failed. |
| Failed | The run failed, for example, because it timed out. Review the failure reason on the results page. |
| Cancelled | A member of your team cancelled the run. |
Succeeded, Partially successful, Failed, and Cancelled are terminal statuses. You can only delete a simulation run after it reaches a terminal status. To delete a run that is still in progress, cancel it first. Cancelling a run stops any episodes that are in progress and marks them Failed with the reason simulation run cancelled. Episodes that already finished or failed don’t change status.
Each episode in the Scenario breakdown section has its own status:
| Status | Description |
|---|---|
| Preparing | The agent session for the episode is being set up. |
| Running | The conversation is in progress. You can view the trajectory while the episode runs. |
| Finished | The conversation ended and the verdict is available. Open the trajectory to review the conversation. |
| Failed | The episode stopped before the conversation finished, for example, because the run was cancelled. Review the failure reason in the trajectory. |
Download Trajectory Data
A trajectory contains the simulated conversation for an individual episode.
In the Scenario breakdown section, expand the scenario you want to review, and then click View trajectory for an episode.
The trajectory page displays the scenario prompt and persona, token usage, number of turns, duration, and messages exchanged between the user simulator and agent.
To download the trajectory data, click Download trajectory.
If a simulation or individual episode fails, its status displays as Failed. To troubleshoot the failure, click View trajectory to review how far the conversation progressed and where the failure occurred.
Then, update the simulation configuration or agent as needed, and then run the simulation again.
Run a Simulation Using the API
You can also create, monitor, and manage simulation runs using the DigitalOcean API. To create a simulation run, you need the scenarios to run and the agent or model to test. To create a scenario set using the API, see Manage Scenario Sets Using the API.
Create a Simulation Run Using the API
To specify the scenarios, set scenario_set_uuid to the UUID of a scenario set in the ready state. Alternatively, set library_scenario_uuid to the UUID of a Common Scenario & Goal Library entry. When you use a library entry, the run copies it into a new scenario set on your team, and the new set appears in your list of scenario sets.
To specify the agent under test, use one of the following options. You cannot use both in the same request.
- Managed agent: Set
agent_config.agent_uuidto the UUID of your managed agent. - Agent from a model: Set
candidate_model_uuidto the UUID of a model, Dedicated Inference deployment, or Inference Router, and setcandidate_model_sourcetoCANDIDATE_MODEL_SOURCE_SERVERLESS(the default),CANDIDATE_MODEL_SOURCE_DEDICATED, orCANDIDATE_MODEL_SOURCE_ROUTER. You must also provide a model access key inmodel_access_key. The simulation run does not return the key, but the agent configuration that DigitalOcean creates for the run stores it. Optionally, set a display name incandidate_model_name.
For example, the following request body runs a scenario set against a serverless model:
{
"name": "support-model-simulation",
"scenario_set_uuid": "<your-scenario-set-uuid>",
"candidate_model_uuid": "<your-model-uuid>",
"candidate_model_source": "CANDIDATE_MODEL_SOURCE_SERVERLESS",
"model_access_key": "<your-model-access-key>"
}When you test a model, DigitalOcean creates an agent configuration that wraps the model and your model access key. The agent configuration remains on your team after the run finishes, is cancelled, or is deleted, and you can select it as an agent in later runs. If you delete the model access key, the agent configuration can no longer call the model.
The user simulator model, judge model, turn limit, and episodes per scenario are optional. If you omit the models, the run uses the platform defaults. The max_turns and exploration_budget fields set the turn limit and episodes per scenario for the run and override the values set on individual scenarios. exploration_budget is the API name for Episodes per scenario in the Control Panel. The exploration_budget value can be up to 5.
To score the trajectories with evaluation metrics after the simulation finishes, include an evaluation_config object with the UUIDs of the metrics you want to use in metric_uuids. The evaluation reuses the run’s judge model.
To find metric UUIDs, use the List Model Evaluation Metrics endpoint. Choose built-in metrics in the conversational category (METRIC_CATEGORY_CONVERSATIONAL), because simulation runs reject built-in metrics that do not support multi-turn conversations. You can also use your team’s custom metrics. The evaluation_config object has the following requirements:
metric_uuidsmust contain from 1 to 10 metrics, with no duplicates.- If you set
star_metric, it must be one of the selected metrics, and its threshold must be from 0 to 100.
The request returns the new run in the pending state.
Monitor a Simulation Run Using the API
To check the run’s status and progress, retrieve the simulation run. Poll this endpoint until the run reaches a terminal status.
To list all simulation runs on your team:
Retrieve Episodes and Trajectories Using the API
To list the episodes in a run and their statuses:
To retrieve a single episode, including its verdict:
To retrieve an episode’s trajectory, send a request to the trajectory endpoint. While the episode is running, poll this endpoint to watch the conversation progress turn by turn.
To download the trajectory as a file, retrieve a download URL:
Rename, Cancel, or Delete a Simulation Run Using the API
You can change a simulation run’s name after you create it. You cannot change the run’s configuration or results.
To stop a run that is in progress, cancel it:
To delete a simulation run, the run must be in a terminal status. To delete a run that is still in progress, cancel it first.