For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Simulations test agent behavior through generated, multi-turn conversations. Effective simulations use focused scenarios, clear completion criteria, consistent model configurations, and enough conversation coverage to reveal how an agent behaves across different paths.

Use these best practices when designing and comparing simulations.

For instructions on configuring a simulation, see How to Create a Simulation.

Define a Focused Simulation Goal

Start with a clear behavior, task, or user goal you want the simulation to test.

A focused goal makes generated scenarios easier to interpret and helps ensure the resulting conversations exercise the behavior you intend to evaluate.

For example, instead of testing whether an agent can “handle customer support,” focus on a specific objective such as:

• Help a user compare two product options.
• Troubleshoot a failed deployment.
• Follow an escalation policy.
• Collect the information required to complete a request.
• Use the correct tool to perform an action.

Use separate simulations for unrelated behaviors when you want to measure them independently.

Create Representative Scenarios

Use scenarios that reflect the situations and user goals your agent is expected to handle.

A scenario should state specifically what the simulated user is trying to accomplish. When defining scenarios manually, you can also provide a user persona, an expected outcome, stopping criteria, and a maximum number of turns.

Include scenarios that cover:

• Common user goals.
• Different ways users may approach the same goal.
• Incomplete or ambiguous requests.
• Situations that require follow-up questions.
• Requests that require tools or external actions.
• Situations where the agent should follow a specific guardrail or policy.

For information on generating and uploading scenario sets, see How to Generate and Manage a Scenario Set.

Write Clear Expected Outcomes and Stopping Criteria

A scenario’s expected outcome and stopping criteria serve different purposes:

• Expected outcome: Defines what a successful conversation achieves. The judge scores the User goal completion metric against the expected outcome.
• Stopping criteria: Define the conditions that tell the user simulator to end the conversation. If a scenario has no expected outcome, the judge scores User goal completion against the stopping criteria instead.

Write both around observable outcomes rather than vague qualities.

Clear expected outcomes make User goal completion scores easier to interpret. For guidance on writing them, see Scenario Set Best Practices.

Use Personas When User Behavior Matters

Use a user_persona when the way the simulated user behaves is important to the scenario.

For example, a persona can describe a user who:

• Has limited technical experience.
• Is cost-conscious.
• Provides incomplete information.
• Changes requirements during the conversation.
• Needs additional explanation before making a decision.

Avoid adding persona details that do not affect the interaction. Keep the persona focused on characteristics that should influence how the conversation develops.

Choose the User Simulator Model Based on Conversation Complexity

The user simulator model plays the end user in each simulated conversation.

For less complex scenarios, a lower-cost model is often enough. For scenarios that require nuanced, varied, or complex conversations, use a more capable model.

When comparing simulation runs, use the same user simulator model whenever possible. Changing the simulator model can change how the simulated user behaves, which can make differences between runs harder to attribute to changes in the agent.

Keep the Judge Model Consistent Across Comparisons

The judge model scores each simulated episode using the evaluation metrics you select. For User goal completion, it scores against the scenario’s expected outcome, or its stopping criteria if the scenario has no expected outcome.

Choose the judge model based on the complexity of the behavior you are evaluating and the cost you want to incur.

For less complex evaluation criteria, a lower-cost model is often enough. For evaluations that require nuanced interpretation of long conversations, complex instructions, or subjective criteria, use a more capable model.

When comparing simulation runs, use the same judge model whenever possible so differences in scores are less likely to result from changes in judging behavior.

Select Metrics That Match the Scenario

Choose metrics that reflect the behavior you want the simulation to measure.

For example:

Avoid selecting metrics only because they are available. Use metrics that provide useful information for the behavior represented by the scenario.

Set Enough Turns for the Scenario to Complete

The turn limit per scenario sets how many user-assistant exchanges can occur before the scenario ends.

Set the limit high enough for the agent to reasonably complete the scenario without allowing conversations to continue unnecessarily.

Short, single-step tasks may require fewer turns. Scenarios involving information gathering, troubleshooting, tool calls, or changing user requirements may require more.

If trajectories frequently end before the agent can complete the task, increase the maximum number of turns and run the simulation again.

Use Multiple Episodes to Explore Different Conversation Paths

The number of episodes per scenario determines how many times each scenario is replayed.

Use a single episode when you want quick feedback while developing or validating a simulation configuration.

Increase the number of episodes when you want to observe whether the same scenario produces different conversation paths or agent behavior across repeated runs.

Increasing episodes also increases the amount of model usage required for the simulation, so balance additional coverage against evaluation cost.

Change One Major Variable at a Time

When comparing simulation configurations, keep the scenario set, user simulator model, judge model, metrics, turn limit, and episodes consistent whenever possible.

Then change one major variable, such as:

• The managed agent configuration.
• The model backing an agent from a model.
• Agent instructions.
• Agent tools.
• An Inference Router configuration.

Keeping the rest of the simulation configuration consistent makes differences between runs easier to interpret.

Review Individual Trajectories

Aggregate scores can show whether performance changed, but individual trajectories show how the conversation reached its outcome.

Review trajectories to identify:

• Where the agent misunderstood the user’s goal.
• Whether the agent retained information from earlier turns.
• Whether unnecessary turns occurred.
• Where tool selection or tool arguments failed.
• Whether the agent followed scenario guardrails.
• Why an episode failed or did not reach its stopping criteria.

Use the trajectory alongside metric scores and judge results when deciding what to change in the agent or simulation configuration.

Start Small Before Expanding Coverage

When developing a new simulation, start with a small scenario set and a limited number of episodes.

Review the generated conversations and confirm that:

• The scenarios test the intended behavior.
• The simulated users behave as expected.
• The expected outcomes represent successful completion.
• The stopping criteria end conversations at the right point.
• The selected metrics measure the behavior you care about.
• The maximum number of turns gives the agent enough time to complete the task.

After validating the setup, add scenarios or increase the number of episodes to broaden coverage.