For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Scenario sets define the situations that a user simulator and agent act out during a simulation. Well-designed scenarios make simulation results easier to interpret by defining the user’s goal, expected outcome, and relevant conversation context.

A scenario describes the user’s side of a conversation: the situation, the persona, what success looks like, and when the conversation ends. The other side is the agent under test, which is either a managed agent or an agent that the simulation creates from a model or Inference Router.

During a simulation, each scenario plays against the agent under test one or more times. Each play is an episode, and the judge model scores the trajectory that each episode produces. Because a scenario describes only the user, you can run the same scenario set against any agent or model, which makes a well-built set a reusable baseline.

Use these best practices when generating scenarios using Scenario Builder, or for writing your own scenario sets for simulations. Scenario Builder can tailor scenarios to a managed agent by summarizing what the agent can do and drafting a goal from that summary.

In the Control Panel, your team’s scenario sets and the Common Scenario & Goal Library are on the Evaluations & Simulations page, under the Simulation Scenarios tab, in My Scenarios and Scenario Library. For information on generating, uploading, importing, and managing scenario sets, see How to Generate and Manage a Scenario Set.

Define a Clear Scenario Goal

Each scenario should test a specific user goal or agent behavior.

Keep the goal focused enough that you can determine whether the agent successfully completed the task.

A specific goal gives the user simulator a clear objective and makes the resulting conversation easier to evaluate.

Keep Each Scenario Focused

Avoid combining unrelated user goals into one scenario.

For example, instead of creating one scenario where a user needs to troubleshoot a deployment, compare pricing, and update billing information, create separate scenarios for each task.

Focused scenarios make it easier to determine which behaviors the agent handles successfully and which require improvement.

Use Representative User Situations

Create scenarios that reflect the situations your agent is likely to encounter.

Include scenarios based on:

• Common user goals.
• Frequently encountered support or application workflows.
• Incomplete or ambiguous requests.
• Requests that require multiple conversational turns.
• Tasks that require the agent to gather additional information.
• Tasks that require tools or external actions.
• Situations where the agent must follow a policy or guardrail.

Avoid relying only on ideal scenarios where the user provides all required information immediately.

Include Variation Across Scenarios

A scenario set should test different ways an agent may encounter the behavior you want to evaluate.

For example, scenarios can vary:

• How much information the user provides initially.
• How directly the user states their goal.
• Whether the user changes or clarifies requirements.
• Whether the agent needs to ask follow-up questions.
• Whether completing the task requires tool use.
• Whether the agent encounters a constraint or guardrail.

Variation helps reveal whether an agent can handle a behavior consistently instead of succeeding only under one specific conversation path.

Episodes per scenario add a second kind of variation. The same scenario replayed in several episodes lets the user simulator take a different path each time, so you see how the agent handles one situation under several conversations without writing more scenarios. Use variation across scenarios to cover different situations, and episodes per scenario to see how stable the agent is within one.

Write Specific Scenario Descriptions

When drafting a custom scenario, the description field explains the situation and the user’s goal.

Include enough context for the user simulator to understand what it is trying to accomplish without prescribing the exact conversation.

Avoid including unnecessary instructions about exactly what the user should say on each turn. The simulation should have room to develop naturally.

Describe the user, not the agent. Avoid naming the agent’s tools, internal steps, or model in the scenario. A scenario that says “the user wants a refund for a damaged order” runs against any agent under test. One that says “the agent calls the refund tool” only fits one build, and breaks as a baseline the first time the agent changes.

Use Personas When They Affect the Conversation

Use user_persona when a user’s characteristics or behavior should meaningfully affect how the scenario develops.

For example:

A first-time cloud user who understands basic web hosting concepts but is unfamiliar with virtual machines.

Useful personas can define characteristics such as:

• Technical experience.
• Familiarity with the product.
• Budget sensitivity.
• Communication style.
• Information the user already knows.
• Constraints relevant to the task.
• How the user behaves during the conversation, such as providing incomplete information or changing requirements.

Avoid adding persona details that do not affect the scenario.

Define the Expected Outcome

Use expected_outcome to define what a successful conversation achieves. The judge scores the User goal completion metric against these criteria. Other metrics do not use them.

If a scenario has no expected outcome, the judge scores User goal completion against the stopping criteria instead. If the scenario has neither, the judge infers the user’s goal from the conversation, which makes scores harder to compare between runs.

Set an expected outcome when success requires more than the condition that ends the conversation. For example, a conversation might end when the agent confirms the user’s plan choice, but success also requires the agent to compare plans within the user’s budget first.

Scenarios generated from a goal usually leave the expected outcome empty, so their stopping criteria also serve as the success criteria. To score generated scenarios against different criteria, edit the scenarios and add an expected outcome.

Write Observable Criteria

Expected outcomes and stopping criteria should both describe outcomes that can be observed in the conversation.

Prefer criteria that identify specific actions, information, or outcomes.

Observable criteria make it easier to determine whether the simulated episode reached the intended outcome.

Separate Complex Outcomes into Multiple Criteria

When successful completion requires several conditions, define each condition as a separate entry in expected_outcome, and keep stopping_criteria focused on when the conversation should end.

For example:

"expected_outcome": [
  "The agent identifies the user's monthly budget.",
  "The agent recommends at least two plans within that budget.",
  "The agent explains the relevant differences between the plans.",
  "The agent confirms which plan the user wants to choose."
],
"stopping_criteria": [
  "The agent confirms which plan the user wants to choose."
]

Separating criteria makes the expected outcome easier to understand and score than combining all requirements into one broad statement.

Set a Realistic Turn Limit

Use the turn limit (max_turns) to give the agent enough time to reasonably complete the scenario.

Simple scenarios may require only a few exchanges. More complex tasks may require additional turns when the agent needs to:

• Gather missing information.
• Ask follow-up questions.
• Use tools.
• Troubleshoot multiple steps.
• Confirm a final decision or action.

Avoid setting the limit higher than necessary because excessively long conversations can add unnecessary model usage without improving the scenario.

The max_turns value for an uploaded scenario must be from 1 through 10.

Build Scenario Sets Around a Related Objective

Group scenarios that evaluate related behaviors into the same scenario set.

For example, you might create separate scenario sets for:

• Product recommendations.
• Troubleshooting.
• Support policy compliance.
• Tool-based workflows.
• Safety and refusal behavior.
• Account or billing requests.

Keeping related scenarios together makes simulation results easier to interpret and lets you rerun the relevant set after changing a specific part of the agent.

Start with a Small Scenario Set

When developing a new scenario set, start with a few representative scenarios and review the resulting trajectories.

Check whether:

• The user simulator interprets each scenario as intended.
• The conversation exercises the behavior you wanted to test.
• The expected outcomes represent successful completion.
• The stopping criteria end conversations at the right point.
• The agent has enough turns to complete the task.
• Different scenarios produce meaningfully different conversations.

After validating the scenarios, expand the set to cover additional situations and edge cases.

A scenario set can contain up to 10 scenarios.

Review Generated Scenarios Before Reusing Them

When you generate scenarios from a goal description, review the generated scenarios before relying on them for repeated evaluations.

Verify that each scenario:

• Tests the intended behavior.
• Has a clear user objective.
• Does not duplicate another scenario in the set.
• Includes relevant stopping criteria.
• Uses a persona only when it affects the conversation.
• Can reasonably be completed within its maximum number of turns.

Edit scenarios that are too broad, repetitive, or unrelated to the goal before using the set as a recurring evaluation baseline.

Keep Scenario Sets Consistent When Comparing Changes

When comparing changes to an agent, use the same scenario set whenever possible.

Keeping scenarios consistent helps isolate the effect of changes such as:

• Agent instructions.
• Agent tools.
• The model backing the agent.
• Managed agent configuration.
• Inference Router configuration.

If you change both the agent and the scenario set at the same time, differences between simulation results may be caused by either change.

Treat a Used Scenario Set as a Fixed Baseline

While a simulation run references a scenario set, you cannot edit its scenarios. This keeps every past run readable, because the scenario you see when you review a run’s episodes is exactly the one the user simulator played. It also makes the set a stable baseline: rerun it against a new agent version, a different model, or a new Inference Router configuration, so differences in scores reflect the change you made rather than a changed scenario. Use several episodes per scenario so that run-to-run variation does not hide the effect.

To revise a set that a run has used, duplicate it and edit the copy, then treat the copy as a new baseline. Name copies so the lineage is clear, for example refund-scenarios-v2, and record which set each comparison used.