For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Scenario sets contain the scenarios used in simulations. Each scenario describes a situation for the user simulator and agent to act out and defines the criteria for determining when the scenario is complete.

You can create a scenario set by generating scenarios from a goal description, uploading your own scenarios in JSONL format, or importing an entry from the Common Scenario & Goal Library.

For guidance on writing focused scenarios, defining stopping criteria, using personas, and building representative scenario sets, see Scenario Set Best Practices.

Create a Scenario Set

To create a scenario set, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, click the Simulation Scenarios tab, click My Scenarios, and then choose one of the following:

• Generate scenarios from a goal when you want a model to create scenarios based on the behavior or task you want to test.
• Import scenarios by either uploading a JSONL file or importing from Signals:
  • Upload a JSONL file when you already have scenarios you want to use or need direct control over their descriptions, personas, stopping criteria, or turn limits.
  • Import from Signals when you want to replay real agent conversations where Signals detected a problem.
• Import from the scenario library when you want a ready-made set of scenarios as a starting point.

Generate Scenarios from a Goal

Generate a scenario set when you want DigitalOcean to create scenarios based on a description of what you want to test.

In the top-right corner, click Scenario builder to open the Build a scenario set window.

In the Set name field, enter a name for the scenario set. For example, support-agent-scenarios.

Optionally, to tailor the scenarios to an agent, click the Managed agent (optional) dropdown menu, and then select a managed agent. Scenario Builder summarizes what the agent can do in the Agent capability summary field and drafts a suggested goal in the Scenario goals field. You can edit the summary to adjust what you want to simulate, and then click Redraft from agent capability summary to draft a new goal from it.

In the Scenario goals field, describe the behavior, task, or user goal you want the generated scenarios to test, or edit the drafted goal. For example, “The user wants to compare the features of the two most recent Python versions to decide which one to use.” To refine the goal, edit it directly, or click Refine goal. The Refine the goal panel suggests options for Persona, Situation, and Sentiment based on the current goal. Pick one option for each, edit it if needed, or select Write your own, and then click Update goal to rewrite the goal from your picks. If you edit the goal after the suggestions appear, click Refresh suggestions to get options based on the new goal. While the Refine the goal panel is open, Generate is unavailable.

In the Model section, click the model dropdown menu, and then select the model you want to use to generate scenarios from the goal in the Scenario goals field. By default, DeepSeek V4 Flash 0731 is selected.

Choose a model based on the complexity of the scenarios you want to generate and the cost you want to incur. For less complex goals, a lower-cost model is often enough. For goals that require nuanced, varied, or complex scenarios, use a more capable model.

In the Number of scenarios field, enter the number of scenarios you want the model to generate. The default is 1.

Generate fewer scenarios when you want to test a focused behavior or review the generated scenarios quickly. Generate more scenarios when you want broader coverage of different ways the goal might be expressed or completed.

Then, click Generate. Generation runs in the background and can take a few minutes. You can close the generation window or leave the page. When generation finishes, the scenario set appears in the My Scenarios list.

Selecting a scenario set in My Scenarios displays the number of scenarios in the set and an overview of each scenario. To configure a new simulation run with the set, click Run simulation.

Upload a JSONL File

Upload a JSONL scenario set when you already have scenarios you want to evaluate or need to define each scenario yourself.

In the top-right corner, click Upload JSONL to open the Upload a scenario set window.

In the Set name field, enter a name for the scenario set. For example, support-agent-scenarios.

In the Scenario file section, either drag and drop a JSONL file, or click Upload JSONL to select a file from your computer.

Each line in the JSONL file must contain one scenario.

A scenario supports the following fields:

A scenario set can contain up to 10 scenarios, and the file can be up to 5 MB.

For example, a JSONL file could contain:

{"name":"Choosing a New Phone Plan","description":"A user asks the agent to find the best mobile phone plan for their usage and budget.","user_persona":"A cost-conscious individual looking for a new plan.","stopping_criteria":["The agent confirms the user's choice and provides an application link."],"expected_outcome":["The agent suggests at least three different phone plans.","The agent compares the advantages of each plan.","The agent confirms the user's choice and provides an application link."],"max_turns":10}

After selecting the file, click Upload.

Import from the Scenario Library

The Common Scenario & Goal Library contains ready-made scenario sets that DigitalOcean provides as templates. DigitalOcean writes these templates, and they do not come from other customers’ data or usage.

To import a library entry, on the Simulation Scenarios tab, click Scenario Library. To find an entry, search by name in the search box, or click Filter and then select a category.

To preview an entry’s scenarios, click the entry’s name, or click … to the right of the entry and then click Preview scenarios. Library scenarios are read-only.

To import the entry, click Import to My Scenarios in the preview, or click … to the right of the entry and then click Import scenario. The import starts immediately and uses the library entry’s name for the new scenario set. When the import finishes, the My Scenarios tab opens. To change the scenario set’s name, rename it.

Importing an entry creates a copy of its scenarios as a new scenario set in My Scenarios. Your team owns the copy, and any edits you make to it are private to your team. The new scenario set is ready to use in a simulation. Importing an entry does not start a simulation.

Import from Signals

If you use Signals with a Harness Runtime agent, you can turn conversations where Signals detected a problem into a scenario set. When a Signals export completes, DigitalOcean generates a scenario set from the exported conversations, so you can replay those situations in a simulation and check whether changes to your agent fix them.

To find the sessions you want to replay, list the agent’s sessions that contain a specific signal type:

doctl signals session list --agent-id <agent-id> --signal-type <signal-type>

Then, create an export that filters on that signal type and lists the session IDs you want to include. Repeat --session-ids for each session:

doctl signals export create --agent-id <agent-id> --signal-type <signal-type> --session-ids <session-id> --session-ids <session-id>

For more export options, see Export Signal Data.

Each export creates one scenario set based on the export’s signal type. If the export filters on more than one signal type, the scenario set uses the first one. Only the following signal types create a scenario set:

• Interaction signals: MisalignmentCorrection, MisalignmentRephrase, MisalignmentClarification, StagnationRepetition, DisengagementEscalation, DisengagementQuit, and DisengagementNegativeStance.
• Execution signals: ExecutionFailureInvalidArgs, ExecutionFailureBadQuery, ExecutionFailureToolNotFound, ExecutionFailureAuthMisuse, ExecutionFailureStateError, ExecutionLoopsRetry, ExecutionLoopsParameterDrift, and ExecutionLoopsOscillation.

Exports without a signal type filter, and exports for other signal types such as satisfaction and environment signals, do not create a scenario set.

The new scenario set appears in My Scenarios on the Simulation Scenarios tab. While its scenarios generate, the set is named after the signal type, such as MisalignmentRephrase (signals). When generation finishes, the set is renamed with a generated title that describes its scenarios. You can then preview, edit, rename, or run a simulation with the set like any other scenario set.

Manage a Scenario Set

After creating a scenario set, you can use it to run a simulation, preview or edit its scenarios, download it as JSONL, duplicate it, rename it, or delete it.

To manage a scenario set, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, click the Simulation Scenarios tab, click My Scenarios, and then click the name of the scenario set you want to manage to open its preview. The preview displays the scenarios in the set and the total number of scenarios.

To act on the scenario set, in the preview, click Run simulation, or click … and then click the action you want.

Run a Simulation

To use a scenario set in a simulation, in the preview, click Run simulation.

This opens the simulation configuration with the scenario set selected.

For information on configuring and running the simulation, see How to Create a Simulation.

Preview or Edit Scenarios

To preview a scenario set, click its name in the My Scenarios list.

You cannot change the scenarios in a set while any simulation run references it. This keeps every past run readable: when you review a run’s episodes, the scenario you see is exactly the one the user simulator played. To change a scenario set that a run has already used, duplicate the set, and then edit the copy. You can still rename the set.

To change the scenarios, in the top-right corner of the preview’s Scenarios section, click Edit scenarios. The button appears only for scenario sets you can edit, which are sets that no simulation run references.

For existing scenarios, you can edit the following:

• In the Scenario name field, enter a descriptive name for the scenario.
• In the Simulator prompt field, describe the situation the simulated user should act out, including the user’s goal and any relevant context.
• In the Persona field, describe the simulated user’s role, characteristics, or circumstances.
• Under Expected outcome, describe what a successful conversation achieves. The judge uses the expected outcome to score the User goal completion metric. To add another outcome, click Add outcome.
• Under Stopping criteria, define the conditions that end the conversation. If you don’t add any, the conversation ends based on the simulator prompt. To add another condition, click Add criterion.
• In the Turn limit field, enter the maximum number of conversation turns the simulation can run before it stops.

To test another scenario, click Add scenario and configure the additional scenario. You can only have 10 scenarios in a set.

To remove a scenario, click the trash icon in the upper-right corner of the scenario.

Afterwards, click Save scenarios.

Download a Scenario Set

To download a scenario set as a JSONL file, in the preview, click …, and then click Download JSONL.

Duplicate a Scenario Set

To create a copy of a scenario set, in the preview, click …, and then click Duplicate.

Duplicate a scenario set when you want to change scenarios in a set that a simulation run has already used.

Rename a Scenario Set

To rename a scenario set, in the preview, click …, and then click Rename to open the Rename scenario set window.

In the Set name field, enter the new name for the scenario set, and then click Save.

Delete a Scenario Set

Deleting a scenario set removes it from My Scenarios. You cannot delete a scenario set while any simulation run references it. To delete the set, first delete the simulation runs that used it.

To delete a scenario set, in the preview, click …, and then click Delete to open the Delete scenario set window.

To confirm deletion, enter the name of the scenario set, and then click Delete.

Manage Scenario Sets Using the API

You can also create and manage scenario sets using the DigitalOcean API.

Generate a Scenario Set Using the API

To generate scenarios from a goal description, send the goal and a name for the set. The number of scenarios and the generator model are optional. If you omit them, the request generates one scenario using the platform default model.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/generate" \
  -d '{
    "name": "billing-support-generated",
    "goal_description": "Test how the agent handles refund requests",
    "num_scenarios": 4
  }'

The request returns the new scenario set in the generating state. Poll the retrieve scenario set endpoint until the set reaches the ready state before you use it in a simulation run.

Upload a Scenario Set Using the API

To upload a JSONL scenario file, first create a presigned URL for the upload:

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/file_upload_presigned_urls" \
  -d '{
    "files": [{
      "file_name": "scenarios.jsonl",
      "file_size": 2048
    }]
  }'

Upload your JSONL file to the presigned URL in the response. Then, create the scenario set and pass the uploaded file in the file_upload_scenario_set field.

Alternatively, you can create a scenario set without a file by passing the scenarios inline in the scenarios field. Each scenario requires a description. Provide either scenarios or file_upload_scenario_set, not both.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets" \
  -d '{
    "name": "support-agent-scenarios",
    "scenarios": [
      {
        "name": "Refund request",
        "description": "The user wants a refund for a delayed order",
        "user_persona": "Frustrated customer",
        "stopping_criteria": ["Refund is approved or a next step is clearly offered"],
        "max_turns": 8
      }
    ]
  }'

Scenario sets created this way are ready to use immediately.

Import from the Scenario Library Using the API

To browse the Common Scenario & Goal Library, list its entries. You can filter the list by category or search entry names and descriptions.

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_library"

To preview the scenarios in a library entry:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_library/12345678-1234-1234-1234-123456789012/scenarios"

To copy a library entry into a new scenario set owned by your team, create a scenario set from the entry. The name is optional and defaults to the library entry’s name. You can also pass a library entry directly when you create a simulation run.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_library/12345678-1234-1234-1234-123456789012/create_scenario_set" \
  -d '{
    "library_scenario_uuid": "12345678-1234-1234-1234-123456789012",
    "name": "Billing Refund Escalation"
  }'

Retrieve Scenario Sets Using the API

To list the scenario sets on your team:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets"

To retrieve a scenario set, including its status:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/123e4567-e89b-12d3-a456-426614174000"

To list the scenarios in a scenario set:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/12345678-1234-1234-1234-123456789012/scenarios"

To download a scenario set as a JSONL file, retrieve a download URL:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/12345678-1234-1234-1234-123456789012/download_url"

Update, Duplicate, or Delete a Scenario Set Using the API

To rename a scenario set or replace its scenarios, update the set. You can rename a set at any time, but you can only replace the scenarios in a set that is ready and that no simulation run references. Replacing the scenarios keeps the scenario set UUID but assigns new UUIDs to the scenarios. You cannot upload a file when updating a set.

curl -X PUT \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/12345678-1234-1234-1234-123456789012" \
  -d '{
    "name": "My new scenario set name"
  }'

To change the scenarios in a set that a simulation run references, duplicate the set, and then update the copy:

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/12345678-1234-1234-1234-123456789012/duplicate" \
  -d '{
    "scenario_set_uuid": "12345678-1234-1234-1234-123456789012"
  }'

To delete a scenario set, first delete any simulation runs that reference it. The request fails while any run references the set.

curl -X DELETE \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/scenario_sets/12345678-1234-1234-1234-123456789012"