For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

DigitalOcean Evaluations help you measure model, Inference Router, and agent performance before using a configuration in production.

You can run:

• Dataset evaluations to test responses against a fixed dataset of single-turn inputs or evaluate existing multi-turn conversations.
• Simulation to test agent behavior through generated, multi-turn conversations.

DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.

Run a Dataset Evaluation

Dataset evaluations use a fixed dataset of test inputs. Use them for repeatable testing and when you want to compare results across the same inputs.

Before creating a dataset evaluation, create or upload the dataset containing the inputs you want to evaluate.

For instructions on formatting and uploading single-turn or multi-turn datasets, see How to Create, Upload, and Manage an Evaluation Dataset.

To run a dataset evaluation:

1. Go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, and then click New Evaluation / Simulation.

2. Select Dataset evaluation.

3. You can load an existing evaluation preset, or click Configure without a preset to configure the evaluation manually.

   If you decide to use a preset, you can still edit the loaded evaluation configuration before running the evaluation. Afterwards, run the evaluation.

   If you decide not to use a preset, further configure your evaluation.

Configure a Dataset Evaluation

To configure a dataset evaluation:

1. In the Set up evaluation section, click the Evaluation type dropdown menu, and then select Single-turn or Multi-turn:

Run and Review the Dataset Evaluation

After configuring your dataset evaluation:

1. Click Run Evaluation.

2. After the evaluation finishes, on the Evaluations & Simulations page, find and open the evaluation to review its results.

For all available configuration options, see How to Create a Dataset Evaluation.

Run a Simulation public

Simulations test an agent through generated, multi-turn conversations. Use them when you want to evaluate how an agent behaves across realistic scenarios instead of replaying a fixed dataset.

Before creating a simulation, create or upload the scenario set you want to use.

For instructions on generating scenarios or uploading a JSONL scenario set, see How to Generate and Manage a Scenario Set.

To run a simulation:

1. Go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, and then click New Evaluation / Simulation.

2. Select Simulations. If you have saved simulation presets, either load a preset or click Configure without a preset to configure the simulation manually.

3. In the Choose agent section, under Agent source, choose how you want to provide the agent for the simulation:

   Managed agent

   Use Managed agent to evaluate an existing deployed agent, including its configured tools, instructions, and other agent settings.

   In the Agent source section, click Managed agent, click the Select a managed agent dropdown menu, and then select the agent you want to evaluate.

   If you need to create an agent first, see Create a Managed Agent.

   Agent from a model

   Use Agent from a model to evaluate a model or Inference Router without deploying a managed agent first. The simulation creates an agent configuration that uses the selected model.

   In the Agent source section, click Agent from a model. In the Model sub-section, click the Select a model type dropdown menu, and then select one of the following types: Serverless Inference, Third Party, Dedicated Inference, or Model Router.

   Then, click the Select a model dropdown menu, and then select one of the options available for that type.

   Note

   When you start the simulation, DigitalOcean creates a model access key and an agent configuration on your team. Both remain after the run ends, and you can select the agent configuration as a managed agent in later runs. For details, see Create a Simulation.

4. Expand Choose simulation scenarios, click the Scenario set dropdown menu, and then select the scenario set you created earlier.

5. Expand Configure models & metrics, click the User simulator model dropdown menu, and then select the model that plays the end user during each scenario.

6. In the Judge model sub-section, click the Select a judge model dropdown menu, and then select the model that scores each simulated episode using the selected evaluation metrics.

7. Under Evaluation metrics, select the metrics you want to use to score the simulated conversations. By default, User goal completion is selected. To select all available metrics, click Select all.

8. Optionally, expand Optional settings to configure the simulation run:

   In the Run name field, enter a name for the simulation.

   In the Turn limit per scenario field, set the maximum number of user-assistant exchanges that can occur before a scenario ends. The default is 5.

   In the Episodes per scenario field, set how many times each scenario runs. The default is 1.

9. After configuring the simulation, click Run simulation.

   The results page displays the simulation status, number of scenarios and episodes, duration, configuration, and a breakdown of the episodes for each scenario.

   To inspect an individual episode, click View trajectory. The trajectory displays the scenario, user persona, token usage, number of turns, duration, and messages exchanged between the user simulator and agent.

For all available configuration options, see How to Create a Simulation.