---
title: DigitalOcean Evaluations Quickstart
description: Create and run dataset evaluations and simulations to measure model, Inference Router, and agent performance.
product: Evaluations
url: https://docs.digitalocean.com/products/evaluations/getting-started/quickstart/
last_updated: "2026-10-13"
---

> **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document).

# DigitalOcean Evaluations Quickstart

DigitalOcean Evaluations measure model, Inference Router, and agent performance using dataset evaluations and simulations.

DigitalOcean Evaluations help you measure model, Inference Router, and agent performance before using a configuration in production.

You can run:

- **Dataset evaluations** to test responses against a fixed dataset of single-turn inputs or evaluate existing multi-turn conversations.
- **Simulation** to test agent behavior through generated, multi-turn conversations.

DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.

## Run a Dataset Evaluation

Dataset evaluations use a fixed dataset of test inputs. Use them for repeatable testing and when you want to compare results across the same inputs.

Before creating a dataset evaluation, create or upload the dataset containing the inputs you want to evaluate.

For instructions on formatting and uploading single-turn or multi-turn datasets, see [How to Create, Upload, and Manage an Evaluation Dataset](https://docs.digitalocean.com/products/evaluations/how-to/manage-evaluation-datasets/index.html.md).

To run a dataset evaluation:

1. Go to the [DigitalOcean Control Panel](https://cloud.digitalocean.com), in the left menu, click **DATA & LEARNING**, click **Evaluations**, and then click **New Evaluation / Simulation**.
2. Select **Dataset evaluation**.
3. You can load an existing evaluation preset, or click **Configure without a preset** to configure the evaluation manually.

   If you decide to use a preset, you can still edit the loaded evaluation configuration before running the evaluation. Afterwards, [run the evaluation](#run).

   If you decide not to use a preset, [further configure your evaluation](#configure).

### Configure a Dataset Evaluation

To configure a dataset evaluation:

1. In the **Set up evaluation** section, click the **Evaluation type** dropdown menu, and then select **Single-turn** or **Multi-turn**:

### Single-turn

Use a single-turn evaluation to evaluate one prompt and candidate response for each dataset row.

2. Expand **Configure your candidate**, click the **Select a candidate type** dropdown menu, and then select **Serverless Inference**, **Third Party**, **Dedicated Inference**, or **Model Router**.

   Then, click the **Select a candidate** dropdown menu, and then choose the candidate you want to evaluate.
3. Expand the dataset section, click the **Select a dataset** dropdown menu to choose an existing dataset, or click **Add dataset** to upload a new one.
4. Expand **Configure judge & metrics**, under **Configure your LLM judge**, click the **Select a model to use as judge** dropdown menu, and then select the judge model that scores the candidate responses.
5. Under **Evaluation metrics**, select the built-in metrics, custom metrics, or combination of both that measure the behaviors you want to evaluate.
6. Under **Star Metric**, select the metric that represents the primary success criterion for the evaluation.

   To choose a different metric, click **Change**, click the star icon next to the metric you want to use, and then click **Update star metric**.
7. In the **Star metric pass threshold** field, enter a threshold from `0` through `100`.

   For metrics where higher scores are better, entries pass when their score is greater than or equal to the threshold.

   For metrics where lower scores are better, entries pass when their score is less than or equal to the threshold.
8. Optionally, expand **Optional settings** and enter a name in the **Name your evaluation run (optional)** field.

### Multi-turn

Multi-turn evaluations score existing prompt-and-response exchanges contained in each dataset row, so you don’t need to configure a candidate.

2. Expand the dataset section, click the **Select a dataset** dropdown menu to choose an existing multi-turn dataset, or click **Add dataset** to upload a new one.
3. Expand **Configure judge & metrics**, under **Configure your LLM judge**, click the **Select a model to use as judge** dropdown menu, and then select the judge model that scores the conversations.
4. Under **Evaluation metrics**, select the multi-turn metrics that measure the conversation behaviors you want to evaluate.
5. Under **Star Metric**, select the metric that represents the primary success criterion for the evaluation.

   To choose a different metric, click **Change**, click the star icon next to the metric you want to use, and then click **Update star metric**.
6. In the **Star metric pass threshold** field, enter a threshold from `0` through `100`.

   For metrics where higher scores are better, entries pass when their score is greater than or equal to the threshold.

   For metrics where lower scores are better, entries pass when their score is less than or equal to the threshold.
7. Optionally, expand **Optional settings** and enter a name in the **Name your evaluation run (optional)** field.

### Run and Review the Dataset Evaluation

After configuring your dataset evaluation:

1. Click **Run Evaluation**.
2. After the evaluation finishes, on the **Evaluations & Simulations** page, find and open the evaluation to review its results.

For all available configuration options, see [How to Create a Dataset Evaluation](https://docs.digitalocean.com/products/evaluations/how-to/create-dataset-evaluations/index.html.md).

## Run a Simulation (public)

Simulations test an agent through generated, multi-turn conversations. Use them when you want to evaluate how an agent behaves across realistic scenarios instead of replaying a fixed dataset.

Before creating a simulation, create or upload the scenario set you want to use.

For instructions on generating scenarios or uploading a JSONL scenario set, see [How to Generate and Manage a Scenario Set](https://docs.digitalocean.com/products/evaluations/how-to/manage-scenario-sets/index.html.md).

To run a simulation:

1. Go to the [DigitalOcean Control Panel](https://cloud.digitalocean.com), in the left menu, click **DATA & LEARNING**, click **Evaluations**, and then click **New Evaluation / Simulation**.
2. Select **Simulations**. If you have saved simulation presets, either load a preset or click **Configure without a preset** to configure the simulation manually.
3. In the **Choose agent** section, under **Agent source**, choose how you want to provide the agent for the simulation:

   ### Managed agent

   Use **Managed agent** to evaluate an existing deployed agent, including its configured tools, instructions, and other agent settings.

   In the **Agent source** section, click **Managed agent**, click the **Select a managed agent** dropdown menu, and then select the agent you want to evaluate.

   If you need to create an agent first, see [Create a Managed Agent](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/how-to/create-managed-agent/index.html.md).

   ### Agent from a model

   Use **Agent from a model** to evaluate a model or Inference Router without deploying a managed agent first. The simulation creates an agent configuration that uses the selected model.

   In the **Agent source** section, click **Agent from a model**. In the **Model** sub-section, click the **Select a model type** dropdown menu, and then select one of the following types: **Serverless Inference**, **Third Party**, **Dedicated Inference**, or **Model Router**.

   Then, click the **Select a model** dropdown menu, and then select one of the options available for that type.

   **Note**:

     When you start the simulation, DigitalOcean creates a model access key and an agent configuration on your team. Both remain after the run ends, and you can select the agent configuration as a managed agent in later runs. For details, see [Create a Simulation](https://docs.digitalocean.com/products/evaluations/how-to/create-simulation/index.html.md).
4. Expand **Choose simulation scenarios**, click the **Scenario set** dropdown menu, and then select the scenario set you created earlier.
5. Expand **Configure models & metrics**, click the **User simulator model** dropdown menu, and then select the model that plays the end user during each scenario.
6. In the **Judge model** sub-section, click the **Select a judge model** dropdown menu, and then select the model that scores each simulated episode using the selected evaluation metrics.
7. Under **Evaluation metrics**, select the metrics you want to use to score the simulated conversations. By default, **User goal completion** is selected. To select all available metrics, click **Select all**.
8. Optionally, expand **Optional settings** to configure the simulation run:

   In the **Run name** field, enter a name for the simulation.

   In the **Turn limit per scenario** field, set the maximum number of user-assistant exchanges that can occur before a scenario ends. The default is `5`.

   In the **Episodes per scenario** field, set how many times each scenario runs. The default is `1`.
9. After configuring the simulation, click **Run simulation**.

   The results page displays the simulation status, number of scenarios and episodes, duration, configuration, and a breakdown of the episodes for each scenario.

   To inspect an individual episode, click **View trajectory**. The trajectory displays the scenario, user persona, token usage, number of turns, duration, and messages exchanged between the user simulator and agent.

For all available configuration options, see [How to Create a Simulation](https://docs.digitalocean.com/products/evaluations/how-to/create-simulation/index.html.md).