---
title: How to Create a Dataset Evaluation
description: Create and run a dataset evaluation to measure the performance of a model, Inference Router, or supported agent against a fixed evaluation dataset.
product: Evaluations
url: https://docs.digitalocean.com/products/evaluations/how-to/create-dataset-evaluations/
last_updated: "2026-10-13"
---

> **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document).

# How to Create a Dataset Evaluation

DigitalOcean Evaluations measure model, Inference Router, and agent performance using dataset evaluations and simulations.

Dataset evaluations test a [model](https://docs.digitalocean.com/products/inference/details/models/index.html.md), [Inference Router](https://docs.digitalocean.com/products/inference/how-to/use-inference-router/index.html.md), or [supported agent](https://docs.digitalocean.com/products/inference/how-to/create-agents/index.html.md) against a fixed dataset of test inputs. Use dataset evaluations for repeatable testing and to compare results across the same set of prompts.

If you want to evaluate an agent through generated, multi-turn conversations instead, see [Create a Simulation](https://docs.digitalocean.com/products/evaluations/how-to/create-simulation/index.html.md).

**Note**:

  Simulation is currently in [public preview](https://docs.digitalocean.com/platform/product-lifecycle/index.html.md#public-preview) and may change as we continue development.

DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.

## Create a Dataset Evaluation

To create a dataset evaluation, go to the [DigitalOcean Control Panel](https://cloud.digitalocean.com), in the left menu, click **DATA & LEARNING**, click **Evaluations**, and then click **New Evaluation / Simulation** to open the **Create Evaluation** page.

In the **How do you want to evaluate?** section, select **Dataset evaluation**.

If needed, click **Change** to select a different evaluation method before running the evaluation.

In the **Presets** section, choose one of the following:

- [Use an existing evaluation preset](#use-an-existing-evaluation-preset) to populate the evaluation configuration from a saved preset.
- [Configure without a preset](#configure-without-a-preset) to configure the evaluation manually.

### Use an Existing Evaluation Preset

To use an existing preset, click the **Select a preset** dropdown menu, and then select a saved preset.

Evaluation presets can automatically populate the dataset, judge model, evaluation metrics, and star metric.

After selecting a preset, the **Load preset?** window displays a summary of the preset configuration.

Then, click **Load preset** to use the preset. To select another preset or configure the evaluation manually instead, click **x** to close the window.

After loading the preset, you can edit its configuration before running the evaluation, including the [evaluation type](#choose-evaluation-type), [judge model](#configure-your-judge), [metrics](#choose-metrics), [star metric](#choose-your-star-metric), and [evaluation name](#name-the-evaluation).

### Configure Without a Preset

To configure the evaluation manually, click **Configure without a preset**, then configure the evaluation type, candidate, dataset, judge model, evaluation metrics, star metric, and optional settings.

### Choose Evaluation Type

In the **Set up evaluation** section, the evaluation type is **Single-turn** by default.

To change the evaluation type, expand the section, click the **Evaluation type** dropdown menu, and select one of the following:

- **Single-turn**: Evaluates one prompt and one candidate response for each dataset row. Use single-turn evaluations for independent requests that do not require conversational context.
- **Multi-turn**: Evaluates multiple prompt-and-response exchanges contained in each dataset row. Because the responses being evaluated are already included in the dataset, multi-turn evaluations do not require you to select a candidate model.

### Configure Your Candidate

For single-turn evaluations, select the candidate that generates responses for the prompts in your evaluation dataset. The judge model evaluates the candidate’s responses using the metrics you configure.

**Note**:

  Multi-turn evaluations do not require a candidate because the responses being evaluated are already included in the dataset.

To configure a candidate, expand the **Configure your candidate** section, in the **Select a candidate to evaluate** sub-section, click the **Select a candidate type** dropdown menu, and then select the type of candidate you want to evaluate:

- **Serverless Inference**: Evaluates a foundation model using Serverless Inference without requiring you to manage inference infrastructure.
- **Third Party**: Evaluates a supported model from a third-party provider using a configured provider key.
- **Dedicated Inference**: Evaluates a model running on an existing Dedicated Inference deployment.
- **Model Router**: Evaluates an existing Inference Router configuration that routes requests across models based on its configured routing rules.

If you need to deploy a dedicated model first, click **Deploy a model** to open the [Inference Deployment creation page](https://docs.digitalocean.com/products/inference/how-to/use-dedicated-inference/index.html.md).

For supported candidates, click **Show advanced settings** to configure the following model parameters:

- **Max Tokens**: Sets the maximum number of tokens the model can generate in a response. The default is `2048`.
- **Temperature**: Controls randomness in the model’s responses. Lower values produce more predictable responses, while higher values produce more varied responses. The default is `1`.

In the **System prompt** field, enter the system prompt the candidate uses when interpreting prompts from the evaluation dataset. For guidance on writing system prompts, see [Evaluation Best Practices](https://docs.digitalocean.com/products/evaluations/concepts/dataset-evaluations-best-practices/index.html.md#write-system-prompts-that-match-the-evaluation-goal).

### Choose a Dataset

Then, select the dataset containing the inputs you want to evaluate. For information about creating and formatting datasets, see [How to Create, Upload, and Manage an Evaluation Dataset](https://docs.digitalocean.com/products/evaluations/how-to/manage-evaluation-datasets/index.html.md).

Evaluation datasets must use CSV or JSONL format, contain up to 1,000 rows, and be up to 1 GB.

To select a dataset, expand the **Choose Dataset** section, then either click the **Select a dataset** dropdown menu and select an existing dataset, or click **Add dataset** to open the **Select dataset** window.

To upload a dataset, in the **Choose file** section, drag and drop the dataset file or click **Upload** to select a local file, and then click **Add**.

After we process the file, its status is either **Ready**, which means the dataset is valid and ready to use, or **Error**, which means the dataset couldn’t be processed.

If the status is **Error**, review the displayed error message, correct the dataset, and then upload it again.

The **Dataset Preview** section displays a preview of the uploaded dataset. For multi-turn datasets, the preview includes each conversation and its number of turns. If needed, click **Show full preview** to view the complete dataset preview.

Uploaded datasets are automatically saved so you can reuse them in future evaluations.

If you need to replace an incorrectly formatted file, click **x** next to the file, correct the dataset, and upload it again.

### Configure Your Judge

The judge model evaluates the candidate output and assigns scores for the metrics you select. For information about available models, see [Available Models](https://docs.digitalocean.com/products/inference/details/models/index.html.md#foundation-models).

To configure the judge, expand the **Configure judge & metrics** section, in the **Configure your LLM judge** sub-section, click the **Select a model to use as judge** dropdown menu, and then select the model you want to use. The dropdown also displays the model’s input and output token pricing.

DigitalOcean sends the dataset inputs, candidate outputs, and ground truth values, when available, to the selected model provider for scoring and score rationale.

Score rationale is automatically generated and may contain errors or omissions. Review and verify evaluation results before relying on them.

### Choose Metrics

Evaluation metrics define the criteria the judge model uses to score the evaluation. The available metrics depend on whether you selected a single-turn or multi-turn evaluation.

You can use built-in metrics, custom metrics, or a combination of both.

To select metrics, expand the **Configure judge & metrics** section, in the **Evaluation metrics** sub-section, select the metrics you want to use.

You can click **Select all metrics** to select all available metrics for the evaluation type.

**Note**:

  For metrics that measure desired behavior, such as Correctness, Completeness, and Ground Truth Faithfulness, higher scores are better. A row passes when its score is greater than or equal to the pass threshold.

  For metrics that measure risk, such as Bias, Toxicity, and PII Leakage, lower scores are better. A row passes when its score is less than or equal to the pass threshold.

### Single-turn metrics

Single-turn evaluations support the following metrics:

| Metric | Description | Recommendation |
|---|---|---|
| **Correctness** | Measures whether the candidate response is factually accurate and answers the query correctly. Correctness does not use the dataset ground truth field. | Use this to identify inaccurate or unsupported responses. For low scores, adjust the system prompt or try a different candidate model. |
| **Completeness** | Measures whether the candidate response fully addresses all aspects of the input query. | Use this when responses need to include specific details or fully address multi-part requests. |
| **Ground Truth Faithfulness** | Measures how closely the candidate response aligns with the expected ground truth answer. This metric requires ground truth values in the dataset. | Use this when the dataset contains expected answers or reference outputs. |
| **Answer Relevancy** | Measures whether the response directly addresses the user’s question and stays on topic without irrelevant content. | Use this to identify responses that are off-topic or contain unnecessary information. |
| **Coherence** | Measures whether the response is logically consistent and well-structured without contradictions or disjointed reasoning. | Use this when response structure and logical consistency are important. |
| **Clarity** | Measures whether the response is clear, well-organized, and understandable without ambiguity or unnecessarily complex phrasing. | Use this to identify responses that are difficult to understand. |
| **Harmfulness** | Evaluates whether the candidate produces harmful or unsafe content. You can select the following harmfulness sub-metrics:, , • **Bias**: Detects unfair or discriminatory language toward individuals or groups., • **Toxicity**: Detects harmful, offensive, or abusive content., • **PII Leakage**: Detects responses that contain or expose personally identifiable information., • **Refusal Appropriateness**: Measures whether the response appropriately refuses harmful or out-of-scope requests while continuing to help with legitimate requests., • **Prompt Injection Resistance**: Measures whether the response resists prompt-injection attempts and remains focused on the legitimate task. | Select the relevant sub-metrics for the safety behaviors you want to evaluate. |

### Multi-turn metrics

Multi-turn evaluations support the following metrics:

| Metric | Description | Recommendation |
|---|---|---|
| **User goal completion** | Measures whether the agent achieved the user’s overall goal by the end of the conversation, using the goal inferred from the conversation. | Use this to evaluate whether the conversation accomplished what the user requested. |
| **Knowledge Retention** | Measures whether the agent remembers and correctly uses information from earlier turns. | Use this when later responses depend on information provided earlier in the conversation. |
| **Conversation Completeness** | Measures whether the agent addressed all the user’s requests across the conversation. | Use this to identify conversations where requests or follow-up questions were left unresolved. |
| **Turn Relevancy** | Measures whether each agent response stays relevant to the conversation so far. | Use this to identify individual turns that drift from the current topic or context. |
| **Tool Use Accuracy** | Measures whether the agent selected the correct tools and called them with the correct arguments across the conversation. | Use this for agents that rely on tools or external actions to complete tasks. |
| **Agent Policy Adherence** | Measures whether the agent’s behavior satisfied the explicit expected-outcome criteria, or guardrails, defined for the conversation. | Use this when the conversation has specific behavioral or outcome requirements the agent must follow. |

## Add a Custom Metric

Custom metrics let you define your own criteria for evaluating responses. Use custom metrics when the built-in metrics do not cover the behavior, policy, format, or use case you want to evaluate.

For guidance on designing custom metrics, see [Dataset Evaluation Best Practices](https://docs.digitalocean.com/products/evaluations/concepts/dataset-evaluations-best-practices/index.html.md#create-focused-custom-metrics).

To use a custom metric, under **Custom metrics**, click the **Custom metrics** dropdown menu and select an existing metric.

To create a metric from the evaluation configuration, click **Add metric** to open the **Create metric** window.

In the **Metric name** field, enter a name for the metric.

Optionally, in the **Description (optional)** field, describe what the metric measures.

In the **Scoring prompt** field, enter detailed instructions that tell the judge model how to score the metric.

For example, a scoring prompt could tell the judge to evaluate whether a response follows a support policy, includes required escalation steps, avoids unsupported guarantees, or maintains a specified tone.

For single-turn evaluations, if the metric requires ground truth values from the dataset, select **Requires ground truth**. Multi-turn evaluations do not use ground truth.

If you select **Requires ground truth** and the evaluation dataset does not contain ground truth values, DigitalOcean skips the metric during the evaluation.

Then, click **Create**. Afterwards, your custom metric is displayed.

### Edit Custom Metric

If you want to edit your custom metric, click **…** next to the metric, and then click **Edit metric**. Update the name, description, scoring prompt, and ground truth requirement, and then click **Save**.

If you want to remove the metric from your evaluation, click **…** next to the metric, and then click **Remove metric**.

If you want to delete the metric entirely, click **…** next to the metric, and then click **Delete** to open the **Delete metric** window. If any saved presets use the metric, the window lists them, and deleting the metric also deletes those presets.

Deleting this custom metric permanently removes it from future evaluations. This action cannot be undone.

To confirm your deletion, enter the name of the metric, and then click **Delete metric**.

### Choose Your Star Metric

The star metric is the primary metric you use to determine whether the evaluation meets your success criteria. Choose the metric that most directly represents the behavior or outcome you are testing.

To configure the star metric, expand the **Configure judge & metrics** section, in the **Star Metric** sub-section, click **Change** to open the **Select star metric** window. A star metric is selected from the metrics configured for the evaluation.

Then, click the star icon next to the metric you want to use, and then click **Update star metric**.

In the **Star metric pass threshold** field, enter a threshold from 0 through 100.

For metrics where higher scores are better, a row passes when its score is greater than or equal to the threshold.

For metrics where lower scores are better, a row passes when its score is less than or equal to the threshold.

Review the results of several evaluation runs and adjust the threshold based on the scores you consider acceptable for your use case.

### Name the Evaluation

To name the evaluation, expand the **Optional settings** section, in the **Name your evaluation run (optional)** field, enter a name that identifies the run.

Using descriptive names can make it easier to distinguish runs when comparing evaluations later.

### Save the Configuration as a Preset

You can save the evaluation configuration as a preset for reuse.

To save it, expand the **Optional settings** section, and then select **Save this configuration as a preset**.

In the **Preset name** field, enter a name for the preset.

In the **Choose what to save** section, either click **Select all sections** or select the individual parts of the configuration you want the preset to retain.

## Run the Evaluation

After configuring the evaluation, click **Run Evaluation**.

**Note**:

  To cancel a queued evaluation, return to the **Evaluations & Simulations** page, click **…** next to the evaluation, and then click **Cancel Evaluation**.

After the evaluation finishes, click its name to open the results page.

Our [data privacy policy](https://docs.digitalocean.com/products/inference/details/data-privacy/index.html.md) describes zero data retention for this flow. Your data is not stored outside of DigitalOcean or used to train models.

## Review Evaluation Results

After an evaluation finishes, its results page displays:

- **Overall Score**: The overall score for the evaluation run.
- **Dataset**: The dataset used for the evaluation.
- **Star Metric**: The primary metric used to determine evaluation success.
- **Pass Threshold**: The threshold configured for the star metric.
- **Candidate Tokens**: The candidate model’s input, output, and total token usage.
- **Judge Tokens**: The judge model’s input, output, and total token usage.
- **Candidate Latency**: The candidate model’s average, P50, P90, P95, minimum, and maximum latency.

The results chart displays the percentage of dataset entries that passed or failed each selected metric.

The **Evaluation results** table displays each dataset entry’s input, candidate output, metric, score, and judge rationale.

Click **Show full output** for an entry to view its complete candidate response.

To download the evaluation results, in the top-right corner, click **Download results**.

For additional actions you can take on completed evaluations, see [Manage Evaluations](https://docs.digitalocean.com/products/evaluations/how-to/manage-evaluations/index.html.md).

For guidance on choosing representative prompts, covering edge cases, and writing ground truth responses, see [Evaluation Dataset Design Best Practices](https://docs.digitalocean.com/products/evaluations/concepts/evaluation-dataset-best-practices/index.html.md).

## Run an Evaluation Using the API

You can also create and retrieve dataset evaluations using the DigitalOcean API.

First, upload an evaluation dataset:

## How to Upload an Evaluation Dataset Using the DigitalOcean API

[Create a personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/index.html.md) and save it for use with the API.

### cURL

Send a POST request to [`https://api.digitalocean.com/v2/gen-ai/model_evaluation/datasets/file_upload_presigned_urls`](https://docs.digitalocean.com/reference/api/reference/agent-platform-api/index.html.md#genai_create_model_eval_dataset_upload_presigned_urls).

Using cURL:

```shell
curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation/datasets/file_upload_presigned_urls" \
  -d '{
    "files": [{
      "file_name": "dataset.jsonl",
      "file_size": 2048
    }]
  }'
```

Then, create an evaluation run. You need the candidate and judge model IDs.

Use the [List Available Models endpoint](https://docs.digitalocean.com/reference/api/reference/agent-platform-api/index.html.md#genai_list_models) to retrieve supported models and Inference Routers.

**Note**:

  Not all models support evaluations.

After you have the candidate and judge model IDs, create the evaluation run:

## How to Create a Model Evaluation Run Using the DigitalOcean API

[Create a personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/index.html.md) and save it for use with the API.

### cURL

Send a POST request to [`https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs`](https://docs.digitalocean.com/reference/api/reference/agent-platform-api/index.html.md#genai_create_model_evaluation_run).

Using cURL:

```shell
curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs" \
  -d '{
    "name": "my-evaluation-run",
    "candidate_model_uuid": "123e4567-e89b-12d3-a456-426614174000",
    "judge_model_uuid": "223e4567-e89b-12d3-a456-426614174001",
    "dataset_uuid": "323e4567-e89b-12d3-a456-426614174002",
    "metric_uuids": [
      "423e4567-e89b-12d3-a456-426614174003"
    ]
  }'
```

After the run finishes, retrieve the evaluation metrics:

## How to List Model Evaluation Metrics Using the DigitalOcean API

[Create a personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/index.html.md) and save it for use with the API.

### cURL

Send a GET request to [`https://api.digitalocean.com/v2/gen-ai/model_evaluation_metrics`](https://docs.digitalocean.com/reference/api/reference/agent-platform-api/index.html.md#genai_list_model_evaluation_metrics).

Using cURL:

```shell
curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_metrics"
```

You can also download a JSON file containing additional evaluation result details:

## How to Download Evaluation Results Using the DigitalOcean API

[Create a personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/index.html.md) and save it for use with the API.

### cURL

Send a GET request to [`https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs/{eval_run_uuid}/results/download_url`](https://docs.digitalocean.com/reference/api/reference/agent-platform-api/index.html.md#genai_get_model_evaluation_run_results_download_url).

Using cURL:

```shell
curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs/123e4567-e89b-12d3-a456-426614174000/results/download_url"
```