---
title: How to Create, Upload, and Manage an Evaluation Dataset
description: Create, format, upload, preview, download, and delete single-turn and multi-turn evaluation datasets.
product: Evaluations
url: https://docs.digitalocean.com/products/evaluations/how-to/manage-evaluation-datasets/
last_updated: "2026-10-12"
---

> **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document).

# How to Create, Upload, and Manage an Evaluation Dataset

DigitalOcean Evaluations measure model, Inference Router, and agent performance using dataset evaluations and simulations.

Evaluation datasets contain the inputs used for dataset evaluations. The required dataset format depends on whether you are running a single-turn or multi-turn evaluation.

For guidance on choosing representative inputs, covering edge cases, and writing ground truth responses, see [Evaluation Dataset Design Best Practices](https://docs.digitalocean.com/products/evaluations/concepts/evaluation-dataset-best-practices/index.html.md).

To create an evaluation dataset, go to the [DigitalOcean Control Panel](https://cloud.digitalocean.com), in the left menu, click **DATA & LEARNING**, click **Evaluations**, click the **Datasets** tab, and then in the top-right corner, click **Upload dataset** to open the **Select dataset** window.

## Choose Dataset Format

Choose the dataset paradigm based on whether each dataset entry contains a single input for a candidate to answer or an existing multi-turn conversation you want to evaluate.

In the **Paradigm** section, click the dropdown menu, and then choose one of the following:

- **Single-turn** (default): Use when each dataset entry contains an independent query that you want a candidate to answer. Single-turn evaluations generate a candidate response for each query and score the response using the metrics you select.
- **Multi-turn**: Use when each dataset entry contains an existing conversation with multiple user and assistant messages. Multi-turn evaluations score the conversation already contained in the dataset and do not require a candidate model.

You can upload a dataset of either paradigm as a CSV or JSONL file. In a CSV file, the first row contains the column names. In a JSONL file, each line is a JSON object that uses the column names as its fields.

### Format a Single-Turn Dataset

Single-turn evaluation datasets must include an `input` column. The `input` column contains the prompt you want the candidate to answer, and every row must have a non-empty value. DigitalOcean ignores any other columns.

For example, a single-turn dataset could contain inputs like this:

```csv
input
"What makes DigitalOcean different from other cloud providers?"
"Explain the benefits of using DigitalOcean for startups."
```

Some single-turn metrics require ground truth values. Ground truth values are expected responses that the judge model uses to compare against the candidate’s response.

For a ground truth dataset, include a `ground_truth` column that contains the expected answer for each input.

For example, a single-turn dataset with ground truth responses could contain:

```csv
input,ground_truth
"What is a Droplet?","A Droplet is a virtual machine that runs on DigitalOcean's cloud infrastructure."
"How do I add a member to my Team account?","To add a member to your DigitalOcean Team account, go to the DigitalOcean Control Panel, open your team settings, click the Members tab, and then click Invite Team Member."
"What is the pricing for Droplets?","Droplets have flexible pricing based on instance type."
```

The same dataset in JSONL format contains one object per line:

```json
{"input":"What is a Droplet?","ground_truth":"A Droplet is a virtual machine that runs on DigitalOcean's cloud infrastructure."}
{"input":"What is the pricing for Droplets?","ground_truth":"Droplets have flexible pricing based on instance type."}
```

Use ground truth responses when you want to evaluate whether a candidate’s answer matches a known correct answer. For example, ground truth datasets are useful for support questions, policy answers, factual product documentation, and other cases where the expected response is known.

For guidance on writing effective ground truth responses, see [Evaluation Dataset Design Best Practices](https://docs.digitalocean.com/products/evaluations/concepts/evaluation-dataset-best-practices/index.html.md#write-representative-ground-truth-responses).

### Format a Multi-Turn Dataset

Multi-turn evaluation datasets must include `id` and `messages` columns.

The `id` column identifies the conversation. Each conversation must have a non-empty `id` that is unique in the dataset.

The `messages` column contains the conversation to evaluate as a non-empty JSON array of messages. Each message must have a non-empty `role`, such as `user` or `assistant`, and a `content` field. The `content` value can be an empty string, but the field must be present. DigitalOcean ignores other columns and keeps additional message fields, such as `tool_calls`.

For example, a multi-turn dataset could contain a conversation like this:

```csv
id,messages
1,"[{""role"":""user"",""content"":""I need to update my mailing address.""},{""role"":""assistant"",""content"":""I can help with that. What is your new mailing address?""},{""role"":""user"",""content"":""123 Example Street.""},{""role"":""assistant"",""content"":""Your mailing address has been updated to 123 Example Street.""}]"
```

The same conversation in JSONL format is one line:

```json
{"id":"1","messages":[{"role":"user","content":"I need to update my mailing address."},{"role":"assistant","content":"I can help with that. What is your new mailing address?"},{"role":"user","content":"123 Example Street."},{"role":"assistant","content":"Your mailing address has been updated to 123 Example Street."}]}
```

Multi-turn evaluations score the conversation already contained in the dataset, so you don’t need to select a candidate model. Multi-turn datasets don’t use ground truth values.

## Review Dataset

Before uploading the dataset, review its format and contents to make sure DigitalOcean can process it correctly.

When reviewing your evaluation dataset, ensure the following requirements are satisfied:

- Use a CSV or JSONL file of up to 1 GB, with up to 1,000 rows or conversations.
- For single-turn evaluations, include an `input` column.
- Include a `ground_truth` column if you plan to use a single-turn metric that requires ground truth.
- For multi-turn evaluations, include `id` and `messages` columns.
- Use [UTF-8 encoding](https://en.wikipedia.org/wiki/UTF-8).
- Review exported spreadsheet files for invalid or non-UTF-8 characters before uploading them.
- Remove empty rows when possible. Empty rows are skipped during scoring, but any tokens used to process them still count toward evaluation cost.
- Make sure the file is readable and correctly formatted. If required columns are missing, or your file is malformed or unreadable, the dataset upload fails.

We recommend using 50 to 100 entries for faster feedback and lower evaluation costs.

After reviewing your dataset, click **Add**.

## Manage Datasets

After uploading a dataset, you can preview its contents, download the original dataset, or delete it from your team.

To manage your datasets, go to the [DigitalOcean Control Panel](https://cloud.digitalocean.com), in the left menu, click **DATA & LEARNING**, click **Evaluations**, and then click the **Datasets** tab.

### Preview a Dataset

Find the dataset you want to preview, click **…** to the right of the dataset, and then click **Preview**.

The preview displays the dataset contents so you can review the entries before using the dataset in an evaluation.

### Download a Dataset

Find the dataset you want to download, click **…** to the right of the dataset, and then click **Download**.

### Delete a Dataset

Find the dataset you want to delete, click **…** to the right of the dataset, and then click **Delete** to open the **Delete dataset** window.

To confirm deletion, enter the name of the dataset, and then click **Delete dataset**.