For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Evaluation datasets contain the inputs used for dataset evaluations. The required dataset format depends on whether you are running a single-turn or multi-turn evaluation.

For guidance on choosing representative inputs, covering edge cases, and writing ground truth responses, see Evaluation Dataset Design Best Practices.

To create an evaluation dataset, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, click the Datasets tab, and then in the top-right corner, click Upload dataset to open the Select dataset window.

Choose Dataset Format

Choose the dataset paradigm based on whether each dataset entry contains a single input for a candidate to answer or an existing multi-turn conversation you want to evaluate.

In the Paradigm section, click the dropdown menu, and then choose one of the following:

• Single-turn (default): Use when each dataset entry contains an independent query that you want a candidate to answer. Single-turn evaluations generate a candidate response for each query and score the response using the metrics you select.
• Multi-turn: Use when each dataset entry contains an existing conversation with multiple user and assistant messages. Multi-turn evaluations score the conversation already contained in the dataset and do not require a candidate model.

You can upload a dataset of either paradigm as a CSV or JSONL file. In a CSV file, the first row contains the column names. In a JSONL file, each line is a JSON object that uses the column names as its fields.

Format a Single-Turn Dataset

Single-turn evaluation datasets must include an input column. The input column contains the prompt you want the candidate to answer, and every row must have a non-empty value. DigitalOcean ignores any other columns.

For example, a single-turn dataset could contain inputs like this:

input
"What makes DigitalOcean different from other cloud providers?"
"Explain the benefits of using DigitalOcean for startups."

Some single-turn metrics require ground truth values. Ground truth values are expected responses that the judge model uses to compare against the candidate’s response.

For a ground truth dataset, include a ground_truth column that contains the expected answer for each input.

For example, a single-turn dataset with ground truth responses could contain:

input,ground_truth
"What is a Droplet?","A Droplet is a virtual machine that runs on DigitalOcean's cloud infrastructure."
"How do I add a member to my Team account?","To add a member to your DigitalOcean Team account, go to the DigitalOcean Control Panel, open your team settings, click the Members tab, and then click Invite Team Member."
"What is the pricing for Droplets?","Droplets have flexible pricing based on instance type."

The same dataset in JSONL format contains one object per line:

{"input":"What is a Droplet?","ground_truth":"A Droplet is a virtual machine that runs on DigitalOcean's cloud infrastructure."}
{"input":"What is the pricing for Droplets?","ground_truth":"Droplets have flexible pricing based on instance type."}

Use ground truth responses when you want to evaluate whether a candidate’s answer matches a known correct answer. For example, ground truth datasets are useful for support questions, policy answers, factual product documentation, and other cases where the expected response is known.

For guidance on writing effective ground truth responses, see Evaluation Dataset Design Best Practices.

Format a Multi-Turn Dataset

Multi-turn evaluation datasets must include id and messages columns.

The id column identifies the conversation. Each conversation must have a non-empty id that is unique in the dataset.

The messages column contains the conversation to evaluate as a non-empty JSON array of messages. Each message must have a non-empty role, such as user or assistant, and a content field. The content value can be an empty string, but the field must be present. DigitalOcean ignores other columns and keeps additional message fields, such as tool_calls.

For example, a multi-turn dataset could contain a conversation like this:

id,messages
1,"[{""role"":""user"",""content"":""I need to update my mailing address.""},{""role"":""assistant"",""content"":""I can help with that. What is your new mailing address?""},{""role"":""user"",""content"":""123 Example Street.""},{""role"":""assistant"",""content"":""Your mailing address has been updated to 123 Example Street.""}]"

The same conversation in JSONL format is one line:

{"id":"1","messages":[{"role":"user","content":"I need to update my mailing address."},{"role":"assistant","content":"I can help with that. What is your new mailing address?"},{"role":"user","content":"123 Example Street."},{"role":"assistant","content":"Your mailing address has been updated to 123 Example Street."}]}

Multi-turn evaluations score the conversation already contained in the dataset, so you don’t need to select a candidate model. Multi-turn datasets don’t use ground truth values.

Review Dataset

Before uploading the dataset, review its format and contents to make sure DigitalOcean can process it correctly.

When reviewing your evaluation dataset, ensure the following requirements are satisfied:

• Use a CSV or JSONL file of up to 1 GB, with up to 1,000 rows or conversations.
• For single-turn evaluations, include an input column.
• Include a ground_truth column if you plan to use a single-turn metric that requires ground truth.
• For multi-turn evaluations, include id and messages columns.
• Use UTF-8 encoding.
• Review exported spreadsheet files for invalid or non-UTF-8 characters before uploading them.
• Remove empty rows when possible. Empty rows are skipped during scoring, but any tokens used to process them still count toward evaluation cost.
• Make sure the file is readable and correctly formatted. If required columns are missing, or your file is malformed or unreadable, the dataset upload fails.

We recommend using 50 to 100 entries for faster feedback and lower evaluation costs.

After reviewing your dataset, click Add.

Manage Datasets

After uploading a dataset, you can preview its contents, download the original dataset, or delete it from your team.

To manage your datasets, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, and then click the Datasets tab.

Preview a Dataset

Find the dataset you want to preview, click … to the right of the dataset, and then click Preview.

The preview displays the dataset contents so you can review the entries before using the dataset in an evaluation.

Download a Dataset

Find the dataset you want to download, click … to the right of the dataset, and then click Download.

Delete a Dataset

Find the dataset you want to delete, click … to the right of the dataset, and then click Delete to open the Delete dataset window.

To confirm deletion, enter the name of the dataset, and then click Delete dataset.