How to Create an Evaluation Datasetpublic

Agent evaluations use datasets of input prompts to test the response of your agent to each prompt in the set. For example, if you want to test the factual accuracy of your marketing agent, you could add a prompt to your dataset that asks, “What is the difference between a Droplet and virtual machine?” Developing a evaluation datasets with specific goals is important to effectively measuring and refining your agent.

This guide provides guidance on how to develop evaluation datasets for specific agent goals and offers advice on how to improve your agent.

Format Test Files for Evaluations

Agent evaulation datasets are .csv files that contain a list of prompts for the agent to respond to during evaluation. Each file should contain a column named query, like this:

ID,query
1,"What makes DigitalOcean different from other cloud providers?"
2,"Explain the benefits of using DigitalOcean for startups."
...

For a Ground-of-Truth dataset, the file should have an additional column named expected_response that contains the expected responses to each query, like this:

ID,query,expected_response
1,"What is a Droplet?","A Droplet is a virtual machine that runs on DigitalOcean’s cloud infrastructure."
2,"How do I add a member to my Team account?", "To add a member to your DigitalOcean Team account, you go to the DigitalOcean Control Panel, then to your Team account’s settings, then click on the Members tab. Under the Members tab, click Invite Team Member."
3,"What is the pricing for Droplet?", "Droplets have flexible pricing based on instance type."

If your dataset contains empty rows, these rows are skipped during scoring, but any tokens used to process these rows still count toward your evaluation cost. If required columns are missing, or your file is malformed or unreadable, the dataset upload fails.

Datasets only support UTF-8 encoding. Encoding errors may occur when exporting your data from a spreadsheet application, such as Microsoft Excel or Google Sheets. We recommend reviewing your data for non-UTF-8 characters before uploading it to the GenAI Platform.

We recommend using 50 to 100 queries for faster feedback and lower evaluation costs. You can upload datasets with more than 500 queries but only the first 500 is used for evaluation.

Develop Your Dataset

Developing your dataset depends on the goals you have for your agent. For example, if you want to test the factual accuracy of your agent, you can create a dataset that contains queries that ask for specific information about your product or service. If you want to test your agent for safety and harmful content, you can create a dataset that contains queries that ask for potentially harmful or sensitive information.

While you may have one specific goal in mind for your agent, we recommend developing multiple datasets that test different aspects of your agent’s performance, such as its:

• Factual accuracy, the correctness of the information provided in the agent’s responses.
• Safety and harmful content, the agent’s ability to avoid generating harmful or sensitive content.
• Instruction following, the agent’s ability to follow instructions and provide relevant responses.
• Robustness and reliability, the agent’s ability to handle unexpected or ambiguous queries without generating incorrect or harmful responses.

This approach allows you to evaluate your agent’s performance across a range of scenarios and helps you identify areas for improvement.

If you’re testing for specific qualities related to your business, it’s a good idea to develop datasets that reflect the specific language and terminology used in your industry. This helps ensure that your agent is able to understand and respond to queries in a way that is relevant and accurate for your target audience. If you have access to historical data, such as customer support tickets or chat logs, you can use this data to develop your datasets.

You can explore the following open-source datasets to help you develop your own to test agents: