For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Evaluation datasets determine the inputs and conversations your evaluations test. A well-designed dataset represents the behaviors, user requests, and edge cases your candidate is expected to handle so evaluation results reflect meaningful performance differences.

Use these best practices to define dataset coverage, write representative test inputs, create useful ground truth responses, and keep test data consistent across evaluation runs.

For information on required columns, file formatting, and other dataset requirements, see How to Create, Upload, and Manage an Evaluation Dataset.

Define Dataset Scope

Decide which behaviors and types of inputs the dataset should cover before writing individual entries. Keeping the scope focused makes it easier to understand which kinds of requests a candidate handles successfully and where failures occur.

For example, you can create datasets that cover:

• Factual questions: Questions with verifiable answers about your product, service, or domain.
• Safety-sensitive requests: Inputs that test whether the candidate handles unsafe, sensitive, or inappropriate requests correctly.
• Instruction-following requests: Inputs that require a specific format, behavior, or set of instructions.
• Ambiguous or unexpected requests: Inputs that test how the candidate handles incomplete information, incorrect assumptions, or unusual phrasing.
• Context-dependent requests: Inputs that require the candidate to use retrieved context, knowledge base content, or other supplied information.

Keep unrelated types of inputs in separate datasets when you want to measure their performance independently.

Write Focused Test Inputs

Write each dataset entry to test one behavior or scenario. Focused inputs make failures easier to diagnose because each entry has a clear purpose.

If you need to test multiple behaviors, split them into separate dataset entries instead of combining them into one query.

Represent Real User Inputs

Write test inputs that reflect how users actually interact with your application. Representative inputs help ensure evaluation results reflect the requests the candidate is likely to receive in production.

Build representative coverage by:

• Including common user phrasing.
• Using relevant product names and terminology.
• Including abbreviations users commonly use.
• Incorporating patterns from support tickets, documentation searches, sales calls, or application logs.
• Including different ways users may express the same intent.

For example:

Both inputs test the same intent using different language.

Include Normal and Edge Cases

Include both common requests and difficult inputs. Testing only ideal inputs can hide failures that occur when users provide incomplete, ambiguous, or unexpected requests.

Include entries that cover:

• Common user requests.
• Ambiguous or incomplete questions.
• Questions containing incorrect assumptions.
• Requests the candidate should refuse or redirect.
• Requests that require retrieved context or a knowledge base.
• Questions where the candidate should indicate that it does not know the answer.

Including a range of inputs helps you determine whether candidate behavior remains consistent outside the most typical cases.

Write Representative Ground Truth Responses

Use a ground_truth value when an input has a known correct answer.

Ground truth responses should contain the information required for a correct response without unnecessarily prescribing exact wording. This gives the judge model enough information to determine whether the candidate returned the expected facts or outcome.

Destroyed Droplets cannot be recovered. Users should restore from a backup or snapshot if one is available.

No, you cannot recover it.

The preferred response identifies the required facts while allowing the candidate to express them differently.

For information on formatting datasets with ground truth responses, see How to Create, Upload, and Manage an Evaluation Dataset.

Separate Datasets by Test Objective

Avoid combining unrelated types of test inputs into one large dataset. Separate datasets make failures easier to diagnose and let you rerun only the tests relevant to a particular change.

For example, you might maintain separate datasets for:

• Billing questions
• Support policy requests
• Safety and refusal cases
• Knowledge base retrieval
• Instruction-following requests
• Regression tests for prompt or model changes

Keep Test Data Consistent Across Comparisons

When comparing candidate configurations, use the same dataset whenever possible.

Keeping the test inputs fixed helps isolate the effect of changes such as:

• Switching candidate models
• Updating a system prompt
• Changing model parameters
• Modifying an agent configuration
• Changing an Inference Router configuration

If you change both the candidate configuration and the dataset, differences between evaluation runs may result from either change.

Use Open-Source Datasets as References

Open-source datasets can provide examples of established test patterns and help you identify additional types of inputs to include.

Use them as references when developing datasets that reflect your own application, users, and expected traffic.

Open-source datasets may not represent your production traffic or application requirements. Adapt their testing patterns to your use case instead of using them as a substitute for representative application data.