For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Evaluations help you compare candidate quality, latency, and token usage across models, agents, and Inference Router configurations.

Use these best practices to keep evaluation configurations consistent, define focused scoring criteria, and write candidate system prompts that produce measurable results.

For guidance on choosing representative test inputs, edge cases, and ground truth responses, see Evaluation Dataset Design Best Practices.

DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.

Use Presets for Repeatable Evaluations

Presets let you save and reuse evaluation configurations instead of recreating the same setup for each run.

Use presets when you want to:

• Reuse the same evaluation configuration across multiple runs.
• Compare candidates using the same judge model and metrics.
• Save a baseline before testing configuration changes.
• Share a standard evaluation configuration with your team.

When you load a preset, review its configuration before running the evaluation. Update only the settings you intentionally want to change.

Keeping the rest of the configuration consistent makes differences between runs easier to interpret.

Change One Evaluation Variable at a Time

When comparing evaluation runs, change one major configuration variable at a time when possible.

For example, keep the judge model, metrics, thresholds, dataset, and candidate settings consistent when comparing different candidate models.

You can then make separate comparisons for changes such as:

• Candidate model
• System prompt
• Temperature or maximum tokens
• Agent configuration
• Inference Router configuration
• Judge model
• Evaluation metrics

Changing several variables at once can make it difficult to determine which change caused a difference in evaluation results.

For guidance on keeping test inputs consistent across comparisons, see Evaluation Dataset Design Best Practices.

Use a Consistent Judge Model

The judge model scores candidate responses using the evaluation metrics you select.

When comparing evaluation runs, use the same judge model whenever possible. Changing the judge model can change how responses are interpreted and scored, making results from different runs harder to compare directly.

Choose a judge model based on the complexity of the behavior you are evaluating and the cost you want to incur.

For less complex evaluation criteria, a lower-cost model is often enough. For evaluations that require nuanced interpretation, long responses, complex instructions, or subjective criteria, use a more capable model.

Create Focused Custom Metrics

Custom metrics let you define evaluation criteria that are specific to your application, policies, or expected behavior.

Use custom metrics when the built-in metrics do not fully capture what you want to evaluate.

For example, you can create custom metrics that evaluate:

• Whether a response follows an internal style guide.
• Whether a response includes required product details.
• Whether a response follows a support policy.
• Whether a response uses a required format or structure.
• Whether a response correctly handles a domain-specific behavior.

Custom metrics are available for single-turn evaluations.

Evaluate One Behavior per Metric

Keep each custom metric focused on one behavior or outcome.

A metric that evaluates one behavior is easier for the judge model to score consistently and makes failures easier to interpret.

The preferred example identifies the behavior being evaluated and describes what should affect the score. The avoided example does not define measurable scoring criteria.

Write Explicit Scoring Prompts

A custom metric scoring prompt should tell the judge model:

• What behavior to evaluate.
• What a successful response looks like.
• What should increase the score.
• What should lower the score.
• Whether the judge should use ground truth.

For example:

Evaluate whether the response follows the required support policy.

Give a high score when the response follows the required escalation process and does not make unsupported guarantees.

Lower the score when required escalation steps are missing or when the response contradicts the policy.

If the metric depends on a known answer, policy, value, or required output, configure the metric to require ground truth.

If a metric requires ground truth and the selected dataset does not contain ground truth values, DigitalOcean skips that metric during the evaluation.

Custom Metric Prompt Template

Use the following structure when writing a custom metric scoring prompt:

Evaluate whether the response [specific behavior to evaluate].

Score the response based on how completely it satisfies the criterion.

Increase the score when:

• [Condition that should improve the score.]
• [Condition that should improve the score.]

Lower the score when:

• [Condition that should reduce the score.]
• [Condition that should reduce the score.]

Use the ground truth field when the response must match a required answer, value, policy, or format.

Choose a Meaningful Star Metric

The star metric represents the primary success criterion for an evaluation.

Choose the metric that most directly reflects the behavior you are trying to improve or protect.

For example:

• Use Correctness when factual accuracy is the primary requirement.
• Use Ground Truth Faithfulness when responses should match known reference answers.
• Use User goal completion when evaluating whether a multi-turn conversation successfully accomplishes the user’s objective.
• Use a custom metric when your primary success criterion is specific to your application or policy.

Avoid choosing a star metric only because it produces the highest score. The metric should represent the outcome that matters most for the use case being evaluated.

Set Pass Thresholds Intentionally

Set the star metric pass threshold based on what you consider an acceptable result for the use case.

Review results from multiple representative evaluation runs before finalizing a threshold.

For metrics where higher scores indicate better performance, entries pass when their score is greater than or equal to the threshold.

For risk-oriented metrics where lower scores are better, entries pass when their score is less than or equal to the threshold.

When comparing runs over time, keep the threshold consistent unless you intentionally change the acceptance criteria.

Write System Prompts That Match the Evaluation Goal

Candidate system prompts define how the candidate should interpret and respond to evaluation inputs.

Write system prompts that specifically describe the behavior you expect from the candidate. Avoid adding instructions that are unrelated to the behavior being evaluated because they can make results harder to interpret.

A useful system prompt can define:

• The candidate’s role
• The expected behavior or objective
• Relevant constraints
• Tone or response style
• How the candidate should handle missing or uncertain information
• Required output formats

When comparing candidate models, use the same system prompt when possible so differences in results reflect model behavior rather than different instructions.