For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Evaluations help you measure and compare model, Inference Router, and agent performance using repeatable evaluation workflows.

Dataset Evaluations

Dataset evaluations test a model, Inference Router, or supported agent against a fixed dataset of test inputs.

You can use dataset evaluations for:

• Single-turn testing: Send independent prompts to a candidate and score its generated responses.
• Multi-turn testing: Score existing conversations contained in a dataset without selecting a candidate model.
• Ground truth evaluation: Compare candidate responses against expected answers included in the dataset.
• Custom evaluation criteria: Define custom metrics for single-turn evaluations when built-in metrics do not cover your use case.
• Evaluation comparison: Compare two dataset evaluations across overall scores, star metrics, pass thresholds, configurations, metric pass rates, token usage, latency, Inference Router task results when applicable, and per-entry results when both evaluations use the same dataset.

Dataset evaluation results include overall and per-metric scores, judge rationale, candidate and judge token usage, and candidate latency. You can also download evaluation results, rename evaluation runs, and delete evaluations from the Evaluations page.

Simulation public

Simulations test agent behavior through generated, multi-turn conversations.

You can evaluate:

• An existing managed agent with its configured instructions, tools, and settings.
• An agent that the simulation creates from a Serverless Inference model, third-party model, Dedicated Inference deployment, or Inference Router.

Simulations use scenario sets that define situations for a user simulator and agent to act out. You can generate scenario sets from a goal description, upload them as JSONL files, or import ready-made sets from the Common Scenario & Goal Library. You can configure the user simulator model, judge model, evaluation metrics, turn limit, and number of episodes for each scenario.

You can create and manage scenario sets and simulation runs in the Control Panel or using the DigitalOcean API.

Simulation results include scenario and episode status, duration, token usage, conversation trajectories, and evaluation metric results.

Evaluation Metrics

Built-in metrics let you evaluate response quality, safety, conversation behavior, tool use, and goal completion.

Single-turn evaluation metrics include criteria such as correctness, completeness, ground truth faithfulness, answer relevancy, coherence, clarity, and harmfulness.

Multi-turn and simulation metrics include User goal completion, Knowledge Retention, Conversation Completeness, Turn Relevancy, Tool Use Accuracy, and Guardrail validation.

For single-turn dataset evaluations, you can also create custom metrics with your own scoring prompts and optional ground truth requirements. Simulation runs that you create using the API can also use custom metrics.

Reusable Evaluation Configurations

Evaluation presets let you save and reuse evaluation configurations across runs.

Depending on the configuration, presets can retain settings such as the dataset, judge model, evaluation metrics, and star metric. Reusing configurations helps keep evaluation settings consistent when comparing changes between runs.