For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Dataset evaluations test a model, Inference Router, or supported agent against a fixed dataset of test inputs. Use dataset evaluations for repeatable testing and to compare results across the same set of prompts.

If you want to evaluate an agent through generated, multi-turn conversations instead, see Create a Simulation.

DigitalOcean Evaluations are advisory tools. We recommend manually reviewing evaluated model outputs before deploying models to production.

Create a Dataset Evaluation

To create a dataset evaluation, go to the DigitalOcean Control Panel, in the left menu, click DATA & LEARNING, click Evaluations, and then in the top-right corner, click New Evaluation to open the Create Evaluation page.

In the How do you want to evaluate? section, select Dataset evaluation, and then click Set up dataset evaluation.

If needed, click Change to select a different evaluation method before running the evaluation.

In the Presets section, choose one of the following:

• Use an existing evaluation preset to populate the evaluation configuration from a saved preset.
• Configure without a preset to configure the evaluation manually.

Use an Existing Evaluation Preset

To use an existing preset, click the Select a preset dropdown menu, and then select a saved preset.

Evaluation presets can automatically populate the dataset, judge model, evaluation metrics, and star metric.

After selecting a preset, the Load preset? window displays a summary of the preset configuration.

Then, click Load preset to use the preset. To select another preset or configure the evaluation manually instead, click x to close the window.

After loading the preset, you can edit its configuration before running the evaluation, including the evaluation type, judge model, metrics, star metric, and evaluation name.

Configure Without a Preset

To configure the evaluation manually, click Configure without a preset, then configure the evaluation type, candidate, dataset, judge model, evaluation metrics, star metric, and optional settings.

Choose Evaluation Type

In the Set up evaluation section, the evaluation type is Single-turn by default.

To change the evaluation type, expand the section, click the Evaluation type dropdown menu, and select one of the following:

• Single-turn: Evaluates one prompt and one candidate response for each dataset row. Use single-turn evaluations for independent requests that do not require conversational context.
• Multi-turn: Evaluates multiple prompt-and-response exchanges contained in each dataset row. Because the responses being evaluated are already included in the dataset, multi-turn evaluations do not require you to select a candidate model.

Configure Your Candidate

For single-turn evaluations, select the candidate that generates responses for the prompts in your evaluation dataset. The judge model evaluates the candidate’s responses using the metrics you configure.

To configure a candidate, expand the Configure your candidate section, in the Select a candidate to evaluate sub-section, click the Select a candidate type dropdown menu, and then select the type of candidate you want to evaluate:

• Serverless Inference: Evaluates a foundation model using Serverless Inference without requiring you to manage inference infrastructure.
• Third Party: Evaluates a supported model from a third-party provider using a configured provider key.
• Dedicated Inference: Evaluates a model running on an existing Dedicated Inference deployment.
• Model Router: Evaluates an existing Inference Router configuration that routes requests across models based on its configured routing rules.

If you need to deploy a dedicated model first, click Deploy a model to open the Inference Deployment creation page.

For supported candidates, click Show advanced settings to configure the following model parameters:

• Max Tokens: Sets the maximum number of tokens the model can generate in a response. The default is 2048.
• Temperature: Controls randomness in the model’s responses. Lower values produce more predictable responses, while higher values produce more varied responses. The default is 1.

In the System prompt field, enter the system prompt the candidate uses when interpreting prompts from the evaluation dataset. For guidance on writing system prompts, see Evaluation Best Practices.

Choose a Dataset

Then, select the dataset containing the inputs you want to evaluate. For information about creating and formatting datasets, see How to Create, Upload, and Manage an Evaluation Dataset.

Evaluation datasets must use CSV or JSONL format, contain up to 1,000 rows, and be up to 1 GB.

To select a dataset, expand the Choose Dataset section, then either click the Select a dataset dropdown menu and select an existing dataset, or click Add dataset to open the Select dataset window.

To upload a dataset, in the Choose file section, drag and drop the dataset file or click Upload to select a local file, and then click Add.

After we process the file, its status is either Ready, which means the dataset is valid and ready to use, or Error, which means the dataset couldn’t be processed.

If the status is Error, review the displayed error message, correct the dataset, and then upload it again.

The Dataset Preview section displays a preview of the uploaded dataset. For multi-turn datasets, the preview includes each conversation and its number of turns. If needed, click Show full preview to view the complete dataset preview.

Uploaded datasets are automatically saved so you can reuse them in future evaluations.

If you need to replace an incorrectly formatted file, click x next to the file, correct the dataset, and upload it again.

Configure Your Judge

The judge model evaluates the candidate output and assigns scores for the metrics you select. For information about available models, see Available Models.

To configure the judge, expand the Configure judge & metrics section, in the Configure your LLM judge sub-section, click the Select a model to use as judge dropdown menu, and then select the model you want to use. The dropdown also displays the model’s input and output token pricing.

DigitalOcean sends the dataset inputs, candidate outputs, and ground truth values, when available, to the selected model provider for scoring and score rationale.

Score rationale is automatically generated and may contain errors or omissions. Review and verify evaluation results before relying on them.

Choose Metrics

Evaluation metrics define the criteria the judge model uses to score the evaluation. The available metrics depend on whether you selected a single-turn or multi-turn evaluation.

You can use built-in metrics, custom metrics, or a combination of both.

To select metrics, expand the Configure judge & metrics section, in the Evaluation metrics sub-section, select the metrics you want to use.

You can click Select all metrics to select all available metrics for the evaluation type.

Choose Your Star Metric

The star metric is the primary metric you use to determine whether the evaluation meets your success criteria. Choose the metric that most directly represents the behavior or outcome you are testing.

To configure the star metric, expand the Configure judge & metrics section, in the Star Metric sub-section, click Change to open the Select star metric window. A star metric is selected from the metrics configured for the evaluation.

Then, click the star icon next to the metric you want to use, and then click Update star metric.

In the Star metric pass threshold field, enter a threshold from 0 through 100.

For metrics where higher scores are better, a row passes when its score is greater than or equal to the threshold.

For metrics where lower scores are better, a row passes when its score is less than or equal to the threshold.

Review the results of several evaluation runs and adjust the threshold based on the scores you consider acceptable for your use case.

Name the Evaluation

To name the evaluation, expand the Optional settings section, in the Name your evaluation run (optional) field, enter a name that identifies the run.

Using descriptive names can make it easier to distinguish runs when comparing evaluations later.

Save the Configuration as a Preset

You can save the evaluation configuration as a preset for reuse.

To save it, expand the Optional settings section, and then select Save this configuration as a preset.

In the Preset name field, enter a name for the preset.

In the Choose what to save section, either click Select all sections or select the individual parts of the configuration you want the preset to retain.

Run the Evaluation

After configuring the evaluation, click Run Evaluation.

After the evaluation finishes, click its name to open the results page.

Our data privacy policy describes zero data retention for this flow. Your data is not stored outside of DigitalOcean or used to train models.

Review Evaluation Results

After an evaluation finishes, its results page displays:

• Overall Score: The overall score for the evaluation run.
• Dataset: The dataset used for the evaluation.
• Star Metric: The primary metric used to determine evaluation success.
• Pass Threshold: The threshold configured for the star metric.
• Candidate Tokens: The candidate model’s input, output, and total token usage.
• Judge Tokens: The judge model’s input, output, and total token usage.
• Candidate Latency: The candidate model’s average, P50, P90, P95, minimum, and maximum latency.

The results chart displays the percentage of dataset entries that passed or failed each selected metric.

The Evaluation results table displays each dataset entry’s input, candidate output, metric, score, and judge rationale.

Click Show full output for an entry to view its complete candidate response.

To download the evaluation results, in the top-right corner, click Download results.

For additional actions you can take on completed evaluations, see Manage Evaluations.

For guidance on choosing representative prompts, covering edge cases, and writing ground truth responses, see Evaluation Dataset Design Best Practices.

Run an Evaluation Using the API

You can also create and retrieve dataset evaluations using the DigitalOcean API.

First, upload an evaluation dataset:

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation/datasets/file_upload_presigned_urls" \
  -d '{
    "files": [{
      "file_name": "dataset.jsonl",
      "file_size": 2048
    }]
  }'

Then, create an evaluation run. You need the candidate and judge model IDs.

Use the List Available Models endpoint to retrieve supported models and Inference Routers.

After you have the candidate and judge model IDs, create the evaluation run:

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs" \
  -d '{
    "name": "my-evaluation-run",
    "candidate_model_uuid": "123e4567-e89b-12d3-a456-426614174000",
    "judge_model_uuid": "223e4567-e89b-12d3-a456-426614174001",
    "dataset_uuid": "323e4567-e89b-12d3-a456-426614174002",
    "metric_uuids": [
      "423e4567-e89b-12d3-a456-426614174003"
    ]
  }'

After the run finishes, retrieve the evaluation metrics:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_metrics"

You can also download a JSON file containing additional evaluation result details:

curl -X GET \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/model_evaluation_runs/123e4567-e89b-12d3-a456-426614174000/results/download_url"