For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Usage

client.genai.create_custom_evaluation_metric(
    body={
        "config": {...},
        "description": "\"Scores adherence to our support macros\"",
        "metric_name": "\"My domain tone metric\"",
    },
)

Description

To create a custom LLM-as-judge metric for model evaluation, send a POST request to /v2/gen-ai/custom_evaluation_metrics.

Parameters

config object optional

Configuration for a custom model-evaluation metric scored by an LLM judge.
Prompt and model response are always included in the judge context.

Show child properties

created_at string optional read-only

Example: 2023-01-01T00:00:00Z

Timestamp when the custom metric was created. Server-assigned; ignored on create/update requests.

deleted_at string optional read-only

Example: 2023-01-01T00:00:00Z

When set, the custom metric has been deleted and is no longer available for use in evaluations. Server-assigned; ignored on create/update requests.

requires_ground_truth boolean optional

Example: True

When true, each row must provide ground truth and it is included in the judge context.
When false, ground truth is not required and is not sent to the judge.

scoring_prompt string optional

Example: example string

Instructions for the judge model (multi-line).

updated_at string optional read-only

Example: 2023-01-01T00:00:00Z

Timestamp when the custom metric was last updated. Server-assigned; ignored on create/update requests.

description string optional

Example: "Scores adherence to our support macros"

metric_name string optional

Example: "My domain tone metric"

Request Sample

import os
from pydo import Client

client = Client(token=os.environ.get("DIGITALOCEAN_TOKEN"))

req = {
  "config": {
    "requires_ground_truth": True,
    "scoring_prompt": "example string"
  },
  "description": "\"Scores adherence to our support macros\"",
  "metric_name": "\"My domain tone metric\""
}

resp = client.genai.create_custom_evaluation_metric(body=req)

Response Example

{
  "metric": {
    "associated_presets": [],
    "category": "METRIC_CATEGORY_UNSPECIFIED",
    "custom_eval_config": {
      "created_at": "2023-01-01T00:00:00Z",
      "deleted_at": "2023-01-01T00:00:00Z",
      "requires_ground_truth": true,
      "scoring_prompt": "example string",
      "updated_at": "2023-01-01T00:00:00Z"
    },
    "description": "example string",
    "evaluation_scope": "EVALUATION_SCOPE_UNSPECIFIED",
    "inverted": true,
    "is_metric_goal": true,
    "metric_name": "example name",
    "metric_rank": 123,
    "metric_type": "METRIC_TYPE_UNSPECIFIED",
    "metric_uuid": "123e4567-e89b-12d3-a456-426614174000",
    "metric_value_type": "METRIC_VALUE_TYPE_UNSPECIFIED",
    "range_max": 123,
    "range_min": 123,
    "source": "EVALUATION_METRIC_SOURCE_UNSPECIFIED"
  }
}

More Information

See /v2/gen-ai/custom_evaluation_metrics in the API reference for additional detail on responses, headers, parameters, and more.