For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Inference Router is available in public preview and enabled for all users. There is no additional cost to use Inference Router during public preview. You are billed for the models that serve each request. For per-model rates, see Inference pricing. You can contact support for questions or assistance.

Inference Router lets you describe in natural language which models should handle different requests. These models can use serverless inference or dedicated inference endpoints. The router analyzes each prompt and routes the request to the most appropriate model based on your routing instructions, such as cost or latency preferences.

Inference Router also makes cache-aware routing decisions to maximize prompt cache reuse. For Anthropic models, it automatically enables prompt caching for eligible requests.

You can select pre-configured routers or build your own router using the Control Panel or using the API.

How Inference Routing Works

A router is a collection of logically grouped tasks and fallback models. A task is a combination of a name, a description, and a model pool with selection policy. The task description and name define which model the router routes to based on the incoming query to the router. Each task has a pool of models and a policy for selecting among them, such as prioritizing the lowest cost or the lowest latency model.

You can choose from preset tasks or define your own custom tasks for specific use cases. Preset tasks have a pre-configured combination of eligible models across different providers and capability tiers in model pools, and a selection policy based on cost and latency that DigitalOcean has benchmarked and optimized. Model pools allow you to have more than one model per task. Default preset tasks have an Optimal selection policy, which is based on benchmarking done by DigitalOcean to determine the best model.

The router reads each incoming request, evaluates the request against the tasks configured in the router, and if it matches, it picks the right model from the model pool, while applying the selection policy to determine the best-fit model.

For custom routers, you can configure the router to skip the automated selection policy and specify a manual order of models to be tried sequentially. This manual override allows for precise control, ensuring that models are attempted in the exact sequence you specified.

To ensure system resiliency, each router incorporates a fallback mechanism. If, for some reason, the selected model is not available, down, or rate limited, the router picks the next best model. For example, if cost is the selection policy, it cycles through and picks the cheapest model, before the fallback models. These fallback models are also tried in a prioritized order, providing an instant and intelligent failover process that maintains session continuity without manual intervention.

Create Inference Router Using the Control Panel

To create an Inference Router, in the DigitalOcean Control Panel, in the left menu, click INFERENCE, and then select Inference Router.

In the Use Preset Routers section, click See Preset Routers to see the available preset routers and the tasks included with each one in the My Routers tab. We provide pre-configured routers for general-purpose language tasks, software engineering, writing and content editing, and knowledge base and document intelligence. These provide one-click solutions for common agentic patterns and use templated routing policies that recommend models for specific use cases.

Alternatively, in the top-right, click Create Router to create a custom router.

Build Custom Router

You can build your own router that defines specific use cases and policies in the Create a Router page.

Name Your Router

In the Choose a unique name field, specify a name for the router. Names must be in lowercase. They can be up to 255 characters long and may contain dashes.

Describe Your Router

In the Description field, add a description of what the router is for. The description is used as a routing prompt so that the router can choose the correct task for each request.

Define Tasks

Define the tasks to include in your router. In the Router Tasks section, you can add custom tasks, pre-configured tasks, or a combination of both to the same router.

Using custom tasks, you can customize the router selection policy and model combinations from scratch. In the Router Tasks section, click the Add Task dropdown menu, and then either select the Add Preset Task or Create Custom Task.

If you select Add Preset Task, the Add Preset Tasks side panel opens. These tasks are based on a hybrid evaluation approach that combines public benchmark signals from Arena and Artificial Analysis with in-house benchmarking on curated, task-specific datasets, then validates the results through human evaluation by DigitalOcean’s data science team.

In the Selected preset tasks section, select the tasks you want to include. Alternatively, in the top-right, click the All dropdown menu to filter the available preset tasks, and then select the tasks you want to include.

You can also click the + next to the task name to view the task description, model pool, and selection policy.

Then, click Save. Afterwards, you can see a summary of your preset task in the Router Tasks section. You can optionally modify a preset task by clicking Edit. In the Edit Task pane, modify the task name, description, selection policy, and selected models. You can select up to three models in the model pool and choose one of the following as the selection policy:

• Optimal: Based on hybrid evaluation approach combining public benchmark signals with DigitalOcean’s in-house benchmarking across curated task-specific datasets
• Cost Efficiency: Token costs for the models in the pool
• Speed Optimization: Time To First Token (TTFT) of the models on DigitalOcean

Then, click Save.

If you select Create Custom Task, the Create Custom Task side panel opens, where you can define the task name, routing description, models, and prioritization policy for workloads not covered by preset tasks.

Specify the following values for the custom task:

• Name and description for the task. Specify a name and description in the Choose a unique name and Description fields. The name and description you provide are important to match the intent for the router. See below for name and description recommendations.

• Router prioritization policy. Select a policy in the Select router prioritization policy field. You can choose one of the following:

  • Cost Efficiency: Token costs for the models in the pool
  • Speed Optimization: Time for first token on DigitalOcean based on Time To First Token (TTFT)
  • Manual Ranking: Order in which you added the models to the pool

• Models to use for the routing. Select models to use for routing in the Selected models section. Make sure to select the models in the order you want the router to use them. Model reordering is only available for Manual Ranking. You can select up to 3 models in the model pool. Models supported for Dedicated Inference are displayed only when the selection policy is Speed Optimization or Manual Ranking.

Then, click Save. Afterwards, you can see a summary of your custom task in the Router Tasks section.

To delete a task, click Delete next to the task you want to remove.

Custom Task Best Practices

We recommend the following when creating custom tasks:

• Consistent naming: Align route names with their descriptions.

  Prefer	Avoid
  {"name": "quadratic_equation", "description": "solving quadratic equations"}	{"name": "math", "description": "handle solving quadratic equations"}

• Clear usage description: Make your route names and descriptions specific, unambiguous, and minimize overlap between routes. The Router has improved performance when it can clearly distinguish between different types of requests.

  Prefer	Avoid
  {"name": "math", "description": "solving, explaining math problems, concepts"}	{"name": "math", "description": "anything closely related to mathematics"}

• Noun-centric descriptors: Preference-based routers offer more stable and semantically rich signals for matching, thus providing improved performance.

Specify Fallback Models

Requests are routed to fallback models when the request does not match any of the configured tasks. In the Fallback Models section, click Add Fallback Models to open the Select Fallback Models panel. Make your selections from the model list. You can also reorder the models by clicking on the right of the selected model and dragging it. Then, click Save.

Create Router

Click Create Router to create the custom router. After the creation completes, the router is displayed in the My Routers tab.

Create Inference Router Using Automation

To create a custom router using the DigitalOcean API, send a POST request to /v2/gen-ai/models/routers. Define the following schema for the router and pass it in the cURL request:

curl -X POST "https://api.digitalocean.com/v2/gen-ai/models/routers" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-test-router",
    "description": "my-test-router",
    "policies": [
      {
        "task_slug": "translation",
         "selection_policy": {
          "prefer": "cheapest"
        }
      },
      {
        "custom_task": {
          "name": "summarization",
          "description": "Summarize documents"
        },
        "models": ["openai-gpt-5.2", "glm-5"],
        "selection_policy": {
          "prefer": "fastest"
        }
      }
    ],
    "fallback_models": [
     "openai-gpt-oss-120b"
    ]
}'

Use the Router

To use the router, send prompts as described in Chat Completions or Responses, using the router name prefixed with router: in the model field in the request body. Using a router is a drop-in replacement for any model call.

For multi-turn conversations, you can provide your own stable X-Model-Affinity session or task identifier across related requests. If you do not provide an explicit identifier, Inference Router can infer affinity from stable session context. For more information, see Use Model Affinity for Caching.

For example, the following requests use the same X-Model-Affinity session key to maintain model affinity across two turns in the same conversation:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $MODEL_ACCESS_KEY" \
--data '{
    "model": "router:test-router",
    "messages": [
      {
        "role": "user",
        "content": "Are there any syntax issues here? Code: \nPython \nprices_usd = {'\''laptop'\'': 1200, '\''mouse'\'': 25, '\''monitor'\'': 300, '\''cable'\'': 12} \nexchange_rate = 0.92 \n# The line below is the focus \nexpensive_items_eur = {k: v * exchange_rate for k, v in prices_usd.items() if v > 50} \nprint(expensive_items_eur)"
      }
    ],
    "stream": true
}'

You can get a model access key to use in your request, as described in Model Access Keys and set it as an environment variable by running the following command:

export MODEL_ACCESS_KEY="<your_model_key>"

You can see which model your request routed to in the response:

data:
{
  "choices": [
    {
      "delta": {
        "content": null,
        "reasoning_content": null,
        "refusal": null,
        "role": "assistant"
      },
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null
    }
  ],
  "created": 1775481301,
  "id": "",
  "model": "openai-gpt-oss-120b",
  "object": "chat.completion.chunk"
}

You can also view which task was ultimately selected in the response header x-model-router-selected-route:

x-model-router-selected-route: fallback

You can use the x-routing-max-switch-spend-pct request header to control cache-aware model switching.

Test Router Performance

You can test the router against a model or another router in the playground. In the My Routers tab, click the … menu for the router you want to test. In the Compare section, choose the model to compare the router with. The Playground tab opens in a comparison view with the selected model and the router. In the text box, enter your question and press Enter. Compare the results which show cost difference, end-to-end latency, and the model selected for the router along with the specific task that was matched for the query. You can also click on the information icon next to Inference Router to quickly review the router configuration.

Using a router adds approximately 200ms of latency overhead.

Cache-Aware Inference Routing

For common agentic requests, Inference Router can infer affinity if an explicit identifier is not available. It derives a stable session key from request context that remains unchanged across turns, including system and developer instructions, tool definitions, and the first user message. If that stable context changes, Inference Router treats the cache as cold and establishes a new model binding, which then begins accumulating its own cached context.

Inference Router uses cache-aware routing to preserve cached context across multi-turn and agentic sessions. Reusing cached context reduces repeated input processing, while switching models can invalidate the cache and increase cost and time to first token.

Inference Router provides two controls for managing cache-aware routing:

• X-Model-Affinity lets applications that already manage sessions or tasks explicitly associate related requests with the same model binding.
• X-Routing-Max-Switch-Spend-Pct controls how much additional input-token cost the router can incur by breaking an existing model binding and switching models.

You can use either control independently or use them together. By default, the maximum switching budget is 20% above what the session would have cost if it had remained on its existing model.

Use Model Affinity for Caching

Applications that already maintain session or task identifiers can use X-Model-Affinity to explicitly associate related requests with the same model.

The first request is routed according to the router’s configured task, model pool, and routing preferences. Requests with the same X-Model-Affinity value are then treated as part of the same unit of work, allowing Inference Router to preserve the session’s model binding and reuse its cached context.

Set your own session or task identifier, such as X-Model-Affinity: example-session, and reuse the same value for each related request.

Affinity identifiers should represent meaningful units of work, such as a coding session, research task, support conversation, or individual agent run. When the application starts a new task, use a new identifier so Inference Router can make a new routing decision.

For example, the following request sets example-session as the model affinity identifier:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "X-Model-Affinity: example-session" \
  -d '{
    "model": "router:test-router",
    "messages": [
      {
        "role": "user",
        "content": "Write a Python function that implements binary search on a sorted array"
      }
    ]
  }' | python3 -m json.tool

For subsequent requests in the same session, reuse the same affinity value:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "X-Model-Affinity: example-session" \
  -d '{
    "model": "router:test-router",
    "messages": [
      {
        "role": "user",
        "content": "Write a Python function that implements binary search on a sorted array"
      },
      {
        "role": "assistant",
        "content": "<previous-response>"
      },
      {
        "role": "user",
        "content": "Now explain its time complexity"
      }
    ]
  }' | python3 -m json.tool

Control Cache-Breaking Switches with a Routing Budget

Inference Router continues evaluating alternative models while a session has a warm cache. When the routing policy proposes switching models, Inference Router calculates the incremental cost of leaving the session’s existing cache.

It compares the cached input cost of staying on the current model with the uncached cost of rebuilding the context on the candidate model, then evaluates that cost against the session’s cumulative switching spend.

By default, Inference Router limits cumulative switching costs to 20% above what the session would have cost if it had stayed on the existing model.

Use the X-Routing-Max-Switch-Spend-Pct header to configure this maximum switching budget. For example, X-Routing-Max-Switch-Spend-Pct: 20.

Model selection and economics remain separate: Inference Router identifies its preferred model, while the routing budget determines whether switching to it is worth the additional input cost.

For example, the following request sets the maximum switching budget to 20%:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "X-Routing-Max-Switch-Spend-Pct: 20" \
  -d '{
    "model": "router:test-router",
    "messages": [
      {
        "role": "user",
        "content": "Summarize this in two sentences."
      }
    ]
  }' | python3 -m json.tool

You do not need to set X-Model-Affinity to use a routing budget. If you omit an explicit affinity identifier, Inference Router automatically derives affinity from stable session context and applies the switching budget to that model binding.

Use Model Affinity and a Routing Budget Together

You can use X-Model-Affinity and X-Routing-Max-Switch-Spend-Pct together when your application manages its own session or task identifiers and you also want to control the cost of breaking that model binding.

For example:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "X-Model-Affinity: example-session" \
  -H "X-Routing-Max-Switch-Spend-Pct: 20" \
  -d '{
    "model": "router:test-router",
    "messages": [
      {
        "role": "user",
        "content": "Help me debug this failing test."
      }
    ]
  }' | python3 -m json.tool

In this example, X-Model-Affinity associates the request with example-session, while X-Routing-Max-Switch-Spend-Pct allows cumulative switching costs up to 20% above the cost of keeping the session on its existing model.

Analyze Router Performance

In the Analyze tab, you can view aggregate metrics across all routers or for a specific router. You can view the following metrics:

• Total requests: Number of requests the router received.
• Total token usage: Total number of input and output tokens consumed across all models.
• Task match rate: Percentage of requests that matched a configured task.
• Fallback rate: Percentage of requests that did not match a configured task and were sent to a fallback model.
• Input tokens cached: Percentage of input tokens read from the provider prompt cache. The metric also shows the number of cached input tokens compared to total input tokens.
• Requests cached: Percentage of requests that used cached input tokens. The metric also shows the number of cached requests compared to total requests.
• Requests held: Percentage of requests the router kept on the current model rather than switching models.
• Requests switched: Percentage of requests the router switched to another model.
• Latency: Average time for requests to complete.

When you select a specific router, the Task distribution chart shows how requests are distributed across configured tasks and fallback traffic.

The Model distribution chart shows how requests are distributed across the models that handled them.

The Request volume chart shows the number of requests the router received over time, helping you identify traffic spikes and periods of lower activity.

The Router resolution latency chart shows how long the router takes to resolve each routing decision over time, helping you identify changes or spikes in routing latency.

The Input tokens cached and Requests cached charts show cache efficiency over time, helping you identify changes in token-level and request-level cache usage and assess the impact of prompt or cache configuration changes.

The Metrics by model table breaks down input tokens cached, requests cached, requests held, requests switched, and average latency for each model. Enable Show empty rows to include models that have no metric data for the selected time range.

The Metrics by task table breaks down requests cached, requests held, and requests switched for each task, helping you compare cache usage and routing behavior across different workloads.

Test Router Accuracy

In the Playground tab, select Router Evaluation. Upload your dataset and click Evaluate. Evaluation runs in a few minutes using LLM-as-a-Judge scoring.

Once the evaluation completes, you can see the following metrics:

• Completeness: Measures how thoroughly the response covers key details from the provided prompt.
• Correctness: Measures accuracy of the response against expected output. High scores indicate likely accuracy; low scores indicate possible hallucinations or errors.
• Tokens Used: Total token consumption per request
• Latency: Average and P95 response time

See more details about these metrics on the Evaluations page.

Edit, Duplicate, or Delete a Router in the Control Panel

In the My Routers tab, click the … menu next to the router. To update the router, select Edit Router to open the Edit a Router page. Make your updates and then click Update Router.

To duplicate an existing router, click Duplicate Router to open the Create a Router page. Update the router configurations and click Create Router.

To delete a router, click Delete Router. In the Delete Router window, enter the name of the router and click Delete.