How to Use Dedicated Inference on DigitalOcean Gradient™ AI Inference Hubpublic

Dedicated Inference is available in public preview and enabled for all users. You can contact support for questions or assistance.

You can create a dedicated inference deployment using the DigitalOcean API and Control Panel. Dedicated Inference is available in ATL1, NYC2 and TOR1 only.

When to Use Dedicated Inference Versus Serverless Inference

Dedicated Inference is a managed inference service that enables you to host and scale open-source and commercial LLMs on dedicated GPUs. It gives you more control over the environment so you can choose the GPU, tune performance, and optimize your models for throughput, latency, cost or concurrency. Dedicated inference is best suited for steady, high-throughput workloads.

Serverless inference lets you send API requests directly to foundation models. Choose serverless inference over dedicated inference when you need to get started quickly without managing any components behind an inference endpoint, don't have a custom model to host or optimize, or have unpredictable or spiky inference traffic.

Pricing for serverless inference is based on the number of tokens used, while pricing for dedicated inference is based on the GPU hours used.

If you want to use serverless inference, see Use Serverless Inference.

Create Dedicated Inference Using Automation

Creating a dedicated inference deployment using the API requires you to send a POST request to the /v2/dedicated-inferences endpoint with the following parameters in the request JSON body:

• name: An inference name.
• region: Datacenter region.
• hugging_face_token: Hugging Face token for the model to be deployed. Required only for gated models.
• model_slug: Model slug for the model to be deployed. Model slugs are same as the slugs present on Hugging Face.
• gpu_plan_slug: Slug for the GPU plan to deploy the model on. Use the /v2/dedicated-inferences/sizes endpoint to view available GPU plans and their slugs.
• node_count: Number of GPU nodes.

curl -i -X POST "https://api.digitalocean.com/v2/dedicated-inferences" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_TOKEN" \
  -d '{
    "spec": {
      "version": 1,
      "name": "new-dedicated-inference",
      "region": "atl1",
      "vpc": { "uuid": "7e5c619c-359c-44ca-87e2-47e98170c012" },
      "enable_public_endpoint": true,
      "model_deployments": [{
        "model_slug": "mistral/mistral-7b-instruct-v3",
        "model_provider": "hugging_face",
        "workload_config": {},
        "accelerators": [{
          "scale": 2,
          "type": "prefill_decode",
          "accelerator_slug": "gpu-mi300x1-192gb"
        }]
      }]
    },
    "access_tokens": { "hugging_face_token": "$HF_TOKEN" }
  }'

In the response body, the status field shows provisioning when the dedicated inference deployment is being provisioned and updates to active when provisioning is complete. If GPU capacity is not available, we attempt to provision the cluster for one to two hours, after which the status shows error.

To list all your dedicated inference deployments, send a GET request to the /v2/dedicated-inferences endpoint.

curl -i -X GET "https://api.digitalocean.com/v2/dedicated-inferences?region=nyc2&page=1&per_page=20" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_TOKEN"

Create Dedicated Inference Using the Control Panel

To create a dedicated inference deployment from the DigitalOcean Control Panel, click Inference Hub in the left menu. Select the Dedicated Inference tab and click Deploy Dedicated Inference to open the Create an Inference Deployment page.

Choose a Datacenter Region

Select the region for your dedicated inference deployment. All resources created in this datacenter are members of the same VPC network.

Select a Pre-trained Model

You can use a pre-trained model hosted on Hugging Face, such as DeepSeek, OpenAI, or Llama models. Click Details next to each model to learn more about the model and its capabilities.

Some models are gated and require you to request access on Hugging Face, as described in their documentation. If you select a gated model, make sure you have been granted access before creating your dedicated inference deployment. Then, provide a Hugging Face access token in the HuggingFace Access Token field. To learn how to create an access token, see the Hugging Face documentation on access tokens.

GPU Plan

Select a GPU plan to deploy your dedicated inference. The GPU plan determines the hardware resources allocated to your deployment. You can choose from 1- or 8-GPU AMD or NVIDIA GPU plans. Next, specify the number of GPU nodes.

Finalize

Specify a name for your dedicated inference deployment. Names must be lowercase and can only contain letters, numbers, and hyphens.

In the Summary section, review the cost based on the selections you made. Then, click Deploy Dedicated Inference to create your deployment. It may take several minutes for the deployment to be provisioned. The page lists all the dedicated inference instances that you have in your team. Here, you can view details such as the name, status, public endpoint and when the dedicated inference was created. The Status column shows Provisioning when the Dedicated inference is provisioning and updates to Active when the provisioning is complete. If GPU capacity is not available, we try to provision the cluster for 1-2 hr and then you see an Error status.

Once the deployment is active, you can view its details and use the endpoint to send inference requests. Make sure to note the access token, which is visible only once immediately after creation. Copy and store it securely, as you will need it to interact with the deployment endpoint.

Use the Dedicated Inference Endpoint

Dedicated inference deployments have two types of endpoints:

• Private Endpoint: Use when you want to allow resources within the VPC network the deployment belongs to access the endpoint.
• Public Endpoint: Use when you want to allow external sources to access the endpoint.

Using dedicated inference endpoints requires an access token to authenticate your requests. We automatically create an access token when provisioning your dedicated inference deployment. The same token can be used to authenticate requests to both the public and private endpoints. If you create the deployment using the Control Panel, the token is visible only once immediately after creation, so make sure to copy and store it securely. You can also send a GET request to the /v2/dedicated-inferences/{dedicated_inference_id}/tokens endpoint to list the current tokens.

If you want to create additional access tokens, in the Control Panel, go to the Endpoint Access Keys section of the Settings tab of your deployment, and click Create Access Key to open the Create Endpoint Access Key window. Provide a name for your access key and click Create Access Key. Copy the access token to use it later in your HTTP requests. You can use the /v2/dedicated-inferences/{dedicated_inference_id}/tokens endpoint.

curl --cacert <CA_CERTIFICATE_FILE_PATH> --location 'https://yeb78cdp0s5ux27pam0g1gg3-private-dedicated-inference.do-infra.ai/v1/chat/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <DEDICATED_INFERENCE_ACCESS_TOKEN>' \
-d '{
  "model": "openai/gpt-oss-120b",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "max_tokens": 150
}'

curl --location 'https://yeb78cdp0s5ux27pam0g1gg3-public-dedicated-inference.do-infra.ai/v1/chat/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <DEDICATED_INFERENCE_ACCESS_TOKEN>' \
-d '{
  "model": "openai/gpt-oss-120b",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "max_tokens": 150
}'

curl --location '$DEDICATED_INFERENCE_PUBLIC_ENDPOINT/v1/chat/completions' \
--header 'accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $AUTH_TOKEN' \
--data '{
  "model": "$MODEL_SLUG",
  "messages": [
    {
      "role": "user",
      "content": "Explain the concept of recursion in programming."
    }
  ],
  "max_tokens": 150
}
'

curl --cacert $CERT_FILE_PATH \
--location '$DEDICATED_INFERENCE_PRIVATE_ENDPOINT/v1/chat/completions' \
--header 'accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $AUTH_TOKEN' \
--data '{
  "model": "$MODEL_SLUG",
  "messages": [
    {
      "role": "user",
      "content": "Explain the concept of recursion in programming."
    }
  ],
  "max_tokens": 150
}
'

View or Update Dedicated Inference Settings

curl -i -X GET "https://api.digitalocean.com/v2/dedicated-inferences/6b5c619c-359c-44ca-87e2-47e98170c01d" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_TOKEN"

curl -i -X PATCH "https://api.digitalocean.com/v2/dedicated-inferences/6b5c619c-359c-44ca-87e2-47e98170c01d" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_TOKEN" \
  -d '{
    "spec": {
      "name": "renamed-dedicated-inference",
      "region": "atl1",
      "vpc": { "uuid": "997615ce-132d-4bae-9270-9ee21b395e5d" },
      "model_deployments": [{
        "model_slug": "mistral/mistral-7b-instruct-v3",
        "accelerator_slug": "gpu-mi300x1-192gb",
        "node_count": 3
      }]
    },
    "access_tokens": { "hugging_face_token": "$HF_TOKEN" }
  }'

Destroy Dedicated Inference Deployment

curl -i -X DELETE "https://api.digitalocean.com/v2/dedicated-inferences/6b5c619c-359c-44ca-87e2-47e98170c01d" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_TOKEN"