For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Create a Knowledge Base (Optional) public

You can optionally create a knowledge base, which is a collection of indexed content on a specific topic or domain, for your agents. A knowledge base gives an agent extra context during generation, such as your company’s internal documentation. To use this feature, enable the preview in the DigitalOcean Control Panel.

To create a knowledge base:

1. From the Control Panel, in the top-right, click Create, and then select Knowledge Bases to open the Create a knowledge base page.

2. In the Add data step, under the Choose your embeddings model section, click the Embedding model dropdown list, and select the embedding model you want to use. Embedding models index your data and convert it into numerical representations that the knowledge base can use during retrieval. Choose a model based on your data type and indexing token budget. For a list of available embedding models, see Available Models.

3. Under the Add data sources section, in the Select data sources to index sub-section, choose the data source you want to upload or connect. You must add at least one data source. Then, click Next step: Configure database.

   Knowledge bases support the following text-based file formats: .csv, .eml, .epub, .xls, .xlsx, .html, .md, .odt, .pdf, .txt, .rst, .rtf, .tsv, .doc, .docx, .xml, .json, and .jsonl.

   You can add any of the following data sources:

   File Upload

   To add files to update, click Upload a file, and then choose at least one file you want to upload. On the right of the file, you can click the trash can icon to remove the file, or on the bottom right, click Upload more files if you want to upload more files.

   For performance and reliability, we recommend uploading files no larger than 2 GB and uploading fewer than 100 files at a time.

   Spaces Bucket or Folder

   To add a Spaces bucket or folder, click Spaces bucket or folder, and then choose at least one bucket or folder you want to index. On the left of the bucket, you can click + to expand their contents and select specific folders to limit the indexed content.

   The system indexes all supported file formats in selected buckets and folders, regardless of privacy settings. For optimal performance and indexing quality, we recommend using five or fewer buckets and uploading only indexing data to your buckets.

   Web or Sitemap URL

   When you specify a website URL as a data source for your knowledge base, DigitalOcean uses a custom agent named DigitalOceanGradientAICrawler/1.0 to index the website content. The crawler indexes up to 5,500 pages and skips inaccessible or disallowed links to prevent excessively large indexing jobs.

   Depending on the behavior you select, the crawler follows HTML links on the site, indexes text and certain image types, and ignores videos and navigation links. It respects the website’s robots.txt rules, including any Disallow directives or the wildcard *.

   To add a URL for web crawling, select Add a web or sitemap URL. You can then choose to specify a Seed URL or a Sitemap URL.

   Specify Seed URL

   This option crawls only seed URLs and linked pages within the same path, domain, or subdomains. To specify a seed URL, select the Seed URL option. Then, in the Seed URL field, enter the public URL you want to crawl. The crawler indexes pages that are reachable from links you provide in this URL and indexes up to 5,500 pages.

   Under the Crawling Rules section, define the crawl scope:

   • Scoped crawls only the seed URL.
   • URL and all linked pages in path crawls the seed URL and all pages within the same path.
   • URL and all linked pages in domain crawls all pages in the same domain.
   • Subdomains crawls the domain and all its subdomains.

   Select the Index embedded media option to index supported images and other media encountered during the crawl. To include each page’s header and footer content, such as links in them, select the Include headers and footers navigation links option. We attempt to index supported embedded media types, such as images and SVGs and may increase indexing token count significantly.

   Specify Sitemap URL

   This option crawls only URLs listed in the sitemap. To crawl other URLs, use the Seed URL option or add another web crawling data source.

   The sitemap URL must be in .xml format where you can identify a specific list of URLs to crawl. You can use this option to add scoped URLs all at once instead of adding them individually or choosing a crawling rule for a seed URL.

   To specify a sitemap URL, select the sitemap URL option. Then, in the Sitemap URL field, type the URL you want to crawl. For example, docs.digitalocean.com/sitemap.xml.

   Select the Index embedded media option to index supported images and other media encountered during the crawl. To include each page’s header and footer content, such as links they contain, select the Include headers and footers navigation links option. We attempt to index supported embedded media types, like images and SVGs. These media types may increase indexing token count significantly.

   To verify the crawl completed, re-add the same seed or sitemap URL as a new data source. If it shows zero tokens, the original crawl indexed all content, and you can delete the duplicate.

   Dropbox Folder

   If you haven’t connected your Dropbox account, on the right of the Dropbox option, click Connect account to first log in to your Dropbox account and authorize the connection.

   To add a Dropbox folder, click Pull from a Dropbox folder, and then choose at least one folder you want to index. On the left of the folder, click the + to expand their contents and select specific folders to limit the indexed content.

   Amazon S3 Bucket or Folder

   To add an Amazon S3 bucket or folder, select Amazon S3 bucket or folder, and then provide the following credentials in the fields provided:

   • Access Key ID, the IAM access key ID for your S3 bucket or folder.
   • Secret Key, the secret key associated with your access key ID.
   • Bucket or folder, the name of the S3 bucket or folder you want to index.
   • Region, the AWS region where your S3 bucket or folder is located, such as us-east-1 or eu-west-1.

   On the right of the S3 bucket or folder, click the + button to add the S3 bucket or folder as a data source, and then below your recently added S3 bucket or folder, you can fill out the Bucket or folder and Region fields to add another S3 bucket or file.

4. In the Knowledge base name field, either enter a name for your knowledge base or use the auto-generated name.

5. In the Where should your knowledge base live? section, under the OpenSearch database options sub-section, select either Use existing to connect to an existing OpenSearch database or Create new to provision a new one.

   Use Existing OpenSearch Database

   If you choose Use existing, under the Select an OpenSearch database section, click the dropdown menu, then select the database you want to use. If it already contains data, it may limit how much new data you can index. You only pay for successfully indexed data.

   Create a New OpenSearch Database

   If you choose Create new, under the Choose a datacenter region section, select the default datacenter region for your knowledge base, or click the dropdown menu on the right of the datacenter region below the default, to choose a different one. We recommend choosing the same region as your Gradient AI Platform agents to reduce latency. Most of the Agent Platform infrastructure is in the TOR1 region. Creating an OpenSearch database in a different region may increase latency between your agents and your knowledge base.

   New databases are automatically sized to the smallest option that fits your data. We recommend allocating about twice the size of your original dataset to efficiently store embeddings.

   Under the VPC Network section, choose the VPC network you want to use.

   Click Next step: Review and create.

6. Under the Final Details section, click the Select a project dropdown list to choose the project your knowledge base should live in.

7. Under the Tags sub-section, optionally add tags to help organize and filter your knowledge base. Tags can include letters, numbers, colons, dashes, and underscores.

8. Under Review, review the selected embeddings model token price, the estimated size of your data sources, and the OpenSearch database configuration, and then click Create knowledge base and database.

After the knowledge base is created, you can attach it to new or existing agents.

Create an Agent

Create an agent to define your instructions, model, workspace, and optional knowledge base attachments.

To create an agent:

1. From the Control Panel, in the top-right, click Create, and then select Agents to open the Create an agent page.

2. Under the Configure your agent section, in the Agent name field, either enter a name for your agent or use the auto-generated name.

3. Under the Agent instructions sub-section, add your agent instructions. You can also start with one of the ready-to-use templates. Agent instructions can be up to 10,000 characters long.

4. Click the Select a model provider dropdown menu, and then choose the foundation model that best fits your use case. Token cost varies by provider and model. For supported models and their capabilities, see Available Models and Pricing. Depending on the provider and model you choose, you may also be able to configure model-specific options, such as reasoning level or other response settings.

   Review and accept the model’s Terms and Conditions checkbox.

5. Under the Where should your agent live? section, choose an existing workspace or create a new one. To create a new workspace, enter a Workspace name and optionally add a Workspace description.

6. Under the Optional Configuration section, in the Add knowledge bases sub-section, select any knowledge bases you want to attach to your agent for retrieval-augmented generation (RAG).

7. Optionally, click Connect agent to VPC network to add the agent to a VPC network if you want it to communicate with private resources in your network or keep traffic isolated from the public internet. In the VPC network dropdown menu, select the VPC network you want to use.

8. Under the Final Details section, choose the project your agent should live in.

9. Under the Tags sub-section, optionally add tags to help organize and filter your agent. Tags can include letters, numbers, colons, dashes, and underscores.

10. Under the Estimated cost summary sub-section, review the configuration and estimated cost, and then click Create Agent.

After creating the agent, you can test its performance.

Interact With Your Agent

After you create an agent, you can interact with it by sending requests to its endpoint. First, create an access key for authentication, then use the agent endpoint and key to submit queries and receive responses.

1. From the Control Panel, in the left menu, click Agent Platform, click the Workspaces tab, and then click the workspace that has the agent you want to interact with.

2. Under the Agents tab, find and click the agent you want to use, and then click its Settings tab.

3. Under the Endpoint Access Keys sub-section, click Create Key to open the the Create Agent Access Key window.

4. Under the Key name field, enter a name for your key, and then click Create. Copy the key, and then save it in a secure location.

5. In the Agent Essentials section of the agent’s Overview tab, copy the agent’s endpoint from the ENDPOINT sub-section. Then, use the endpoint and the access key to send requests and generate responses to user queries.

   The following cURL and Python OpenAI examples show how to use the agent’s endpoint:

   cURL example

   The cURL example uses environment variables to store the agent’s endpoint ($AGENT_ENDPOINT) and access key ($AGENT_ACCESS_KEY). To return retrieval information about how the response was generated, such as the knowledge base data, guardrails, and functions used, the include_retrieval_info, include_guardrails_info, and the include_functions_info parameters to true in the request body are set to true.

   curl -i \
     -X POST \
     $AGENT_ENDPOINT/api/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $AGENT_ACCESS_KEY" \
     -d '{
       "messages": [
         {
           "role": "user",
           "content": "What is the capital of France?"
         }
       ],
       "stream": false,
       "include_functions_info": true,
       "include_retrieval_info": true,
       "include_guardrails_info": true
     }'

   Python OpenAI example

   use-agent-endpoint-key.py

   # Install OS, JSON, and OpenAI libraries.
   import os
   import json
   from openai import OpenAI
   
   # Set your agent endpoint and access key as environment variables in your OS.
   agent_endpoint = os.getenv("agent_endpoint") + "/api/v1/" 
   agent_access_key = os.getenv("agent_access_key")
   
   if __name__ == "__main__":
       client = OpenAI(
           base_url = agent_endpoint,
           api_key = agent_access_key,
       )
   
       response = client.chat.completions.create(
           model = "n/a",
           messages = [{"role": "user", "content": "Can you provide the name of France's capital in JSON format."}],
           extra_body = {"include_retrieval_info": True}
       )
   
   # Prints response's content and retrieval object.
       for choice in response.choices:
           print(choice.message.content)
       
       response_dict = response.to_dict()
   
       print("\nFull retrieval object:")
       print(json.dumps(response_dict["retrieval"], indent=2))

Embed a Chatbot

You can make your agent available through an embeddable chatbot interface.

To embed a chatbot:

1. From the Control Panel, in the left menu, click Agent Platform, click the Workspaces tab, and then click the workspace that has the agent you want to use for your chatbot.

2. Under the Agents tab, find and click the agent you want to use.

3. In the Agent Essentials section of the agent’s Overview tab, click Edit in the ENDPOINT sub-section to open the Set endpoint availability to public window.

4. Set your agent endpoint to Public so outside applications can access it without requiring the agent access key, and then click Save.

   This adds the CHATBOT embed code below the ENDPOINT section. The chatbot embed code is an HTML <script> element that you can copy and paste into your application or website.

5. Copy the HTML chatbot embed code, and then paste it into your application or website.

Next Steps

Once you’ve set up an agent, you can:

• Test your agent’s responses and adjust its model settings and configuration in the Agent Playground.
• Add rules and constraints to help control agent responses with Guardrails.