For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

A knowledge base stores data sources (such as local file uploads, DigitalOcean Spaces buckets or folders, public seed or sitemap URLs, Dropbox folders, and Amazon S3 buckets), that AI agents can use to answer questions with retrieval-augmented generation (RAG). RAG helps agents provide more accurate, current, and domain-specific responses.

When you create a knowledge base, your data is immediately indexed by converting the content into vector embeddings using an embeddings model. These embeddings capture the meaning of your content and are stored in a Managed OpenSearch database, which you can scale to improve performance. The embeddings model determines token limits, chunk size ranges, and whether size estimates are available when you download the CSV for the indexing job.

Each knowledge base requires at least one data source, and you can add or remove sources after creation.

Create a Knowledge Base Using the Control Panel

To create a knowledge base, go to the DigitalOcean Control Panel, in the left menu, click INFERENCE, click Agent Platform, and then click the Knowledge bases tab.

Then, in the top-right, click Create Knowledge Base to open the Create a knowledge base page.

Choose Your Embeddings Model

Under the Add data step, click the Choose your embeddings model dropdown list, and choose an embeddings model. You can’t change the model after creating your knowledge base.

An embeddings model converts your data into vector embeddings which are stored in an OpenSearch database. We offer multiple embeddings models for different use cases. The indexing costs depend on the selected model and the size of your data.

To understand your indexing costs, click How much will I pay for an indexing job?. This opens the Estimating index job costs window, which shows estimated indexing costs by embeddings model token rate and dataset size. Larger datasets cost more to index, and you only pay for successfully indexed data. Final costs may vary. For details, see embeddings model pricing.

Configure Reranking

You can optionally enable reranking for retrieval results when you create the knowledge base. Reranking reorders results after the initial search so the most relevant chunks are more likely to appear first and be used in generated responses.

If reranking is enabled, reranking tokens are billed separately from vectorizing retrieval queries. For more information, see knowledge base pricing.

Under the Choose your reranking model section, click the Reranking model (optional) dropdown list, and then select a reranking model. Choose a reranking model based on the relevance quality you need, your latency requirements, and your cost considerations. You can’t change the reranking model after enabling reranking. To see the reranking models, see our available reranking models.

If reranking is enabled, retrieval results show that reranking is active, along with the model name and per-token pricing. Reranking applies to all retrieval requests and incurs charges on each request.

Add Data Sources

You can add multiple types of data sources and include as many as needed. To save processing time and cost, organize your files in dedicated Spaces buckets, specific folders, or local storage containing only relevant files.

To avoid delays, we recommend uploading fewer than 100 files at a time, each under 2 GB. For larger uploads, use the DigitalOcean API. If uploads continue to stall, contact support.

In the Add data sources section, under the Select data sources to index sub-section, select the type of data you want to add.

Knowledge bases support the following text-based file formats: .csv, .eml, .epub, .xls, .xlsx, .html, .md, .odt, .pdf, .txt, .rst, .rtf, .tsv, .doc, .docx, .xml, .json, and .jsonl. When supported files contain embedded media, such as images or SVGs, we also attempt to index that content.

You can add any of the following data sources:

If you want to control how the data source is split into chunks during indexing, click Advanced Options to configure its chunking strategy. By default, all data sources use section-based chunking. For more information about chunking strategies, see our chunking strategy best practices.

Then, click Add selected data source.

After adding your data source, add another data source if you want, and then in the top-right, review the data sources you’ve added with their data source type, estimated size, and configuration (such as, chunking configuration).

If one of your data sources fails to be added, click the data source method you chose for that data source, and then on the right of the failed file, bucket, folder or URL, click the trash icon, and then try again. If it fails again, contact support.

After adding all your data sources and reviewing them, click Next step: Configure database, or configure chunking strategies for your data sources

Configure Chunking Strategy

Chunking controls how each data source is split before embedding and indexing the source into your knowledge base. Data sources use section-based chunking by default, and you can use different strategies in the same knowledge base by adding content as separate data sources.

Chunking strategies depend on the selected embeddings model. Chunk sizes must stay within the model’s token window and be at least approximately 100 tokens. For insights on which strategy to choose and configuration setup, see our chunking best practices and the chunking parameters reference.

To configure chunking after selecting your data source, at the bottom of the data source’s selection window, click Advanced Options to open the Chunking strategy section.

Under the Select a chunking strategy for this data source sub-section, click the strategy you want to use, and then configure its parameters.

You can choose one of the following strategies:

If you decide to change chunking strategies after creating your knowledge base, this requires re-indexing, which consumes additional tokens.

After setting up your chunking configuration your data source, click Add selected data source, and repeat for another data source, if necessary. Then, click Next step: Configure database.

Choose Knowledge Base Name

In the Configure database step, either keep the autogenerated name or choose a unique name using 3 to 63 characters, including only letters, numbers, dashes, and periods.

Choose Your OpenSearch Database

Knowledge bases require an OpenSearch database to store vector embeddings. On the top-right of the page, use the estimated sizes in your added data sources list to choose a database size. We recommend allocating at least twice the total estimated data size. Database size is based on OpenSearch pricing.

If you want to remove a data source, click Edit data sources, which takes you back to the Add data step, and then under the Add data sources section, select the data source method with the data you want to remove, and then click the trash icon next to the data.

In the Where should your knowledge base live? section, under the OpenSearch database options sub-section, select either Use existing to connect to an existing OpenSearch database or Create new to provision a new one.

Afterwards, click Next step: Review and create.

Finalize Details

In the Review step, under the Final Details section, click the Select a project dropdown list to choose the project you want your knowledge base stored in.

In the Tags field, add tags to help organize and filter your knowledge base. Tags can include letters, numbers, colons, dashes, and underscores. Choose a tag name, then press ENTER or SPACEBAR to add it. Use the arrow keys to navigate and the BACKSPACE key to remove tags.

Under the Review section, use the configuration sub-sections to confirm your embeddings model and token cost, selected data sources and estimated total size, and OpenSearch database specs and price. Existing databases don’t add new database charges.

To change a configuration, click Edit next to the sub-section you want to update, and then make your changes in its respective step.

After finalizing and reviewing your knowledge base setup, click Create knowledge base (or Create Knowledge Base and database).

Provisioning Your Knowledge Base

After creating your knowledge base, it appears under Knowledge Bases tab and immediately begins provisioning. Provisioning typically takes five minutes or longer as the process embeds and indexes your data sources.

After provisioning completes, click your knowledge base to view its Overview section. Under the LATEST INDEXING DETAILS sub-section view a summary of the indexing results, including final costs. You can download a CSV file of this index job in the knowledge base’s Activity tab.

If indexing takes longer than expected, let the indexing job continue running until it either completes or fails. If it fails, check the Activity tab for detailed logs to understand what went wrong (for example, failed or skipped files).

After reviewing the indexing job logs and fixing any issues, on the right of the log, click Re-run to restart indexing. If problems persist, contact support.

If you added a seed or site map URL as a data source, verify web crawling is indexed successfully by re-adding the same seed or sitemap URL as a new data source. If the indexing job results of the duplicated data source shows zero tokens, the original crawl indexed all content, and you can delete the duplicate.

To keep your data sources up to date automatically without manual re-ingest, we recommend also setting up scheduled indexing.

Create a Knowledge Base via the API

To create a knowledge base via the DigitalOcean API, provide a name, an embeddings model, a project ID, and a datacenter region.

You can also specify the ID of an existing OpenSearch database or a chunking strategy. If you don’t provide a database, we create one and size one for you automatically.

You can configure your chunking strategy for a data source when creating your knowledge base with the following optional fields:

• chunking_algorithm: The chunking strategy (section, semantic, hierarchical, or fixed).
• chunking_options: A configuration object containing parameters such as max_chunk_size, semantic_threshold, parent_chunk_size, or child_chunk_size.

Chunking is applied per data source. Updating chunking settings triggers re-indexing, which consumes tokens. For details and recommendations, see our chunking best practices and chunking parameters reference.

If you don’t configure a chunking strategy for a data source, the knowledge base uses section-based chunking (section) by default.

To list available embeddings models and their IDs, call the /v2/gen-ai/models endpoint with the usecases query parameter.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/knowledge_bases" \
  -d '{
    "name": "kb-api-create",
    "embedding_model_uuid": "05700391-7aa8-11ef-bf8f-4e013e2ddde4",
    "project_id": "37455431-84bd-4fa2-94cf-e8486f8f8c5e",
    "tags": [
      "tag1"
    ],
    "database_id": "abf1055a-745d-4c24-a1db-1959ea819264",
    "datasources": [
      {
          "spaces_data_source": {
              "bucket_name": "test-public-gen-ai",
              "region": "tor1"
            },
            "chunking_algorithm": "CHUNKING_ALGORITHM_HIERARCHICAL",
            "chunking_options": {
              "parent_chunk_size": 1000,
              "child_chunk_size": 350
            }
      },
      {
        "web_crawler_data_source": {
          "base_url": "https://example.com",
          "crawling_option": "SCOPED",
          "embed_media": false,
          "exclude_tags": ["nav","footer","header","aside","script","style","form","iframe", "noscript"]
        },
        "chunking_algorithm": "CHUNKING_ALGORITHM_SEMANTIC",
        "chunking_options": {
          "max_chunk_size": 500,
          "semantic_threshold": 0.6
        }
      },
      {
        "spaces_data_source": {
            "bucket_name": "test-public-gen-ai-2",
            "region": "tor1"
          },
          "chunking_algorithm": "CHUNKING_ALGORITHM_FIXED_LENGTH",
          "chunking_options": {
            "max_chunk_size": 400
          }
      },
    ],
    "region": "tor1",
    "vpc_uuid": "f7176e0b-8c5e-4e32-948e-79327e56225a",
    "reranking_config": {
      "enabled": true,
      "model": "bge-reranker-v2-m3"
    }
  }'

After creating your knowledge base, indexing begins automatically. You can list all knowledge bases, view a knowledge base, or update one.

To add another data source, use the Data Sources endpoint.

To retrieve metadata for embeddings models, use the List Models endpoint.

View Indexing Job Logs

When you create or update a knowledge base, its data sources are indexed. Each indexing job is logged and can be downloaded as a CSV for debugging or auditing.

To track these logs, go to the Control Panel, in the left menu, click INFERENCE, and then click Agent Platform.

Then, click the Knowledge bases tab, select the knowledge base you want to viewing indexing job logs for, and then click its Activity tab.

Above the Activity section, the indexing job shows real-time progress. When the job finishes, its status helps you interpret the results:

• Completed: All files indexed successfully.
• No Changes: No updates were detected in the data sources and no files or URLs were re-indexed.
• Partially Completed: Some files were indexed, but others were skipped or failed. This may create gaps in your knowledge base results.
• Failed: No files were indexed, usually due to a system or configuration issue (for example, formatting problems or unsupported characters).

If you try to run another indexing job while another one is in progress, the new job is skipped since only one job can run at a time.

Under the Activity section, you can see the 15 most recent indexing jobs. Since only 15 jobs are stored, we recommend downloading the CSV by clicking Download Details for each index job ran.

Under the Activity section, each logged indexing job shows:

• The overall status (see above).
• The token count with the indexing rate (for example, “0 tokens @ $0.09/1M”). You only pay for successfully indexed data, and prices are rounded down to two decimal places. For more details on indexing costs, see knowledge base pricing.
• How many files were indexed, skipped, or failed (for example, “Indexed 0 of 2 files/URLs (0%)”).
• A date and timestamp of when the indexing job ran.
• A Download Details option to download the indexing job as a CSV file.

File-level details, such as filenames and error reasons, are only available in the downloadable CSV. Files with unchanged content are skipped to avoid extra charges.

If files fail, click Download Details for that run, review the errors, fix the data, then re-upload the file and re-run the job. If the file continues to fail, contact support.

Test Chunking Strategy

Chunking quality depends on document structure and formatting. To improve chunking, test retrieval results by running retrieval queries or, if the knowledge base is attached to an AI agent, by running agent evaluations.

To change chunking settings, delete the data source, and then add it again with the new chunking configuration.

Since each indexing job consumes tokens, choose chunking settings carefully. For reference, semantic and hierarchical chunking can increase token usage, while section-based and fixed length chunking offer more predictable costs. For details on chunking costs, see knowledge base pricing.

Test Reranking

Reranking can improve retrieval relevance by reordering results after the initial search so the most relevant chunks are more likely to appear first. To evaluate its impact, test retrieval results in the Control Panel, compare outputs with and without reranking enabled, and adjust the setting based on answer quality and cost.

If reranking is enabled, reranking tokens are billed separately from vectorizing retrieval queries. For more information, see knowledge base pricing.

Before testing your reranking model, go to the Control Panel, in the left menu, click Agent Platform.

In the Knowledge Bases tab, click the knowledge base you want to enable reranking model for, click its Settings tab, and then under the Reranking model section, in the right, click Enable.

Under the Model dropdown list, choose the reranking model you want to choose. Choose a reranking model based on the relevance quality you need, your latency requirements, and your cost considerations. You can’t change the reranking model after enabling reranking, and then click Enable reranking.

To test your reranking model, you can send queries to your knowledge base in the Control Panel, use the DigitalOcean API, or test retrieval and responses in RAG Playground.

Disable Reranking

Disabling reranking stops re-scoring retrieved results. This affects all future retrievals immediately. You may need to disable reranking for an individual retrieval request to compare baseline retrieval results, reduce cost, or troubleshoot whether reranking is improving relevance for a specific query.

If needed, to disable your reranking model, go to the Control Panel, in the left menu, click Agent Platform. In the Knowledge Bases tab, click the knowledge base with the reranking model you want to disable, and then click the Settings tab.

In the Reranking model section, in the right, click Edit, and then click Disable ranking to open the Disable reranking model window.

To confirm disabling the reranking model, click Disable reranking.

After disabling reranking, test retrieval results again, and then compare outputs with and without reranking enabled to confirm it improves relevance for your use case.

Retrieve Data from a Knowledge Base Using the Control Panel

Retrieve data from a knowledge base to test how well it returns relevant content for semantic, keyword, or hybrid searches. Results are returned as chunks, which are smaller sections of your source content along with any metadata attached during ingestion, such as category, product, source, or page information.

To retrieve data from a knowledge base, go to the Control Panel, in the left menu, click Agent Platform, click the Knowledge Bases tab, click the knowledge base you want to query, and then click its Retrieve tab.

To enable reranking, go to the knowledge base’s Settings tab. This means retrieved results are reordered by the reranking model to improve relevance.

If you want to disable reranking for one query, go to the knowledge base’s Settings tab, and click Disable reranking.

In the Search query field, type the text you want to search for.

In the num_results field, set how many results to return.

You can run one of the following search types against your knowledge base using the alpha field:

• Semantic search: Finds results by meaning, even when the exact words do not match. Best for conversational or open-ended queries. Set alpha: 1.
• Keyword search: Finds results by exact word matches. Best for precise terms such as names, dates, IDs, or product codes. Set alpha: 0.
• Hybrid search: Combines semantic and keyword search in one result set. Best as the default for most queries. Start with alpha: 0.5, and then tune lower for more exact matching or higher for more meaning-based matching.

An alpha value of 0 uses keyword retrieval, a value of 1 uses semantic retrieval, and values in between use hybrid retrieval.

Under the Filters (optional) tab, you can combine filters with and_all, which returns only chunks that match every condition, or or_all, which returns chunks that match at least one condition.

You can filter results by first clicking the Category dropdown list and choosing one of the following chunk metadata fields:

• item_name: The source item name, such as a file name or URL path. For example, https://docs.digitalocean.com/products/ai-platform/.
• ingested_timestamp: The date and time when the chunk was ingested into the knowledge base. For example, 2025-12-01T00:00:00Z.
• page_number: The page number of the source document, when available. For example, 3.
• chunk_category: The category assigned to the chunk during processing, such as the extracted content type.

Then, click the condition dropdown to choose one of the following:

• equals: Returns chunks where the field exactly matches the value. We recommend using this with item_name to target a specific file or URL, or with chunk_category when you want only one specific chunk type.
• not_equals: Excludes chunks where the field exactly matches the value.
• greater_than: Returns chunks where the field value is greater than the value you enter. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
• greater_than_or_equals: Returns chunks where the field value is greater than or equal to the value you enter. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
• less_than: Returns chunks where the field value is less than the value you enter. We recommend using this with ingested_timestamp or page_number to filter by date or page range.

Then, in the Value field, enter the value for the filter.

To add a filter, click the + Add condition. If you want to delete a filter, on the right of the filter condition, click x.

After setting up your search query, click Retrieve.

The Results section shows the total number of results retrieved, the query time in milliseconds (ms), the alpha value, and each retrieved chunk with its source file, page number when available, category metadata, and relevance score when available. Higher-relevance chunks are visually emphasized to help you spot the strongest matches more quickly.

If needed, you can also compare how different models answer questions using retrieved knowledge base content in the RAG Playground.

Use Code Snippets to Retrieve Data

Use the code snippets to turn your current retrieval settings into working API requests for testing, automation, or application development. This lets you reuse the same query, filters, and retrieval settings outside the Control Panel without writing the request from scratch.

To use the live auto-generated code snippets for the Gradient SDK (Python) or cURL based on your current query settings, click the Retrieve Endpoint tab, select the snippet you want from the code snippet dropdown list, and then click Copy to paste it where you need it or Download to save it.

Retrieve Data from a Knowledge Base Using the Knowledge Base API

You can query a knowledge base to retrieve the most relevant chunks, along with metadata, scores, and any hierarchical context. The knowledge base retrieve API is available at https://kbaas.do-ai.run and has the following endpoint:

Data retrieval depends on how the knowledge base was configured and how the request is sent:

• Chunking affects how content is split during indexing, which can affect retrieval results. Chunking is configured when you create or update a knowledge base data source, not during retrieval.
• Reranking optionally improves the ordering of retrieved results. If reranking is enabled for the knowledge base, you can disable it for an individual retrieval request.

You can also call the retrieve endpoint using the Gradient SDK.

To retrieve chunks from a knowledge base, send a POST request to /v1/<knowledge-base-uuid>/retrieve using your DigitalOcean API token. Requests to the retrieve API require a DigitalOcean API token created from the Settings page with the GenAI:read scope enabled.

You can include the following fields in the request body:

• query: Specifies the search query string.

• num_results: Defines the number between 0 and 100 of results to return.

• alpha: Controls the balance between lexical and semantic retrieval. Values range from 0 (lexical only) to 1 (semantic only). We recommend setting lower values for structured or technical queries, higher values for conversational or exploratory queries, and mid-range values for balanced precision and recall.

• reranking: Optionally controls reranking behavior for the request. Use this to disable reranking for an individual retrieval request.

• filters: Optionally narrows results by matching chunk metadata, such as:

  • item_name: The source item name, such as a file name or URL path.
  • ingested_timestamp: The date and time when the chunk was ingested into the knowledge base.
  • page_number: The page number of the source document, when available.
  • chunk_category: The category assigned to the chunk during processing.

  Supported operations include:

  • equals: Matches values exactly. We recommend using this with item_name to target a specific file or URL, or with chunk_category to return one specific chunk type.
  • not_equals: Excludes exact matches.
  • greater_than: Matches values greater than the input. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
  • greater_than_or_equals: Matches values greater than or equal to the input. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
  • less_than: Matches values less than the input. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
  • less_than_or_equals: Matches values less than or equal to the input. We recommend using this with ingested_timestamp or page_number to filter by date or page range.
  • starts_with: Matches text values that begin with the input. We recommend using this with item_name to restrict results to a specific file, URL, or path prefix.

  Combine filters with and_all to match all conditions or or_all to match any condition. You can also nest groups for more complex logic.

The following example shows a hybrid retrieval request with no filters:

curl --location 'https://kbaas.do-ai.run/v1/<knowledge-base-uuid>/retrieve' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DO_API_TOKEN' \
--data '{
    "query": "How do I build an agent on DigitalOcean?",
    "num_results": 5,
    "alpha": 0.5
}'

When using filters, the key must reference a metadata field in the dataset, and the value must match the field type, such as string, number, or boolean. Use operators compatible with the field type. For example, avoid greater_than on string fields unless they use a supported date or version format.

The following example uses or_all filter to return chunks where the source path starts with a specific documentation URL or the chunk was ingested on or after a specified date:

curl --location 'https://kbaas.do-ai.run/v1/<knowledge-base-uuid>/retrieve' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DO_API_TOKEN' \
--data '{
    "query": "How do I build an agent on DigitalOcean",
    "num_results": 5,
    "filters": {
        "or_all": [
            {
                "starts_with": {
                    "key": "item_name",
                    "value": "https://docs.digitalocean.com/products/ai-platform/"
                },
                "greater_than_or_equals": {
                    "key": "ingested_timestamp",
                    "value": "2025-12-01"
                }
            }
        ]
    },
    "alpha": 0.5
}'

The following example shows a response returned by the knowledge base API endpoint:

{
    "results": [
        {
            "metadata": {
                "chunk_category": "CompositeElement",
                "ingested_timestamp": "2025-12-15T15:23:19.191428+00:00",
                "item_name": "https://docs.digitalocean.com/products/ai-platform/how-to/create-agents/",
            },
            "text_content": "Chunk 1 Content"
        },
        {
            "metadata": {
                "chunk_category": "CompositeElement",
                "ingested_timestamp": "2025-12-15T15:23:19.191428+00:00",
                "item_name": "https://docs.digitalocean.com/products/ai-platform/how-to/use-serverless-inference/",
            },
            "text_content": "Chunk2 content"
        },
        ...
    ],
    "total_results": 5
}

Enable reranking for the request by setting the enabled parameter in reranking to true (or false).

The following example enables reranking for a hybrid retrieval request:

curl --location 'https://kbaas.do-ai.run/v1/<knowledge-base-uuid>/retrieve' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DO_API_TOKEN' \
--data '{
    "query": "How do I build an agent on DigitalOcean?",
    "num_results": 5,
    "alpha": 0.5,
    "reranking": {
        "enabled": true
    }
}'

The following example shows a reranked response returned by the knowledge base API endpoint:

{
    "results": [
        {
            "metadata": {
                "chunk_category": "CompositeElement",
                "ingested_timestamp": "2025-12-15T15:23:19.191428+00:00",
                "item_name": "https://docs.digitalocean.com/products/ai-platform/how-to/create-agents/"
            },
            "text_content": "Chunk 1 content"
        },
        {
            "metadata": {
                "chunk_category": "CompositeElement",
                "ingested_timestamp": "2025-12-15T15:23:19.191428+00:00",
                "item_name": "https://docs.digitalocean.com/products/ai-platform/how-to/use-serverless-inference/"
            },
            "text_content": "Chunk 2 content"
        }
    ],
    "total_results": 5
    ...
}

Manage Data Sources

You can add, remove, reindex, or enable auto-indexing for your data sources as needed.

To manage your data sources, go to the Control Panel, in the left-hand menu, click Agent Platform, and then click the Knowledge Bases tab.

Then, find the knowledge base with the data sources you want to manage, on the right of it, click …, and then click Manage data sources to open the Data sources tab.

Add a Data Source Using the Control Panel

You can add multiple types of data sources and include as many as needed. To save processing time and cost, organize your files in dedicated Spaces buckets, specific folders, or local storage containing only relevant files.

To avoid delays, we recommend uploading fewer than 100 files at a time, each under 2 GB. For larger uploads, use the DigitalOcean API. If uploads continue to stall, contact support.

To add a data source, on the right, click Add source to open the Add Data Source page.

Under the Select data sources to index section, select the type of data you want to add.

Knowledge bases support the following text-based file formats: .csv, .eml, .epub, .xls, .xlsx, .html, .md, .odt, .pdf, .txt, .rst, .rtf, .tsv, .doc, .docx, .xml, .json, and .jsonl. When supported files contain embedded media, such as images or SVGs, we also attempt to index that content.

You can add any of the following data sources:

If you want to control how the data source is split into chunks during indexing, click Advanced Options to configure its chunking strategy. By default, all data sources use section-based chunking. For more information about chunking strategies, see our chunking strategy best practices.

Then, click Add selected data source.

Below the Data sources to be indexed section, review each data source’s estimated size, configuration, and status:

• Ready: The data source is uploaded and ready for indexing.
• Uploading: The data source is still uploading and isn’t ready for indexing.
• Error: Upload or processing failed. If your data source failed, remove the data source and try again. If the issue persists, contact support.

If you want to remove a data source, click the trash icon next to it.

Under Summary, review the embeddings model and token cost, total estimated dataset size, and number of data sources.

To estimate indexing costs, click How much will I pay for an indexing job? to open the Estimating indexing job costs window. Larger datasets cost more to index, but you only pay for successfully indexed data. Final costs may vary. For details, see embeddings model pricing of the duplicated data source shows zero tokens, the original crawl indexed all content, and you can delete the duplicate.

Add a Data Source via the API

To add a data source via the API, provide the knowledge base’s unique identifier and specify the source you want to index, such as a bucket, folder, file, or URL.

To retrieve knowledge base IDs, use the /v2/gen-ai/knowledge_bases endpoint.

You can optionally configure chunking for each data source:

• chunking_algorithm: The chunking strategy (section, semantic, hierarchical, or fixed).
• chunking_options: A configuration object defining parameters such as max_chunk_size, semantic_threshold, parent_chunk_size, or child_chunk_size.

If you decide to change your chunking configuration later, this triggers a re-indexing job, which consumes tokens.

After adding the data source, indexing automatically starts to make its content available for retrieval.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/knowledge_bases/20cd8434-6ea1-11f0-bf8f-4e013e2ddde4/data_sources" \
  -d '{
"knowledge_base_uuid": "20cd8434-6ea1-11f0-bf8f-4e013e2ddde4",
"web_crawler_data_source": {
  "base_url": "https://example.com",
  "crawling_option": "SCOPED",
  "embed_media": false,
  "exclude_tags": ["nav","footer","header","aside","script","style","form","iframe", "noscript"]
},
"chunking_algorithm": "CHUNKING_ALGORITHM_SECTION_BASED",
"chunking_options": {
  "max_chunk_size": 500
}
}'

To confirm the data source was added, list the knowledge base’s data sources.

Remove a Data Source Using the Control Panel

Removing a data source removes it and its configurations, such as chunking settings, from the knowledge base without deleting the original file, folder, bucket, or URL.

To remove a data source, on the right of the data source you want to delete, click the trash icon to open the Remove data source window.

Then, enter the name of the data source to confirm its removal, and then click Destroy.

After removal, the knowledge base automatically reindexes the remaining data sources. You can track the reindexing process in the Activity tab.

Remove a Data Source via the API

To remove a data source via the API, provide the knowledge base ID and the data source ID. Removing a data source removes it and its configurations, such as chunking settings, from the knowledge base without deleting the original file, folder, bucket, or URL.

You can find data source IDs by listing the knowledge base’s data sources.

curl -X DELETE \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/knowledge_bases/9a6e3975-b0c6-11ef-bf8f-4e013e2ddde4/data_sources/bd2a2db5-b8b0-11ef-bf8f-4e013e2ddde4"

Re-Index Data Sources Using the Control Panel

Re-indexing data sources embeds any new data, which consumes tokens based on the knowledge base’s embeddings model. Buckets, folders, and URLs are updated. Uploaded files aren’t updated. To update uploaded files, remove the files, and re-add it.

You aren’t charged for removed data or indexing events that create no new embeddings. Exact indexing costs are available only after the job completes. For estimates, see knowledge base pricing.

Re-indexing may not fix skipped or failed files unless you resolve the underlying issues first. Review indexing details before retrying.

To reindex data sources, click the Activity tab, under the Activity section, and then on the right of the most recent indexing job, click Re-run to open the Confirm sources update window.

Click Update sources to reindex the data. You are only charged for any new data found during the indexing. You can view the results of the reindexing job in the Activity tab.

Re-Index Data Sources via the API

To reindex data sources via the API, create an indexing job with the knowledge base ID and data source ID. Use the Create Indexing Job endpoint to start the indexing job.

curl -X POST \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/indexing_jobs" \
  -d '{
    "knowledge_base_uuid": "9758a232-b351-11ef-bf8f-4e013e2ddde4",
    "data_source_uuids": [
      "9a825ee0-bbb1-11ef-bf8f-4e013e2ddde4"
    ]
  }'

To check the indexing job’s progress, use the Get Indexing Job endpoint.

After indexing completes, use the Get Knowledge Base endpoint to confirm completion and review the final token count and indexing cost.

If the job takes longer than expected, cancel it using the Cancel Indexing Job endpoint, and then restart it. If issues persist, contact support for assistance.

Schedule Indexing Data Sources Using the Control Panel

You can schedule auto-indexing to keep your knowledge base up to date. Indexing jobs may incur costs when they process new data and create embeddings. If no changes are detected, the job completes with no changes and you are not billed.

To set up auto-indexing, on the right, click Schedule Indexing to open the Create Indexing Schedule window.

Under the Days section, select the days you want indexing to run.

Under the Trigger Time section, set the time of day using the Hrs and Mins dropdown lists. Scheduling time is in UTC.

Then, click Create Indexing Schedule.

Your schedule appears under the Auto-Indexing sub-section, showing its current schedule, the next scheduled run, the last indexing job (manual or scheduled), whether the last job is in progress, complete, or failed, and when the schedule was created.

Scheduled jobs are skipped if they overlap with manual or re-indexing jobs. Failed scheduled jobs don’t cancel the schedule.

You can view your indexing job logs in the Activity tab.

To manage your indexing schedule, on the right of your schedule, click …, and then you can either click:

• Pause Indexing to pause auto-indexing.
• Resume Indexing to resume auto-indexing.
• Edit Schedule to open the Update Indexing Schedule window where you can update the days and the time your auto-indexing runs.
• Destroy to open the Remove Scheduled Indexing window. In the textbox, enter “delete” to confirm deletion, and then click Destroy.

Schedule Indexing Data Sources via the API

To schedule indexing via the API, create an indexing schedule with the knowledge base ID, the time of day you want the job to run (in UTC, using 24-hour format), and the days of the week to schedule the runs, where days are numbered 1 (Monday) through 7 (Sunday). Use the Create Scheduled Indexing Job endpoint.

curl --location --request POST 'https://api.digitalocean.com/v2/gen-ai/scheduled-indexing' \
--header 'Authorization: Bearer $DIGITALOCEAN_TOKEN"' \
--header 'Content-Type: application/json' \
--data '{
    "knowledge_base_uuid": "knowledge_base_uuid",
    "time": "18:00",
    "days": [
        1,
        2,
        3
    ]
}'

After you set up auto-indexing, use the List Scheduled Indexing Jobs for a Knowledge Base endpoint to verify the schedule on your knowledge base.

If you no longer need the schedule, use the Delete Scheduled Indexing Job endpoint. Deleting a schedule doesn’t affect existing knowledge base data or previously completed indexing jobs.

Edit Knowledge Base Settings

You can edit a knowledge base’s name, project, tags, or reranking settings, or destroy it.

The following knowledge base details are view-only:

• Embeddings model shows the model in use and the token rate for indexing events.
• Associated agents lists any agents attached using the knowledge base. You can attach or detach any agent as needed.
• OpenSearch DB show the databases in use and its region. To manage databases, see the OpenSearch documentation.

To edit a knowledge base, go to the Control Panel, in the left menu, click INFERENCE, click the Agent Platform tab, and then click the Knowledge bases tab.

Then, find the knowledge base you want to edit, on the right of it, click …, and then click Manage Settings to open its Settings tab.

You can edit the following attributes:

• Knowledge base info, change the knowledge base name or select a different project.
• Tags, add or remove tags.
• Reranking, enable or disable reranking for the knowledge base.
• Destroy, destroy the knowledge base.

Click Edit on the right of the attribute you want to update, and then click Submit to apply your changes.

Destroy a Knowledge Base Using the Control Panel

When you destroy a knowledge base, all of its data sources and indexed data is irreversibly and permanently deleted. Any DigitalOcean AI agents using this knowledge base are automatically redeployed to remove it from their resources. This may affect their performance.

To destroy a knowledge base, go to the Control Panel, in the left menu, click INFERENCE, click Agent Platform, and then click the Knowledge Bases tab.

Find the knowledge base you want to destroy, on the right of it, click …, and then select Destroy.

In the confirmation window, type the knowledge base name to confirm deletion, then click Destroy to complete the deletion.

Once a knowledge base is destroyed, its indexing job logs are no longer available. If you need to keep activity history for record-keeping or debugging, download all relevant CSV files from the Activity tab before destroying the knowledge base.

The OpenSearch database storing your knowledge base isn’t destroyed along with it. To destroy your OpenSearch database, see How to Destroy OpenSearch Clusters.

Destroy a Knowledge Base Using the API

To destroy a knowledge base using the DigitalOcean API, provide its unique identifier. You can retrieve available knowledge bases and their IDs using the /v2/gen-ai/knowledge_bases endpoint.

curl -X DELETE \
  -H "Content-Type: application/json"  \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/knowledge_bases/8241f44e-b0da-11ef-bf8f-4e013e2ddde4"