For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

The expensive part of an incident is the first ten minutes, when someone wakes up and starts gathering context that the systems already have. This example runs that gathering step automatically: an alert fires, an agent queries your logs and metrics, and a summary lands in Slack with the alert.

It introduces the one integration path the other examples skip, an MCP server you host yourself, and it is the case where the egress rule that catches everyone actually bites.

Attach Your Own MCP Server

Action Gateway covers a large catalog, but your observability stack, internal runbook service, or deployment tracker may only be reachable through an MCP server you run. The tools list accepts those inline: give it a name, an https:// URL, and a credential drawn from a secrets slot.

Environment Spec

Save the following as incident-triage.yaml:

name: incident-triage
agent: codex
size: mars-2vcpu-4gb
idle_timeout: 15m
env:
  HARNESS_INFERENCE_MODEL: openai-gpt-5.3-codex
secrets:
  HARNESS_INFERENCE_API_KEY: ${HARNESS_INFERENCE_API_KEY}
  OBSERVABILITY_TOKEN: ${OBSERVABILITY_TOKEN}
egress:
  - mcp.internal.example.com
tools:
  - name: observability
    url: https://mcp.internal.example.com/mcp
    tools: [search_logs, get_service_health, list_recent_deploys]
    auth:
      header: Authorization
      secret: OBSERVABILITY_TOKEN
permissions:
  default: allow
  rules:
    - tool: bash
      match: { command: "rm -rf *" }
      action: deny

The host appears twice on purpose: once under egress so the sandbox can open the connection, and once as the url so the agent knows where the server is. Codex CLI reaches its model through DigitalOcean Serverless Inference with the model access key in HARNESS_INFERENCE_API_KEY, and that endpoint is one that the platform allowlists for you. Your own MCP host is the exception, which is the whole point of this example.

There are no permission rules naming the observability tools, and adding them would not do what it looks like. An inline MCP server is reached directly from the sandbox over TLS, so the platform never sees the individual calls and cannot enforce a rule against them. What actually scopes this agent’s access is the tools list, which decides the three tools the server exposes to it, and the egress entry, which decides whether it can reach the host at all. Permission rules bite on tools the platform mediates, which means the adapter’s own tools and anything routed through Action Gateway.

OBSERVABILITY_TOKEN is declared under secrets rather than env. The guest environment is debug-readable, and auth.secret requires a declared secret slot that is visible to the environment. A slot marked env: false is rejected here.

There are no repos and no GitHub credential. This agent reads telemetry and writes a summary; it has no reason to touch source.

Create the Trigger

Point your alerting system at a webhook trigger. --provider custom is plain HMAC, which suits most alerting tools:

doctl harness-runtime triggers create \
  --kind webhook \
  --name incident-triage \
  --session-mode fresh \
  --spec incident-triage.yaml \
  --provider custom \
  --prompt "An alert fired: {{payload}}. Identify the affected service, check its health, search the last hour of logs for errors, and list deploys in the last six hours. Report the most likely cause and the evidence for it. If the evidence is weak, say so rather than guessing." \
  --output-mode slack \
  --output-slack-webhook <your-slack-incoming-webhook-url>

{{payload}} embeds the whole alert body. To pull one field instead, use dotted paths such as {{.alert.service}}.

Copy the webhook URL and secret from the response and configure them in your alerting tool. Deliveries are signature-verified and de-duplicated, so a sender that retries does not trigger two triage runs for one alert.

The last sentence of the prompt is load-bearing. An agent asked to find a cause finds one, so telling it that weak evidence is a valid answer is what keeps the summary useful at 3 AM.

Adapt It

Related

• Connect Action Gateway to Harness Runtime for the catalog alternative to a self-hosted server.
• Network Egress for why an attached host still needs an allowlist entry.
• Run Agents With Triggers for providers, payload placeholders, and execution history.