--- title: Triage Incidents From an Alert (public) description: Start a Harness Runtime session from an alerting webhook, let the agent query your own observability MCP server, and deliver a triage summary to Slack … product: Managed Agents url: https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/examples/triage-incidents/ last_updated: "2026-09-22" --- > **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document). # Triage Incidents From an Alert (public) DigitalOcean Harness Runtime provides managed, hardware-isolated microVM sandboxes with built-in tools such as Chromium to run harnesses and execute arbitrary code. Rich lifecycle APIs preserve conversational history and working state across sessions, letting you pause, resume, and fork work to control costs and adapt to the nonlinear nature of agentic workflows. Scale complete agents such as Claude Code or use sandboxes independently for code execution, all through the same service. See [What You Can Build](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/details/what-you-can-build/index.html.md) for example use cases. The expensive part of an incident is the first ten minutes, when someone wakes up and starts gathering context that the systems already have. This example runs that gathering step automatically: an alert fires, an agent queries your logs and metrics, and a summary lands in Slack with the alert. It introduces the one integration path the other examples skip, an MCP server you host yourself, and it is the case where the egress rule that catches everyone actually bites. ## Attach Your Own MCP Server Action Gateway covers a large catalog, but your observability stack, internal runbook service, or deployment tracker may only be reachable through an MCP server you run. The `tools` list accepts those inline: give it a name, an `https://` URL, and a credential drawn from a `secrets` slot. **Warning**: Attaching an inline MCP server does not allowlist its host. If you set `egress` and leave the server’s host out of it, the connection fails, usually as a timeout rather than a clear error. List the host in both places. ## Environment Spec Save the following as `incident-triage.yaml`: ```yaml name: incident-triage agent: codex size: mars-2vcpu-4gb idle_timeout: 15m env: HARNESS_INFERENCE_MODEL: openai-gpt-5.3-codex secrets: HARNESS_INFERENCE_API_KEY: ${HARNESS_INFERENCE_API_KEY} OBSERVABILITY_TOKEN: ${OBSERVABILITY_TOKEN} egress: - mcp.internal.example.com tools: - name: observability url: https://mcp.internal.example.com/mcp tools: [search_logs, get_service_health, list_recent_deploys] auth: header: Authorization secret: OBSERVABILITY_TOKEN permissions: default: allow rules: - tool: bash match: { command: "rm -rf *" } action: deny ``` The host appears twice on purpose: once under `egress` so the sandbox can open the connection, and once as the `url` so the agent knows where the server is. Codex CLI reaches its model through [DigitalOcean Serverless Inference](https://docs.digitalocean.com/products/inference/how-to/use-serverless-inference/index.html.md) with the [model access key](https://docs.digitalocean.com/products/inference/how-to/manage-model-access-keys/index.html.md) in `HARNESS_INFERENCE_API_KEY`, and that endpoint is one that the platform allowlists for you. Your own MCP host is the exception, which is the whole point of this example. There are no permission rules naming the observability tools, and adding them would not do what it looks like. An inline MCP server is reached directly from the sandbox over TLS, so the platform never sees the individual calls and cannot enforce a rule against them. What actually scopes this agent’s access is the `tools` list, which decides the three tools the server exposes to it, and the `egress` entry, which decides whether it can reach the host at all. Permission rules bite on tools the platform mediates, which means the adapter’s own tools and anything routed through Action Gateway. `OBSERVABILITY_TOKEN` is declared under `secrets` rather than `env`. The guest environment is debug-readable, and `auth.secret` requires a declared secret slot that is visible to the environment. A slot marked `env: false` is rejected here. There are no `repos` and no GitHub credential. This agent reads telemetry and writes a summary; it has no reason to touch source. ## Create the Trigger Point your alerting system at a webhook trigger. `--provider custom` is plain HMAC, which suits most alerting tools: ```shell doctl harness-runtime triggers create \ --kind webhook \ --name incident-triage \ --session-mode fresh \ --spec incident-triage.yaml \ --provider custom \ --prompt "An alert fired: {{payload}}. Identify the affected service, check its health, search the last hour of logs for errors, and list deploys in the last six hours. Report the most likely cause and the evidence for it. If the evidence is weak, say so rather than guessing." \ --output-mode slack \ --output-slack-webhook ``` `{{payload}}` embeds the whole alert body. To pull one field instead, use dotted paths such as `{{.alert.service}}`. Copy the webhook URL and secret from the response and configure them in your alerting tool. Deliveries are signature-verified and de-duplicated, so a sender that retries does not trigger two triage runs for one alert. The last sentence of the prompt is load-bearing. An agent asked to find a cause finds one, so telling it that weak evidence is a valid answer is what keeps the summary useful at 3 AM. ## Adapt It | To do this | Change this | |---|---| | Use a tool that Action Gateway already covers | Replace the inline entry with `- do.actions: [...]`. Catalog tools use platform-managed networking, so they need no `egress` entry. | | Attach several internal servers | Add one `tools` entry per server, each with its own name, credential, and `egress` host. One entry per server; declaring the same server twice is a duplicate-server error. | | Keep context across related alerts | Pause a session and bind it with `--session-mode reuse`, so the agent keeps what it learned on the previous alert. | | Let the agent act, not only report | Attach a deployment or runbook tool and allow that specific server-qualified target. Keep everything else denied. | | Test without an alert | Launch the spec interactively and paste a sample payload into the prompt. | ## Related - [Connect Action Gateway to Harness Runtime](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/how-to/connect-action-gateway/index.html.md) for the catalog alternative to a self-hosted server. - [Network Egress](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/concepts/egress/index.html.md) for why an attached host still needs an allowlist entry. - [Run Agents With Triggers](https://docs.digitalocean.com/products/managed-agents/agent-harness-runtime/how-to/run-agents-with-triggers/index.html.md) for providers, payload placeholders, and execution history.