Rootly Incidents

Rootly Incident Management

Overview

Incidents are the primary resource in Rootly — the central record for any production issue, outage, or service degradation. Rootly is used by SRE and platform engineering teams to coordinate response in real time and drive continuous improvement via postmortems. In MSP environments, Rootly typically manages internal infrastructure incidents rather than per-client ticketing, but can be configured with team-based routing to support multi-customer workflows.

The incident system supports:

• Automated Detection - Incidents created from monitoring alert integrations (Datadog, PagerDuty, Prometheus, Grafana, Sentry, etc.)
• Manual Creation - On-call engineers and responders create incidents via Slack, web UI, or API
• AI-Assisted Analysis - find_related_incidents and suggest_solutions surface past patterns and recommendations
• Real-Time Coordination - Slack channel auto-creation, Zoom bridges, timeline tracking
• Post-Incident Learning - Automated postmortem generation and action item tracking

All read and write operations on incidents are available through the Rootly MCP tools.

Anti-triggers

"Incident" names three different objects across this marketplace, and the routing test is what the operator does next: coordinate a response and write it up (this skill), approve a remediation on a compromised endpoint, or answer a customer under an SLA clock.

• A confirmed security incident — malware, intrusion, a host to contain. That object carries a SOC-recommended remediation to approve, not a response team to assemble; use huntress-incidents, or sentinelone-alerts for raw EDR detections.
• A customer ticket typed "Incident" — ITIL ticket classification inside a PSA or helpdesk, where the deliverable is an SLA-timed response and a billable time entry rather than a war room; use freshdesk-ticketing, halopsa-tickets, connectwise-psa-tickets, or autotask-tickets.
• A PagerDuty incident — a different vendor with a different lifecycle (triggered → acknowledged → resolved, not detected → in_triage → mitigated → resolved → closed); use pagerduty-incidents.
• The retrospective written after resolution — postmortem content and its action items are rootly-postmortems; this skill covers action items only as live response tasks.
• What fired and who it routed to — alert sources, routing rules, and escalation policies are rootly-alerts.
• A monitoring or uptime incident — an incident opened by a failing health check, whose deliverable is a status-page update rather than a response team, is betterstack-incidents.

MCP Tools

Core Incident Tools

AI Analysis Tools

Supporting Lookups

Discover Available Tools

Call list_endpoints to get the current list of all available API endpoints and tool names from the Rootly OpenAPI specification.

Key Concepts

Incident Lifecycle

┌───────────┐   Triage starts   ┌────────────┐   Contained   ┌───────────┐
│ detected  │ ─────────────────> │  in_triage │ ────────────> │ mitigated │
└───────────┘                   └────────────┘               └───────────┘
                                                                     │
                                                               Full fix done
                                                                     ▼
                                                               ┌──────────┐
                                                               │ resolved │
                                                               └──────────┘
                                                                     │
                                                             PIR complete
                                                                     ▼
                                                               ┌────────┐
                                                               │ closed │
                                                               └────────┘

Key lifecycle timestamps on the incident record:

Severity Levels

Severities are configurable per Rootly organization. Common conventions:

Note: Severity IDs are UUIDs in Rootly. Always call severities_get to map severity slugs to IDs before creating an incident.

Incident Status Values

Rootly uses free-text status descriptors alongside timestamps. Common values:

• detected — Newly created, not yet acknowledged
• in_triage — Actively being investigated
• mitigated — Impact contained, monitoring for recurrence
• resolved — Issue fixed, normal operation restored
• cancelled — False alarm or invalid incident

Field Reference

Common Workflows

Triage New Incidents

1. Call incidents_get with status=in_triage or status=detected, sorted by severity
2. For each high-severity incident, review title and summary
3. Call find_related_incidents with the incident_id to surface similar past incidents
4. Call suggest_solutions with the incident_id to get AI-generated remediation suggestions
5. Create action items via incidents_by_incident_id_action_items_post for each remediation step
6. Update status as the incident progresses

Create an Incident

1. Call severities_get to find the correct severity ID for the impact level
2. Call services_get to identify which service is affected
3. Call teams_get to find the on-call team to assign
4. Call incidents_post with title, severity_id, service_ids, team_ids
5. Attach any triggering alerts via incidents_by_incident_id_alerts_post

Investigate a Specific Incident

1. Find the incident via incidents_get or by sequential_id
2. Call find_related_incidents — look for recurring patterns (same service, same time of day, same symptoms)
3. Call suggest_solutions — surface past resolutions that worked for similar incidents
4. Review action items via incidents_by_incident_id_action_items_get
5. Review attached alerts via incidents_by_incident_id_alerts_get

Daily Incident Review

1. Call incidents_get with a 24-hour window and status=resolved
2. Count by severity for a daily health summary
3. Identify incidents that exceeded SLA targets (compare detected_at vs resolved_at)
4. Flag any incidents without postmortem action items for follow-up

Cross-Vendor PSA Ticket Correlation

Rootly incidents often correspond to PSA service tickets for MSP billing:

1. When an incident is created or resolved, create a matching ticket in your PSA (ConnectWise, HaloPSA, Autotask, etc.)
2. Include the Rootly sequential_id (e.g., INC-342) and url in the PSA ticket body
3. Map Rootly severity → PSA priority: SEV-1 → Critical, SEV-2 → High, SEV-3 → Medium, SEV-4 → Low
4. When the Rootly incident resolves, update the PSA ticket with resolution summary from summary
5. Use action items as sub-tasks in the PSA ticket for follow-up work

On-Call Handoff Summary

Before handing off to the next on-call engineer:

1. Call get_oncall_handoff_summary to see current and next on-call status plus open incidents
2. Call incidents_get with status=in_triage to review any actively open incidents
3. Add handoff notes as action items on open incidents
4. See the oncall skill for full on-call workflows

Error Handling

Common Errors

Authentication Error

401 Unauthorized

Verify your Rootly API token:
- Generate at Account > Manage API Keys
- Token type: Global (full access) or Team (limited to team resources)
- Pass as: Authorization: Bearer <token>

Best Practices

1. Use AI analysis first — Always call find_related_incidents and suggest_solutions before manual investigation
2. Track action items — Every resolved incident should have at least one follow-up action item
3. Attach alerts — Link the triggering alert to the incident for audit trail completeness
4. Scope to the right team — Include team_ids when creating incidents so on-call routing works correctly
5. Map severity consistently — Use severities_get to confirm slug-to-ID mapping rather than hardcoding
6. Use sequential IDs in external tools — Reference INC-{sequential_id} in PSA tickets and Slack
7. Check on-call health — Use check_oncall_health_risk before major deployments or planned maintenance

Related Skills

• On-Call Management — On-call schedules, handoffs, shift metrics
• API Patterns — Auth, pagination, all available tools