Mean Time to Resolve Incidents

Goal of the analysis:

Measure the average time it takes to restore and fully resolve incidents from detection or creation to closure. Executives use this metric to gauge operational effectiveness, customer impact, and SLA compliance. Shorter resolution times reduce business disruption, minimize financial and reputational risk, and indicate mature incident response and problem management. This analysis also clarifies where time is spent (queueing, diagnosis, repair, vendor waits) and guides investment in process, tooling, and staffing.

Data required:

• Incident management records:
  • Ticket IDs, priority/severity, category/subcategory, configuration item (CI)/service, environment, region.
  • Timestamps: detection time (from monitoring), ticket creation, first response/acknowledgment, assignment, escalation, service restored/mitigated, resolved, closed, and any reopen timestamps.
  • Status history with state changes (New, In Progress, On Hold, With Vendor, Resolved, Closed).
  • Resolution code, workaround used, root cause link (Problem ticket), and major incident flag.
• SLA/SLO and business calendars:
  • Contracted SLAs and internal SLOs by severity/tier; SLA “clock” rules (pauses for vendor waits, customer pending).
  • Business hours and holiday calendars per region/timezone for business-hours calculations.
• Monitoring and alerting data:
  • Alert timestamps (trigger, clear), correlated events, and affected services (Datadog, New Relic, CloudWatch, Azure Monitor, Zabbix).
  • Noise suppression/duplication indicators to avoid counting alert storms as multiple incidents.
• Change and deployment data:
  • Change requests, deployment windows, release identifiers to correlate incidents with changes.
• Workforce and on-call context:
  • On-call schedules, team rosters, skill matrices, escalation policies (PagerDuty, Opsgenie).
• User impact and demand signals (optional):
  • Transaction volume, active users, revenue at risk, CSAT/NPS during incident windows for impact-weighted views.

Detailed step-by-step instruction on how to conduct the analysis:

1. Define scope and definitions:
   • Scope the period (e.g., last 3–6 months) and incident types (production only; exclude service requests).
   • Clarify definitions: “Time to Restore Service” (detect/creation → service restored) vs. “Time to Resolve” (detect/creation → resolved/closed). Decide which is the primary KPI.
   • Agree on SLA rules: whether to pause clocks for On Hold/Vendor states; decide on 24×7 vs. business-hours calculation.
2. Extract data:
   • From ITSM (ServiceNow, Jira Service Management, Remedy): export ticket fields and full state-change history.
   • From monitoring: export first alert time and clear time for correlation; de-duplicate alert storms.
   • From change tools: export deployments to tag incidents as change-related.
3. Normalize and cleanse:
   • Standardize timezones; ensure timestamps are ordered and non-null.
   • Remove or flag duplicates/parent-child related tickets; treat major incident bridges as a single incident.
   • For reopened incidents, either measure to final closure or count a separate cycle; document the rule.
4. Derive stage durations:
   • Compute MTTA (Mean Time to Acknowledge) = first response − detection/creation.
   • Time to Assign = first assignment − creation; Time in Queue = sum of time in waiting states.
   • Time to Restore = service restored − detection/creation.
   • Time to Resolve = resolved/closed − detection/creation (per agreed definition).
   • If SLA pauses apply, compute both “calendar time” and “SLA time.”
5. Choose robust aggregations:
   • Report mean, median (p50), and p90/p95 to handle long tails; avoid relying on average alone.
   • For business-hours view, exclude non-business time per regional calendar.
6. Segment the results:
   • By severity/priority (P1–P4), service tier, application, CI, team, resolver group, channel (monitoring-detected vs. user-reported), region, and time of day.
   • By cause: change-related vs. non-change; vendor-caused vs. internal.
7. Trend and benchmark:
   • Build monthly time series for p50/p90 restore and resolve times and SLA attainment (% met).
   • Compare to internal SLOs/SLAs; quantify minutes/hours above target and error budget burn if used.
8. Identify bottlenecks:
   • Decompose average incident journey into stages (acknowledge, triage, diagnose, repair, validate, close).
   • Run Pareto of total delay by stage and by category (e.g., network, database, end-user device).
   • Heatmap resolution time by hour/day to surface after-hours gaps.
9. Correlate with changes and workload:
   • Overlay deployments to quantify change-failure impact on resolution time.
   • Analyze by incident volume cohorts: high-volume periods typically lengthen times due to queueing.
10. Validate and communicate:
    • Spot-check a sample of long-duration tickets for data quality (incorrect states, missing restore timestamps).
    • Publish a standardized dashboard and an executive readout with key insights and recommended actions.

Format of the output of analysis:

• Executive summary with p50/p90 Time to Restore and Time to Resolve by severity, SLA attainment, and top bottlenecks.
• Stage-time stacked bar showing average minutes spent in each phase (queue, triage, diagnosis, repair, vendor wait).
• Box/violin plots of resolution time by team/service to highlight variability and outliers.
• Time series of monthly medians with deployment markers and major incidents annotated.
• Heatmap of resolution time by hour-of-day and day-of-week.
• Pareto chart of categories contributing to total delay and a table of top 10 longest incidents with RCA summaries.

How to interpret results:

• Lower times (especially p90) indicate effective detection, triage, and remediation. Confirm that “restore” and “resolve” are both strong; restoring quickly but resolving slowly may accumulate risk.
• High values or rising trends suggest bottlenecks: long queue times imply staffing/triage gaps; long diagnosis points to observability or knowledge base issues; long vendor waits require supplier management.
• Severity differences matter: P1/P2 should be markedly faster than P3/P4. If not, revisit prioritization and escalation policies.
• Large variance across teams/services signals uneven capabilities or complexity; target coaching or architectural fixes where variability is highest.
• Benchmark gaps versus SLAs/SLOs indicate potential penalties or customer dissatisfaction; quantify business impact (e.g., hours of degraded service times average revenue/hour).
• Watch for data artifacts: extreme outliers due to tickets left open, or “Resolved” without “Restored.” Use medians and audit trails to validate.

Steps a company can take to improve on this measure:

• Process and policy:
  • Implement major incident management with clear roles, swarming, and timeboxed decision points.
  • Tighten prioritization and escalation matrices; auto-escalate P1/P2 after defined thresholds.
  • Standardize triage playbooks and closure criteria; enforce accurate “service restored” timestamps.
• Tooling and automation:
  • Adopt automated alert correlation, noise suppression, and intelligent routing to the right resolver group.
  • Enable runbooks, self-healing scripts, and chatops for common remediations.
  • Integrate ITSM with monitoring, CI/CD, and CMDB to accelerate diagnosis and change rollback.
• Observability and knowledge:
  • Enhance logs, metrics, traces, and SLO-based alerting to cut diagnosis time.
  • Build and curate a searchable knowledge base with known errors, workarounds, and RCAs.
• People and coverage:
  • Ensure 24×7 on-call coverage for critical services; add follow-the-sun support where needed.
  • Upskill teams on troubleshooting, major incident facilitation, and systems architecture.
• Change and problem management:
  • Use gradual rollouts, canaries, and feature flags; pre-approved standard changes for low risk.
  • Drive post-incident RCAs with action tracking; prioritize permanent fixes that reduce recurrent incidents.
• Scenario tips:
  • If MTTA is low but resolve time is high: improve diagnostics and provide better runbooks/observability.
  • If restore time is good but resolve time is poor: strengthen problem management and backlog prioritization.
  • If vendor waits dominate: renegotiate SLAs, add secondary providers, or redesign to reduce dependency.
  • If after-hours incidents take longer: add on-call coverage or implement automated mitigations.

Benchmark comparisons:

General benchmarks:

• P1 (critical) incidents: Time to Restore median 0.5–2 hours; p90 4–8 hours. Full resolution often within 24–72 hours depending on root cause complexity.
• P2 (high): Restore median 2–6 hours; full resolution within 1–3 days.
• P3 (moderate): Resolution typically 2–5 business days; P4 (low) 5–10 business days.
• High-performing digital/SaaS teams often target P1 mitigation in <15–30 minutes via automation and robust on-call.

Segment- or industry-specific benchmarks:

• Customer-facing revenue services aim for fastest mitigation; internal end-user support tolerates longer windows aligned to business hours.
• Regulated and high-availability environments (payments, telco) frequently enforce strict restore targets and continuous coverage.

If external benchmarks are insufficient, build internal references: set severity-based SLOs, track p50/p90 monthly, compare across services and resolver groups, and target the internal top quartile. Use error budgets and RCA-driven improvements to ratchet performance annually.