Incident Management: Building Effective On-Call and Response Practices

# Prometheus alerting rules
groups:
  - name: payment-service
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(payment_requests_total{status=~"5.."}[5m])) 
          / sum(rate(payment_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on payment service"
          description: "Error rate is {{ $value | humanizePercentage }}"
          
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, 
            rate(payment_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency on payment service"

# On-call rotation configuration
oncall_config = {
    "primary": {
        "engineer": "[email protected]",
        "phone": "+1234567890",
        "hours": "Monday 9am - Monday 9am"
    },
    "secondary": {
        "engineer": "[email protected]",
        "hours": "Always available for escalation"
    },
    "rotation": {
        "type": "weekly",
        "start": "Monday 9am UTC",
        "handoff_meeting": "Friday 3pm"
    }
}

# Escalation tiers
escalation_policy:
  - level: 1
    name: "On-Call Engineer"
    response_time: "15 minutes"
    contacts:
      - phone: oncall_phone
      - slack: "#oncall"
      
  - level: 2
    name: "Team Lead"
    response_time: "30 minutes"
    contacts:
      - phone: team_lead_phone
      - slack: "#engineering"
      
  - level: 3
    name: "Engineering Manager"
    response_time: "1 hour"
    contacts:
      - phone: manager_phone
      - slack: "#leadership"

  - level: 4
    name: "VP Engineering"
    response_time: "2 hours"
    contacts:
      - phone: vp_phone

# Runbook: Payment Service High Error Rate

## Symptoms
- Alert: HighErrorRate on payment-service
- Error rate > 5%

## Impact
- Users cannot complete payments
- Revenue impact immediately

## Steps

### 1. Check Service Health
```bash
kubectl get pods -n payments
kubectl describe deployment payment-service

kubectl rollout history deployment payment-service
git log --oneline -10 --since="1 hour ago"

kubectl logs -n payments -l app=payment-service --tail=100

kubectl rollout undo deployment/payment-service

kubectl exec -it payment-db-0 -- psql -c "SELECT * FROM pg_stat_activity"

# Incident Communication Template

## Initial Alert (within 15 min)
**Subject:** [INCIDENT] Payment Service Outage - Investigating

**Status:** 🔴 Investigating

**Impact:** Users unable to complete purchases

**What we're doing:** Looking into elevated error rates

**Next update:** In 30 minutes

---

## Update (every 30 min)

**Subject:** [INCIDENT] Payment Service Outage - Update #2

**Status:** 🟡 Identified

**Root cause:** Database connection pool exhausted due to connection leak

**What we're doing:** 
- Rolling back to previous version
- Clearing stuck connections

**ETA:** 15 minutes

**Next update:** In 15 minutes

---

## Resolution

**Subject:** [INCIDENT] Payment Service - Resolved

**Status:** 🟢 Resolved

**Summary:** Deployed fix for connection leak, service recovering

**Root cause:** Connection leak in payment processor library

**Action items:**
- [ ] Monitor connection pool metrics
- [ ] Add circuit breaker
- [ ] Schedule post-mortem

**Learnings:** Post-mortem to follow within 48 hours

# Post-Mortem: Payment Service Outage
**Date:** March 12, 2026
**Duration:** 45 minutes
**Severity:** SEV-1

## Summary
Payment service experienced complete outage for 45 minutes affecting all transactions.

## Timeline (UTC)
- 10:15 - Alert fired: High error rate
- 10:17 - On-call acknowledged
- 10:22 - Root cause identified: Connection pool exhausted
- 10:35 - Rollback initiated
- 10:45 - Service recovered
- 11:00 - All systems operational

## Root Cause
Connection leak in payment processor library introduced in v2.3.0. 
Connections were not being returned to pool after successful transactions.

## Impact
- 2,347 failed transactions
- ~$50,000 in estimated lost revenue
- 45 minutes of degraded service

## What Went Well
- Alert fired quickly (< 2 min from issue start)
- Team responded promptly
- Rollback worked correctly

## What Could Be Improved
- Add connection pool monitoring before deployment
- Better staging environment parity
- Faster escalation path for DB issues

## Action Items
| Task | Owner | Due |
|------|-------|-----|
| Add connection_pool_available SLI | @alice | Mar 15 |
| Add circuit breaker to payment calls | @bob | Mar 20 |
| Improve staging DB load testing | @carol | Apr 1 |
| Review all library upgrades | @team | Ongoing |

# Alert effectiveness metrics
alert_metrics = {
    "total_alerts": 100,
    "actionable_alerts": 85,
    "false_positives": 5,
    "noise_rate": "5%",
    "time_to_acknowledge_avg": "3 min",
    "time_to_resolve_avg": "15 min"
}