On-Call Health
Sustainable on-call keeps Rust specialists effective: rotations short enough to recover, alerts that mean action, and runbooks that reduce 3am improvisation. Burned-out on-call engineers ship more incidents, not fewer.
Recipe
Quick-reference recipe card - copy-paste ready.
# On-call health checklist (quarterly)
## Rotation
- [ ] Primary shift <= 7 days (or follow-the-sun)
- [ ] Secondary defined for every primary
- [ ] Handoff doc template used every Monday
## Alert hygiene
- [ ] < 5 pages/week per service (median)
- [ ] Every paging alert has runbook link
- [ ] Flapping alerts fixed or silenced with ticket
## Wellbeing
- [ ] Pages after incident reviewed for toil reduction
- [ ] Comp policy applied when > 2 pages/night
- [ ] No "hero" culture for preventable alertsWhen to reach for this:
- Setting up first production on-call for Rust service
- Alert volume doubled after new deploy
- Engineer burnout or attrition on platform team
- SRE review of paging policy
Working Example
//! alert_context.rs - attach runbook URL to exported metrics for Grafana annotations.
use prometheus::{IntCounterVec, Opts};
pub fn panic_counter() -> IntCounterVec {
IntCounterVec::new(
Opts::new("rust_panic_total", "Panics caught by hook"),
&["service", "runbook"],
)
.expect("metric descriptor valid")
}
pub fn record_panic(service: &str) {
let counter = panic_counter();
counter
.with_label_values(&[service, "https://wiki.example.com/runbooks/panic-storm"])
.inc();
}# Weekly on-call report (pages per service)
alertmanager silences --filter 'service=orders-api' --since 7d
# Target: investigate any service > 10 pages/weekAnnotate metrics with runbook URLs so dashboards remind responders where to start, reducing time-to-mitigate and stress.
Deep Dive
Rotation Models
| Model | Pros | Cons |
|---|---|---|
| Weekly primary | Simple schedule | Fatigue if noisy |
| Follow-the-sun | Sleep protected | Handoff discipline required |
| Secondary only nights | Lighter load | Primary still bears context |
Alert Tiers
- Page - Customer SLO at risk; wake someone.
- Ticket - Fix business hours; threshold sustained 30+ min.
- Log - Informational trend; dashboard only.
Rust services: prefer paging on error rate, panic rate, and readiness failures over raw CPU unless CPU correlates with SLO miss.
Reducing Toil
- Automate rollback buttons in runbook (scripted
kubectl). - Fix flaky synthetics that false-page.
- Batch low-urgency dependency updates to office hours.
- Post-mortem every page that required >30 min manual toil without runbook step.
Gotchas
- Alert on every pod restart - Noise during deploys. Fix: Exclude rollout windows or use SLO burn rate.
- No secondary - Vacation = blind spot. Fix: Always schedule secondary with escalation policy.
- Runbook stale - Worse than none (false confidence). Fix: Quarterly drill updates
last_verifieddate. - Same person always on-call - Bus factor and burnout. Fix: Rotate includes juniors with shadow weeks.
- Ignoring worker alerts - API on-call surprised by queue backlog. Fix: Unified service paging includes workers.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Managed on-call vendor | Small team, 24/7 requirement | Strong in-house SRE culture |
| Error budget pause on deploys | Chronic CFR | Emergency security patch |
| Dedicated SRE rotation | Large fleet | Two-service startup |
| Office-hours only on-call | Internal tools, no SLA | Customer payment path |
FAQs
How many services per rotation?
1-3 tier-1 services per primary; more requires secondary split or SRE pool.
Junior on-call alone?
Shadow 2+ weeks; secondary is senior; explicit escalation in first month.
Comp for rough week?
Policy: comp day if 2+ night pages or SEV1 > 4 hours; track equitably.
Rust-specific noisy alerts?
Tune Tokio runtime metrics separately from app SLO; avoid paging idle pool stats.
Handoff doc contents?
Open incidents, deploys this week, known flaky alerts, debt tickets affecting stability.
Measure on-call health?
Pages per person, MTTR trend, post-incident toil tags, voluntary survey quarterly.
When remove alert?
When no action taken in 3 consecutive pages; replace with dashboard if still useful.
Cross-team paging?
Service owner pages first; platform escalation path documented with SLA.
Holiday coverage?
Plan 4 weeks ahead; no single engineer covering holiday + regular week alone.
On-call for security CVE?
Separate from app rotation if patch cadence differs; link cargo audit runbook.
Related
- Incident Response Playbooks - response process
- Turning Incidents into Guardrails - reduce repeat pages
- DORA Metrics - MTTR tracking
- Production Debugging - faster resolution
- Delegation & Spreading Context - bus factor
Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.