On-Call Health

Q: On-call for security CVE?

Separate from app rotation if patch cadence differs; link cargo audit runbook.

Sustainable on-call keeps Rust specialists effective: rotations short enough to recover, alerts that mean action, and runbooks that reduce 3am improvisation. Burned-out on-call engineers ship more incidents, not fewer.

Recipe

Quick-reference recipe card - copy-paste ready.

# On-call health checklist (quarterly)
 
## Rotation
- [ ] Primary shift <= 7 days (or follow-the-sun)
- [ ] Secondary defined for every primary
- [ ] Handoff doc template used every Monday
 
## Alert hygiene
- [ ] < 5 pages/week per service (median)
- [ ] Every paging alert has runbook link
- [ ] Flapping alerts fixed or silenced with ticket
 
## Wellbeing
- [ ] Pages after incident reviewed for toil reduction
- [ ] Comp policy applied when > 2 pages/night
- [ ] No "hero" culture for preventable alerts

When to reach for this:

Setting up first production on-call for Rust service
Alert volume doubled after new deploy
Engineer burnout or attrition on platform team
SRE review of paging policy

Working Example

//! alert_context.rs - attach runbook URL to exported metrics for Grafana annotations.
use prometheus::{IntCounterVec, Opts};
 
pub fn panic_counter() -> IntCounterVec {
    IntCounterVec::new(
        Opts::new("rust_panic_total", "Panics caught by hook"),
        &["service", "runbook"],
    )
    .expect("metric descriptor valid")
}
 
pub fn record_panic(service: &str) {
    let counter = panic_counter();
    counter
        .with_label_values(&[service, "https://wiki.example.com/runbooks/panic-storm"])
        .inc();
}

# Weekly on-call report (pages per service)
alertmanager silences --filter 'service=orders-api' --since 7d
# Target: investigate any service > 10 pages/week

Annotate metrics with runbook URLs so dashboards remind responders where to start, reducing time-to-mitigate and stress.

Deep Dive

Rotation Models

Model	Pros	Cons
Weekly primary	Simple schedule	Fatigue if noisy
Follow-the-sun	Sleep protected	Handoff discipline required
Secondary only nights	Lighter load	Primary still bears context

Alert Tiers

Page - Customer SLO at risk; wake someone.
Ticket - Fix business hours; threshold sustained 30+ min.
Log - Informational trend; dashboard only.

Rust services: prefer paging on error rate, panic rate, and readiness failures over raw CPU unless CPU correlates with SLO miss.

Reducing Toil

Automate rollback buttons in runbook (scripted kubectl).
Fix flaky synthetics that false-page.
Batch low-urgency dependency updates to office hours.
Post-mortem every page that required >30 min manual toil without runbook step.

Gotchas

Alert on every pod restart - Noise during deploys. Fix: Exclude rollout windows or use SLO burn rate.
No secondary - Vacation = blind spot. Fix: Always schedule secondary with escalation policy.
Runbook stale - Worse than none (false confidence). Fix: Quarterly drill updates last_verified date.
Same person always on-call - Bus factor and burnout. Fix: Rotate includes juniors with shadow weeks.
Ignoring worker alerts - API on-call surprised by queue backlog. Fix: Unified service paging includes workers.

Alternatives

Alternative	Use When	Don't Use When
Managed on-call vendor	Small team, 24/7 requirement	Strong in-house SRE culture
Error budget pause on deploys	Chronic CFR	Emergency security patch
Dedicated SRE rotation	Large fleet	Two-service startup
Office-hours only on-call	Internal tools, no SLA	Customer payment path

FAQs

How many services per rotation?

1-3 tier-1 services per primary; more requires secondary split or SRE pool.

Junior on-call alone?

Shadow 2+ weeks; secondary is senior; explicit escalation in first month.

Comp for rough week?

Policy: comp day if 2+ night pages or SEV1 > 4 hours; track equitably.

Rust-specific noisy alerts?

Tune Tokio runtime metrics separately from app SLO; avoid paging idle pool stats.

Handoff doc contents?

Open incidents, deploys this week, known flaky alerts, debt tickets affecting stability.

Measure on-call health?

Pages per person, MTTR trend, post-incident toil tags, voluntary survey quarterly.

When remove alert?

When no action taken in 3 consecutive pages; replace with dashboard if still useful.

Cross-team paging?

Service owner pages first; platform escalation path documented with SLA.

Holiday coverage?

Plan 4 weeks ahead; no single engineer covering holiday + regular week alone.

On-call for security CVE?

Separate from app rotation if patch cadence differs; link cargo audit runbook.

Incident Response Playbooks - response process
Turning Incidents into Guardrails - reduce repeat pages
DORA Metrics - MTTR tracking
Production Debugging - faster resolution
Delegation & Spreading Context - bus factor

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Recipe

Quick-reference recipe card - copy-paste ready.

# On-call health checklist (quarterly)
 
## Rotation
- [ ] Primary shift <= 7 days (or follow-the-sun)
- [ ] Secondary defined for every primary
- [ ] Handoff doc template used every Monday
 
## Alert hygiene
- [ ] < 5 pages/week per service (median)
- [ ] Every paging alert has runbook link
- [ ] Flapping alerts fixed or silenced with ticket
 
## Wellbeing
- [ ] Pages after incident reviewed for toil reduction
- [ ] Comp policy applied when > 2 pages/night
- [ ] No "hero" culture for preventable alerts

When to reach for this:

Setting up first production on-call for Rust service
Alert volume doubled after new deploy
Engineer burnout or attrition on platform team
SRE review of paging policy

Working Example

//! alert_context.rs - attach runbook URL to exported metrics for Grafana annotations.
use prometheus::{IntCounterVec, Opts};
 
pub fn panic_counter() -> IntCounterVec {
    IntCounterVec::new(
        Opts::new("rust_panic_total", "Panics caught by hook"),
        &["service", "runbook"],
    )
    .expect("metric descriptor valid")
}
 
pub fn record_panic(service: &str) {
    let counter = panic_counter();
    counter
        .with_label_values(&[service, "https://wiki.example.com/runbooks/panic-storm"])
        .inc();
}

# Weekly on-call report (pages per service)
alertmanager silences --filter 'service=orders-api' --since 7d
# Target: investigate any service > 10 pages/week

Annotate metrics with runbook URLs so dashboards remind responders where to start, reducing time-to-mitigate and stress.

Deep Dive

Rotation Models

Model	Pros	Cons
Weekly primary	Simple schedule	Fatigue if noisy
Follow-the-sun	Sleep protected	Handoff discipline required
Secondary only nights	Lighter load	Primary still bears context

Alert Tiers

Page - Customer SLO at risk; wake someone.
Ticket - Fix business hours; threshold sustained 30+ min.
Log - Informational trend; dashboard only.

Rust services: prefer paging on error rate, panic rate, and readiness failures over raw CPU unless CPU correlates with SLO miss.

Reducing Toil

Automate rollback buttons in runbook (scripted kubectl).
Fix flaky synthetics that false-page.
Batch low-urgency dependency updates to office hours.
Post-mortem every page that required >30 min manual toil without runbook step.

Gotchas

Alert on every pod restart - Noise during deploys. Fix: Exclude rollout windows or use SLO burn rate.
No secondary - Vacation = blind spot. Fix: Always schedule secondary with escalation policy.
Runbook stale - Worse than none (false confidence). Fix: Quarterly drill updates last_verified date.
Same person always on-call - Bus factor and burnout. Fix: Rotate includes juniors with shadow weeks.
Ignoring worker alerts - API on-call surprised by queue backlog. Fix: Unified service paging includes workers.

Alternatives

Alternative	Use When	Don't Use When
Managed on-call vendor	Small team, 24/7 requirement	Strong in-house SRE culture
Error budget pause on deploys	Chronic CFR	Emergency security patch
Dedicated SRE rotation	Large fleet	Two-service startup
Office-hours only on-call	Internal tools, no SLA	Customer payment path

FAQs

How many services per rotation?

1-3 tier-1 services per primary; more requires secondary split or SRE pool.

Junior on-call alone?

Shadow 2+ weeks; secondary is senior; explicit escalation in first month.

Comp for rough week?

Policy: comp day if 2+ night pages or SEV1 > 4 hours; track equitably.

Rust-specific noisy alerts?

Tune Tokio runtime metrics separately from app SLO; avoid paging idle pool stats.

Handoff doc contents?

Open incidents, deploys this week, known flaky alerts, debt tickets affecting stability.

Measure on-call health?

Pages per person, MTTR trend, post-incident toil tags, voluntary survey quarterly.

When remove alert?

When no action taken in 3 consecutive pages; replace with dashboard if still useful.

Cross-team paging?

Service owner pages first; platform escalation path documented with SLA.

Holiday coverage?

Plan 4 weeks ahead; no single engineer covering holiday + regular week alone.

On-call for security CVE?

Separate from app rotation if patch cadence differs; link cargo audit runbook.

Incident Response Playbooks - response process
Turning Incidents into Guardrails - reduce repeat pages
DORA Metrics - MTTR tracking
Production Debugging - faster resolution
Delegation & Spreading Context - bus factor

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

On-Call Health

Buscar en la documentación

On-Call Health