Lessons-Learned Catalog
A fleet catalog of recurring Rust pitfalls and proven fixes - panics in handlers, Mutex across await, FFI ABI mismatches, pool misconfiguration, and deploy skew. New hires read the top ten; post-mortems add entries with guardrail links.
Recipe
Quick-reference recipe card - copy-paste ready.
## Lesson LL-0042: sqlx pool typo in Helm
**Tags:** deploy, oom, latency
**Symptom:** Rising p95, low CPU, acquire timeouts
**Root cause:** `DATABASE_MAX_CONNECTIONS=2` in prod values typo
**Fix:** Helm schema validation + pool_acquire alert
**Guardrail:** PR #8821, Grafana rule 445
**Refs:** PM-2026-07-09When to reach for this:
- Onboarding week one reading list
- Pre-release checklist review
- Pattern matching during incident
- Quarterly reliability guild
Working Example
| ID | Pattern | Quick fix |
|---|---|---|
| LL-0001 | unwrap in Axum handler | Map to AppError; clippy deny |
| LL-0002 | std::fs in async route | spawn_blocking |
| LL-0003 | API rolled back, workers not | Coupled deploy runbook |
| LL-0004 | #[repr(C)] struct drift | Layout test + cbindgen |
| LL-0005 | Per-request format! hot path | Buffer reuse; Criterion guard |
| LL-0006 | Missing GIT_SHA in logs | Env at startup + span field |
| LL-0007 | WASM size bloat | wasm-opt CI gate |
| LL-0008 | Feature flag default on | Default off policy |
| LL-0009 | cargo.lock skew API/worker | Single workspace release |
| LL-0010 | Miri skipped on unsafe PR | Nightly miri job on FFI crates |
Deep Dive
Maintaining the Catalog
- Owner: reliability guild or platform team.
- Source: SEV2+ post-mortems, repeated support themes, game days.
- Review quarterly: merge duplicates, archive fixed-class issues.
Using in Process
- PR template: "Does this touch LL-0003 class?" checkbox.
- Design review appendix links relevant lessons.
- Interview panel uses catalog for scenario questions.
Tags for Search
panic, async, ffi, oom, deploy, sqlx, tokio, wasm, supply-chain
Gotchas
- Catalog in wiki only - Drifts from code. Fix:
docs/lessons-learned.mdin main repo or generated index. - Blameful entries - Fix: Systems focus, no names.
- Stale guardrail links - Fix: CI check links in quarterly audit.
- 1000 entries - Fix: Top 20 curated + searchable archive.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| ADR-only history | Small team | Need symptom-based lookup |
| Incident tool tags only | Strong IM process | Onboarding narrative weak |
| Runbook per service | Tier-1 isolation | Fleet patterns repeat |
FAQs
Who can add entries?
Any engineer via PR; guild maintainer merges.
Customer-specific lessons?
Redact names; keep technical pattern.
Link from alerts?
Runbook URLs may include lesson ID anchor.
Version by edition?
Tag lessons with edition/toolchain if relevant.
Overlap with post-mortem?
Post-mortem is incident-specific; lesson is generalized pattern.
Related
- Turning Incidents into Guardrails - prevention
- Common Production Failure Modes - symptoms
- Blameless Post-Mortems - source material
- Before/After Cases - deep dives
- Governance Best Practices - fleet consistency
Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.