Lessons-Learned Catalog

A fleet catalog of recurring Rust pitfalls and proven fixes - panics in handlers, Mutex across await, FFI ABI mismatches, pool misconfiguration, and deploy skew. New hires read the top ten; post-mortems add entries with guardrail links.

Recipe

Quick-reference recipe card - copy-paste ready.

## Lesson LL-0042: sqlx pool typo in Helm
 
**Tags:** deploy, oom, latency
**Symptom:** Rising p95, low CPU, acquire timeouts
**Root cause:** `DATABASE_MAX_CONNECTIONS=2` in prod values typo
**Fix:** Helm schema validation + pool_acquire alert
**Guardrail:** PR #8821, Grafana rule 445
**Refs:** PM-2026-07-09

When to reach for this:

Onboarding week one reading list
Pre-release checklist review
Pattern matching during incident
Quarterly reliability guild

Working Example

ID	Pattern	Quick fix
LL-0001	`unwrap` in Axum handler	Map to `AppError`; clippy deny
LL-0002	`std::fs` in async route	`spawn_blocking`
LL-0003	API rolled back, workers not	Coupled deploy runbook
LL-0004	`#[repr(C)]` struct drift	Layout test + cbindgen
LL-0005	Per-request `format!` hot path	Buffer reuse; Criterion guard
LL-0006	Missing `GIT_SHA` in logs	Env at startup + span field
LL-0007	WASM size bloat	`wasm-opt` CI gate
LL-0008	Feature flag default on	Default off policy
LL-0009	`cargo.lock` skew API/worker	Single workspace release
LL-0010	Miri skipped on `unsafe` PR	Nightly miri job on FFI crates

Deep Dive

Maintaining the Catalog

Owner: reliability guild or platform team.
Source: SEV2+ post-mortems, repeated support themes, game days.
Review quarterly: merge duplicates, archive fixed-class issues.

Using in Process

PR template: "Does this touch LL-0003 class?" checkbox.
Design review appendix links relevant lessons.
Interview panel uses catalog for scenario questions.

Tags for Search

panic, async, ffi, oom, deploy, sqlx, tokio, wasm, supply-chain

Gotchas

Catalog in wiki only - Drifts from code. Fix: docs/lessons-learned.md in main repo or generated index.
Blameful entries - Fix: Systems focus, no names.
Stale guardrail links - Fix: CI check links in quarterly audit.
1000 entries - Fix: Top 20 curated + searchable archive.

Alternatives

Alternative	Use When	Don't Use When
ADR-only history	Small team	Need symptom-based lookup
Incident tool tags only	Strong IM process	Onboarding narrative weak
Runbook per service	Tier-1 isolation	Fleet patterns repeat

FAQs

Who can add entries?

Any engineer via PR; guild maintainer merges.

Customer-specific lessons?

Redact names; keep technical pattern.

Link from alerts?

Runbook URLs may include lesson ID anchor.

Version by edition?

Tag lessons with edition/toolchain if relevant.

Overlap with post-mortem?

Post-mortem is incident-specific; lesson is generalized pattern.

Turning Incidents into Guardrails - prevention
Common Production Failure Modes - symptoms
Blameless Post-Mortems - source material
Before/After Cases - deep dives
Governance Best Practices - fleet consistency

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Recipe

Quick-reference recipe card - copy-paste ready.

## Lesson LL-0042: sqlx pool typo in Helm
 
**Tags:** deploy, oom, latency
**Symptom:** Rising p95, low CPU, acquire timeouts
**Root cause:** `DATABASE_MAX_CONNECTIONS=2` in prod values typo
**Fix:** Helm schema validation + pool_acquire alert
**Guardrail:** PR #8821, Grafana rule 445
**Refs:** PM-2026-07-09

When to reach for this:

Onboarding week one reading list
Pre-release checklist review
Pattern matching during incident
Quarterly reliability guild

Working Example

ID	Pattern	Quick fix
LL-0001	`unwrap` in Axum handler	Map to `AppError`; clippy deny
LL-0002	`std::fs` in async route	`spawn_blocking`
LL-0003	API rolled back, workers not	Coupled deploy runbook
LL-0004	`#[repr(C)]` struct drift	Layout test + cbindgen
LL-0005	Per-request `format!` hot path	Buffer reuse; Criterion guard
LL-0006	Missing `GIT_SHA` in logs	Env at startup + span field
LL-0007	WASM size bloat	`wasm-opt` CI gate
LL-0008	Feature flag default on	Default off policy
LL-0009	`cargo.lock` skew API/worker	Single workspace release
LL-0010	Miri skipped on `unsafe` PR	Nightly miri job on FFI crates

Deep Dive

Maintaining the Catalog

Owner: reliability guild or platform team.
Source: SEV2+ post-mortems, repeated support themes, game days.
Review quarterly: merge duplicates, archive fixed-class issues.

Using in Process

PR template: "Does this touch LL-0003 class?" checkbox.
Design review appendix links relevant lessons.
Interview panel uses catalog for scenario questions.

Tags for Search

panic, async, ffi, oom, deploy, sqlx, tokio, wasm, supply-chain

Gotchas

Catalog in wiki only - Drifts from code. Fix: docs/lessons-learned.md in main repo or generated index.
Blameful entries - Fix: Systems focus, no names.
Stale guardrail links - Fix: CI check links in quarterly audit.
1000 entries - Fix: Top 20 curated + searchable archive.

Alternatives

Alternative	Use When	Don't Use When
ADR-only history	Small team	Need symptom-based lookup
Incident tool tags only	Strong IM process	Onboarding narrative weak
Runbook per service	Tier-1 isolation	Fleet patterns repeat

FAQs

Who can add entries?

Any engineer via PR; guild maintainer merges.

Customer-specific lessons?

Redact names; keep technical pattern.

Link from alerts?

Runbook URLs may include lesson ID anchor.

Version by edition?

Tag lessons with edition/toolchain if relevant.

Overlap with post-mortem?

Post-mortem is incident-specific; lesson is generalized pattern.

Turning Incidents into Guardrails - prevention
Common Production Failure Modes - symptoms
Blameless Post-Mortems - source material
Before/After Cases - deep dives
Governance Best Practices - fleet consistency

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Lessons-Learned Catalog

Buscar en la documentación

Lessons-Learned Catalog