Process & Resource Management
Debug running Rust services with ps, htop, signals, and perf. Validate graceful shutdown, thread counts, and memory growth under load.
Recipe
pgrep -a my-service
kill -TERM "$(pgrep my-service)"
sleep 2
pgrep my-service || echo "exited cleanly"
htop -p "$(pgrep -d',' my-service)"When to reach for this:
- Service won't stop on deploy
- CPU pegged after traffic spike
- Investigating memory leak suspicions
Working Example
# Watch open files for connection leak
PID=$(pgrep my-service)
ls -l /proc/$PID/fd | wc -l
# Sample stack with perf (Linux)
sudo perf record -g -p $PID -- sleep 30
sudo perf report
# Memory map summary
pmap -x $PID | tail -1// Ensure SIGTERM handled in service
use tokio::signal;
async fn shutdown_signal() {
signal::ctrl_c().await.ok();
signal::unix::signal(signal::unix::SignalKind::terminate())
.unwrap()
.recv()
.await
.ok();
}What this demonstrates:
- TERM before KILL matches Kubernetes pod termination
/proc/$PID/fdcounts open sockets/filesperfattributes CPU to symbols (with debug symbols)
Deep Dive
Signals
| Signal | Use |
|---|---|
| SIGTERM (15) | Graceful shutdown request |
| SIGINT (2) | Ctrl-C local dev |
| SIGKILL (9) | Force kill, no cleanup |
| SIGHUP (1) | Reload config (if implemented) |
Tokio Runtime Threads
Default multi-thread runtime uses ~N worker threads; blocking pool adds more under spawn_blocking load.
Gotchas
- SIGKILL in tests - hides shutdown bugs. Fix: integration test SIGTERM path.
- perf without permissions -
perf_event_paranoid. Fix: sysctl adjust orsudo. - Debug symbols stripped - useless perf stacks. Fix: retain symbols in staging builds.
- Reading RSS only - allocator caches memory. Fix: track over time and load patterns.
- Wrong PID after reload - systemd restarts. Fix:
systemctl statusfor current PID.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
tokio-console | Async task introspection | Production always-on |
heaptrack / dhat | Allocator profiling | Quick CPU check |
bpftrace | Kernel-level tracing | Simple local dev |
FAQs
How many threads is normal?
Tokio worker threads plus a few blocking threads; investigate hundreds.
perf on macOS?
Use samply or Instruments instead of Linux perf.
FD limit in Docker?
Set ulimits in compose or K8s security context for connection-heavy Axum services.
Related
Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.