A/B Testing & Experimentation

A/B tests measure whether a change improves outcomes - conversion, latency, revenue - with statistical discipline. Rust services need server-side assignment, guardrail metrics, and variant tags in tracing so experiments do not become undebuggable flag spaghetti.

Recipe

Quick-reference recipe card - copy-paste ready.

use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
 
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Variant { Control, Treatment }
 
pub fn assign(experiment: &str, unit_id: &str, pct_treatment: u8) -> Variant {
    let mut h = DefaultHasher::new();
    experiment.hash(&mut h);
    unit_id.hash(&mut h);
    let bucket = (h.finish() % 100) as u8;
    if bucket < pct_treatment { Variant::Treatment } else { Variant::Control }
}

When to reach for this:

• Product asks "does new checkout flow convert better?"
• Algorithm tweak with measurable business metric
• UX change with latency guardrail
• Pricing or ranking experiments

Working Example

use axum::{Json, Router, routing::post};
use serde::{Deserialize, Serialize};
use tracing::instrument;
 
#[derive(Deserialize)]
struct CheckoutRequest { tenant_id: String, order_id: String }
 
#[derive(Serialize)]
struct CheckoutResponse {
    variant: &'static str,
    order_id: String,
}
 
#[instrument(skip(req), fields(experiment = "checkout_v2", variant))]
async fn checkout(Json(req): Json<CheckoutRequest>) -> Json<CheckoutResponse> {
    let variant = match assign("checkout_v2", &req.tenant_id, 10) {
        Variant::Control => "control",
        Variant::Treatment => "treatment",
    };
    tracing::Span::current().record("variant", variant);
    // branch logic ...
    Json(CheckoutResponse { variant, order_id: req.order_id })
}

# Guardrail: abort experiment if p95 latency > baseline + 20% for 15m
# Wire to experiment controller or manual IC decision

What this demonstrates:

• Deterministic sticky assignment per tenant_id
• Variant recorded in span for log/metric correlation
• Treatment percentage explicit (10% ramp)

Deep Dive

Experiment Lifecycle

1. Hypothesis - "Treatment reduces checkout p95 by 10ms without increasing errors."
2. Pre-register - Primary metric, guardrails, duration, minimum sample.
3. Ramp - 5% -> 10% -> 50% with guardrail checks.
4. Analyze - Statistical test or Bayesian CI; avoid peeking without correction.
5. Ship or revert - Merge winner; delete experiment code path.

Guardrails

• Error rate, p95 latency, revenue per minute floors/ceilings.
• Auto-disable treatment via flag when guardrail burns.
• Ethics: no dark patterns; document in product review.

Rust Observability

Export experiment and variant labels on Prometheus metrics. Use structured logs for funnel analysis export to warehouse.

Gotchas

• Non-sticky assignment - Same user sees both variants. Fix: Hash stable unit id.
• Peeking and stopping early - False positives. Fix: Pre-set sample size or sequential test method.
• Simpson's paradox - Aggregate wins, segments lose. Fix: Slice by region/plan.
• Experiment without flag off path - Cannot abort quickly. Fix: Treatment behind flag.
• Variant in response body only - Analytics miss server-side failures. Fix: Server-side event log.

Alternatives

FAQs

Related

• Feature Flags & Progressive Delivery - ramp and kill
• DORA Metrics - delivery vs experiment cadence
• Tracing - variant in spans
• Stakeholder Communication - report results
• Product Collaboration Best Practices - product alignment

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.