Product experiment design

ust carry assignment context, user identifiers, and timestamp precision. Idempotency keys prevent double-counting from retries or SDK misfires.

export interface ExperimentEvent {
  experimentId: string;
  variant: ExperimentVariant;
  userId: string;
  eventId: string; // UUID v4
  timestamp: number; // ISO 8601
  payload: Record<string, unknown>;
}

export class ExperimentTracker {
  private readonly endpoint: string;
  private readonly queue: ExperimentEvent[] = [];

  constructor(endpoint: string) {
    this.endpoint = endpoint;
  }

  track(event: ExperimentEvent): void {
    this.queue.push(event);
    if (this.queue.length >= 50) {
      this.flush();
    }
  }

  private async flush(): Promise<void> {
    const batch = this.queue.splice(0, 50);
    try {
      await fetch(this.endpoint, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(batch)
      });
    } catch (err) {
      // Requeue on failure; implement exponential backoff in production
      this.queue.unshift(...batch);
    }
  }
}

Architecture decision: Batched, asynchronous emission prevents blocking the critical path. The idempotency key (eventId) enables downstream deduplication. Payloads remain flexible but must never mutate after emission to preserve auditability.

Step 4: Decouple Analysis from Infrastructure

Analysis should never live inside the application runtime. Route events to a streaming warehouse (e.g., Kafka, Kinesis) or batch pipeline (e.g., Airflow, dbt), then apply statistical tests in a dedicated environment. Pre-register the analysis protocol: sample size calculation, primary metric definition, stopping rules, and correction method for multiple comparisons.

Architecture decision: Separating assignment, emission, and analysis eliminates coupling between product velocity and statistical rigor. It enables sequential testing, Bayesian updating, and retrospective validation without modifying application code.

Pitfall Guide

1. Peeking Before Sample Size: Checking results mid-flight inflates Type I error. A 5% alpha threshold becomes 20-30% with repeated checks. Use sequential testing boundaries or Bayesian credible intervals with pre-defined stopping rules.
2. Ignoring Multiple Comparisons: Running 10 experiments without correction guarantees at least one false positive. Apply Bonferroni, Holm-Bonferroni, or false discovery rate (FDR) control depending on hypothesis independence.
3. Inconsistent Traffic Allocation: Mixing user-level and session-level assignment creates overlapping groups and violates independence assumptions. Lock assignment level at experiment creation and enforce it in the routing layer.
4. Metric Contamination: Tracking secondary metrics as primary dilutes statistical power. Define one primary metric per experiment. Secondary metrics are for diagnostic insight, not decision gates.
5. Novelty and Habituation Effects: Initial engagement spikes or drops distort early results. Exclude the first 48-72 hours from analysis or model time-decay explicitly.
6. Infrastructure Latency Skew: Slow flag evaluation or event emission biases results toward faster clients or regions. Measure assignment latency and drop events that exceed P95 thresholds from analysis.
7. Poor Stratification: Failing to account for geography, device type, or acquisition channel introduces confounding variables. Stratify randomization or include covariates in the analysis model.

Best practices from production: Pre-register hypotheses and analysis protocols before launch. Maintain a 5-10% holdout group to detect long-term drift. Cache assignment results locally to avoid repeated hash computation. Validate power calculations with realistic effect size estimates, not optimistic projections. Treat experiment configuration as immutable infrastructure; never update weights or variants mid-flight.

Production Bundle

Action Checklist

• Pre-register hypothesis and primary metric: Document the expected effect size, target power (≥0.80), and analysis protocol before writing code.
• Implement deterministic assignment: Use salted hashing with consistent routing across client and server boundaries.
• Enforce idempotent event emission: Attach UUID v4 identifiers to every experiment event and enable downstream deduplication.
• Validate power and sample size: Run a priori power analysis using historical variance; do not launch without sufficient traffic.
• Configure holdout and stratification: Reserve a control holdout and stratify by key covariates (region, device, acquisition source).
• Decouple analysis pipeline: Route events to a warehouse; apply statistical tests in a dedicated environment, never in application runtime.
• Implement sequential stopping rules: Use alpha-spending functions or Bayesian thresholds to prevent peeking bias.

Decision Matrix

Configuration Template

{
  "experimentId": "exp_onboarding_v3",
  "version": 1,
  "status": "active",
  "assignmentLevel": "user",
  "holdoutPercent": 0.05,
  "variants": {
    "control": 0.50,
    "treatment_progress_bar": 0.25,
    "treatment_tooltips": 0.25
  },
  "primaryMetric": "completion_rate",
  "secondaryMetrics": ["time_to_complete", "drop_off_step_2"],
  "analysisProtocol": {
    "testType": "two_sample_proportion",
    "alpha": 0.05,
    "power": 0.80,
    "minimumDetectableEffect": 0.03,
    "stoppingRule": "sequential_alpha_spending",
    "multipleComparisonCorrection": "holm_bonferroni"
  },
  "routing": {
    "salt": "prod_exp_salt_2024_q3",
    "cacheTtlSeconds": 3600,
    "fallbackVariant": "control"
  }
}

Quick Start Guide

1. Define the experiment contract: Create a configuration object matching the template. Set assignment level, variant weights, primary metric, and analysis protocol. Commit it to version control.
2. Deploy the assignment engine: Integrate the hash-based AssignmentEngine into your routing layer. Cache results per user/session with a TTL matching your traffic pattern.
3. Instrument event emission: Wrap critical user actions with the ExperimentTracker. Attach experimentId, variant, and eventId to every payload. Route to your event pipeline.
4. Validate and launch: Run a dry-run with synthetic traffic to verify assignment distribution. Confirm event pipeline ingestion. Start the experiment and monitor assignment latency and event drop rates.
5. Analyze post-launch: Wait until pre-calculated sample size is reached. Apply the registered statistical test. Compare against holdout. Document results and archive the configuration.