ws for orthogonal test allocation.

// assignment-engine.ts
import { createHash } from 'crypto';

export interface ExperimentConfig {
  id: string;
  variants: string[];
  trafficAllocation: number; // 0.0 to 1.0
  namespace: string; // For orthogonal test isolation
}

export class ExperimentEngine {
  private experiments: Map<string, ExperimentConfig> = new Map();

  register(config: ExperimentConfig): void {
    this.experiments.set(config.id, config);
  }

  /**
   * Assigns a variant based on a consistent hash of the user ID and experiment config.
   * Uses MurmurHash3 for distribution quality and performance.
   */
  assignVariant(userId: string, experimentId: string): string | null {
    const config = this.experiments.get(experimentId);
    if (!config) return null;

    // Traffic filtering
    const hashKey = `${config.namespace}:${experimentId}:${userId}`;
    const hashValue = this.murmurHash3(hashKey);
    const normalizedHash = (hashValue % 10000) / 10000;

    if (normalizedHash >= config.trafficAllocation) {
      return 'control'; // Out of traffic
    }

    // Variant selection
    const variantIndex = Math.floor(normalizedHash * config.trafficAllocation * config.variants.length);
    return config.variants[variantIndex] || 'control';
  }

  private murmurHash3(key: string): number {
    // Simplified MurmurHash3 implementation for Node.js
    // In production, use a library like 'murmurhash-js' or native WASM implementation
    const buffer = Buffer.from(key);
    const hash = createHash('sha256').update(buffer).digest();
    return hash.readUInt32BE(0);
  }
}

2. Idempotent Tracking Pipeline

Client-side events must be reconciled with server-side actions to prevent double-counting and ensure conversion attribution is tied to the correct variant.

// tracking-pipeline.ts
export interface TrackingEvent {
  eventId: string; // UUID for idempotency
  userId: string;
  experimentId: string;
  variant: string;
  eventType: 'exposure' | 'conversion';
  timestamp: number;
  metadata?: Record<string, any>;
}

export class TrackingPipeline {
  private deduplicationStore: Map<string, boolean> = new Map();

  /**
   * Processes events with idempotency checks.
   * Prevents duplicate conversions from retries or client-side duplicates.
   */
  async processEvent(event: TrackingEvent): Promise<void> {
    const idempotencyKey = `${event.eventId}:${event.eventType}`;

    if (this.deduplicationStore.has(idempotencyKey)) {
      return; // Duplicate detected
    }

    // Validate user-variant consistency
    const assignedVariant = this.engine.assignVariant(event.userId, event.experimentId);
    if (assignedVariant !== event.variant) {
      // Log anomaly: User reporting variant different from assignment
      await this.logAnomaly(event, assignedVariant);
      return;
    }

    // Write to analytics warehouse
    await this.writeToWarehouse(event);
    
    // Mark as processed (TTL based on retention policy)
    this.deduplicationStore.set(idempotencyKey, true);
  }

  private async logAnomaly(event: TrackingEvent, expected: string | null): Promise<void> {
    // Alerting mechanism for SRM or assignment drift
    console.error(`Variant Mismatch: User ${event.userId} reported ${event.variant}, expected ${expected}`);
  }
}

3. Server-Side Feature Flag Integration

For high-impact changes, variants should be resolved server-side to eliminate flicker and ensure SEO compatibility.

// server-renderer.ts
export async function renderPage(req: Request, res: Response) {
  const userId = req.session?.userId || req.headers['x-anonymous-id'] as string;
  
  // Parallel assignment for multiple experiments
  const assignments = await Promise.all([
    engine.assignVariant(userId, 'checkout-flow-v2'),
    engine.assignVariant(userId, 'pricing-table-redesign'),
  ]);

  const [checkoutVariant, pricingVariant] = assignments;

  // Inject assignments into SSR context
  const html = await renderTemplate(req, {
    checkoutVariant,
    pricingVariant,
    // ... other context
  });

  res.send(html);
}

Pitfall Guide

Engineering teams implementing CRO infrastructure must avoid these critical pitfalls derived from production failures.

1. Sample Ratio Mismatch (SRM):

   • Issue: The observed ratio of users in variants deviates significantly from the configured allocation (e.g., 50/50 config results in 60/40 traffic).
   • Cause: Bugs in assignment logic, race conditions, or differential bot filtering.
   • Impact: Invalidates statistical significance; results are unreliable.
   • Mitigation: Implement automated SRM checks using Chi-squared tests before releasing results. Block analysis if SRM p-value < 0.05.

2. Peeking Problem:

   • Issue: Stopping a test as soon as statistical significance is reached during interim analysis.
   • Cause: Misunderstanding of frequentist statistics; p-values are not constant over time.
   • Impact: Inflated false positive rates up to 40%.
   • Mitigation: Pre-calculate sample size requirements. Use Bayesian methods or sequential testing corrections (e.g., Alpha Spending) if interim looks are required.

3. Flicker Effect:

   • Issue: Users briefly see the control variant before the variant is applied via client-side JS.
   • Cause: Asynchronous script loading and DOM manipulation after paint.
   • Impact: Increased bounce rate, poor UX, and skewed metrics for time-sensitive interactions.
   • Mitigation: Use server-side rendering or Edge Compute for variant resolution. If client-side is necessary, hide the affected DOM elements until assignment resolves.

4. Test Interference:

   • Issue: Running multiple tests on the same funnel simultaneously without isolation.
   • Cause: Lack of namespace management; users are exposed to multiple conflicting variants.
   • Impact: Interaction effects mask true lift; attribution becomes impossible.
   • Mitigation: Implement orthogonal allocation using namespaces. Ensure tests in the same namespace do not overlap, or use multi-armed bandit approaches carefully.

5. Bot Contamination:

   • Issue: Automated traffic skews conversion rates and assignment ratios.
   • Cause: Bots trigger events but do not convert, or convert via automated scripts.
   • Impact: Artificial inflation or deflation of metrics.
   • Mitigation: Filter traffic at the edge using bot detection headers, behavioral analysis, and CAPTCHA challenges. Exclude identified bot traffic from analysis datasets.

6. Privacy Compliance Drift:

   • Issue: Experimentation code violates GDPR/CCPA by tracking PII or ignoring consent signals.
   • Cause: Hardcoded tracking IDs; lack of consent management integration.
   • Impact: Legal liability, data loss, and reputational damage.
   • Mitigation: Integrate with Consent Management Platforms (CMP). Anonymize user IDs in analytics pipelines. Ensure data retention policies are enforced programmatically.

7. Guardrail Metric Neglect:

   • Issue: Optimizing for conversion rate degrades other critical metrics like latency or error rate.
   • Cause: Single-metric focus in test evaluation.
   • Impact: Technical debt accumulation; user experience degradation.
   • Mitigation: Define guardrail metrics (e.g., P95 latency, error rate, support ticket volume) in the experiment config. Halt tests if guardrails are breached.

Production Bundle

Action Checklist

• Implement SRM Detection: Add Chi-squared test validation to the analysis pipeline; block results if p-value < 0.05.
• Establish Orthogonal Namespaces: Configure experiment namespaces to prevent interference; map traffic allocations per namespace.
• Deploy Edge Assignment: Move variant resolution to the Edge layer to eliminate flicker and reduce latency to <5ms.
• Configure Guardrails: Define and monitor guardrail metrics for every experiment; set automated alerts for breaches.
• Integrate Consent Signals: Ensure tracking pipeline respects user consent; suppress events for non-consented users.
• Validate Idempotency: Implement event deduplication using UUIDs and server-side reconciliation to prevent double-counting.
• Pre-calculate Sample Size: Use power analysis to determine required sample size before launching; document minimum detectable effect.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High Traffic E-commerce	Edge-Compute Hybrid	Minimizes latency impact on conversion; handles scale efficiently; ensures SEO compatibility.	Medium (Edge compute costs + Dev effort)
Regulated Industry (Health/Finance)	Server-Side + Strict Audit	Full control over data flow; easier compliance auditing; eliminates client-side data leakage risks.	High (Server resources + Compliance overhead)
Low Traffic SaaS	Bayesian Analysis + Server-Side	Bayesian methods provide faster insights with lower sample sizes; server-side ensures data fidelity.	Low-Medium
Static Site / Jamstack	Client-Side with Pre-rendering	No server infrastructure available; pre-rendering variants can mitigate flicker; lightweight.	Low
Mobile App	Server-Driven Config	Consistent assignment across sessions; instant updates without app store releases; reduces bundle size.	Medium (Backend config service)

Configuration Template

Use this TypeScript interface and JSON structure to define experiments programmatically.

// experiment-definition.ts
export interface ExperimentDefinition {
  id: string;
  name: string;
  namespace: string;
  trafficAllocation: number;
  variants: {
    key: string;
    weight: number;
    description: string;
  }[];
  metrics: {
    primary: string;
    guardrails: string[];
    secondary: string[];
  };
  targeting?: {
    userAttributes?: Record<string, any>;
    geo?: string[];
    minVersion?: string;
  };
  duration?: {
    startDate: string;
    endDate: string;
    minDurationDays: number;
  };
}

// Example Configuration
const checkoutExperiment: ExperimentDefinition = {
  id: "checkout-v2-test",
  name: "Checkout Flow Redesign",
  namespace: "checkout",
  trafficAllocation: 0.1,
  variants: [
    { key: "control", weight: 0.5, description: "Existing flow" },
    { key: "variant-a", weight: 0.5, description: "Single-page checkout" }
  ],
  metrics: {
    primary: "purchase_completion_rate",
    guardrails: ["p95_latency_ms", "api_error_rate"],
    secondary: ["cart_abandonment_rate", "time_to_checkout"]
  },
  targeting: {
    userAttributes: { plan: ["pro", "enterprise"] },
    minVersion: "2.4.0"
  },
  duration: {
    startDate: "2024-01-01T00:00:00Z",
    endDate: "2024-03-01T00:00:00Z",
    minDurationDays: 14
  }
};

Quick Start Guide

1. Initialize the Engine: Import the ExperimentEngine, register your experiments using the configuration template, and deploy to your Edge runtime or Node.js server.
2. Integrate Assignment: Replace hardcoded feature flags with engine.assignVariant(userId, experimentId) calls in your rendering logic. Ensure the variant is passed to your analytics layer.
3. Instrument Tracking: Wrap conversion events in the TrackingPipeline. Include the experimentId, variant, and eventId. Verify idempotency by testing duplicate event submissions.
4. Validate SRM: After traffic accumulates, run the SRM check. If the test passes, proceed to analysis. If it fails, debug assignment logic and traffic routing immediately.
5. Monitor Guardrails: Set up dashboards for guardrail metrics. Configure alerts to notify engineering if latency or error rates exceed thresholds during the experiment.

Technical CRO is not an afterthought; it is a discipline of precision engineering. By implementing deterministic assignment, idempotent tracking, and rigorous statistical validation, engineering teams enable sustainable growth while maintaining system integrity and user trust.