impact on connection limits.

• Critical for preventing connection pool exhaustion. */ static simulateConnectionBurst( currentConnections: number, maxConnections: number, avgRequestDurationMs: number, expectedRPS: number ): { projectedConnections: number; burstRisk: boolean } { // Little's Law: L = λ * W // L = average number of items in a queuing system // λ = average arrival rate // W = average time an item spends in the system const projectedConnections = (expectedRPS * avgRequestDurationMs) / 1000; const burstRisk = projectedConnections > (maxConnections * 0.8);

return {

  projectedConnections: Math.ceil(projectedConnections),
  burstRisk
};

} }

// Usage Example const storageMetric: CapacityMetric = { name: 'Primary Disk', current: 4500, // GB limit: 6000, // GB growthRatePerHour: 0.5, // GB/hour safetyMargin: 0.15 // 15% headroom required };

const forecast = CapacityModeler.forecast(storageMetric); console.log(forecast); // Output: { daysRemaining: 135, status: 'healthy', ... }

### Architecture Decisions and Rationale

*   **Predictive vs. Reactive Scaling:** Implement predictive scaling for storage and compute. Reactive scaling is acceptable only for ephemeral workloads with strict cost caps. Predictive scaling requires integration with your CI/CD pipeline to automate infrastructure changes based on forecast data.
*   **Sharding Strategy:** When single-node capacity limits are reached, sharding is inevitable. Plan for sharding early by designing keys that distribute load evenly. Avoid range-based sharding on monotonically increasing keys (e.g., timestamps) to prevent hot spots.
*   **Connection Pooling:** Database connection limits are often the first bottleneck. Implement aggressive connection pooling (e.g., PgBouncer, ProxySQL) at the application layer. This reduces the connection footprint on the database and allows higher concurrency with fewer resources.
*   **Storage Tiering:** Implement automated tiering for time-series or log data. Move cold data to cheaper storage tiers or archive tables to keep the hot dataset within memory/buffer pool limits.

## Pitfall Guide

### Common Mistakes

1.  **Ignoring IOPS Throughput Limits:** Cloud databases often have IOPS limits tied to storage size or instance class. You may have free disk space but hit IOPS caps, causing severe latency. Always model IOPS consumption alongside storage volume.
2.  **Assuming Linear Growth for Exponential Data:** User-generated content or log volumes often grow exponentially. Linear models will underestimate capacity needs, leading to sudden breaches. Use exponential smoothing for metrics with accelerating growth rates.
3.  **Neglecting Index Bloat:** In MVCC databases like PostgreSQL, indexes can bloat significantly over time, consuming disk space and degrading read performance without a corresponding increase in row count. Capacity plans must account for vacuum/maintenance overhead.
4.  **Overlooking Backup and Restore Capacity:** Backup operations consume IOPS and storage. If your capacity plan does not reserve headroom for backup windows, backup jobs may throttle production traffic or fail due to disk full errors.
5.  **Single-Metric Monitoring:** Focusing only on CPU or Memory ignores bottlenecks like network bandwidth, replication lag, or disk queue depth. A holistic capacity model tracks all resource vectors.
6.  **Static Threshold Alerts:** Setting alerts at fixed percentages (e.g., 80% CPU) fails to account for rate of change. A system at 80% CPU growing at 1% per hour is stable; a system at 80% growing at 5% per hour is minutes from failure. Use rate-of-change alerts.
7.  **Testing with Synthetic Load:** Capacity tests using uniform load patterns often miss real-world burstiness. Production traffic has spikes from marketing events, batch jobs, or cache evictions. Load tests must replicate production traffic distributions.

### Best Practices

*   **Chaos Engineering for Capacity:** Regularly inject capacity constraints in staging environments to validate alerting and scaling logic. Simulate disk full scenarios to ensure graceful degradation.
*   **Capacity Review Cadence:** Schedule monthly capacity reviews. Analyze forecast accuracy against actual growth and adjust growth rate models.
*   **Document Capacity SLOs:** Define Service Level Objectives for capacity, such as "Storage must not breach 85% utilization" or "P99 latency must remain under 50ms at 110% projected load."
*   **Automate Remediation:** Where safe, automate scaling actions. For example, automatically provision additional read replicas when replication lag exceeds thresholds during predicted peak hours.

## Production Bundle

### Action Checklist

- [ ] **Inventory Resource Vectors:** List all critical metrics (CPU, Memory, Disk, IOPS, Connections, Network) for each database instance.
- [ ] **Instrument Predictive Metrics:** Configure monitoring to calculate `predict_linear` or equivalent time-series forecasts for all resource vectors.
- [ ] **Define Safety Margins:** Set safety margins per metric based on risk tolerance (e.g., 15% for storage, 20% for connections).
- [ ] **Implement TTT Alerts:** Replace absolute threshold alerts with Time-to-Threshold alerts (e.g., alert if storage breach in < 7 days).
- [ ] **Run Load Validation:** Execute load tests that simulate peak production traffic plus 20% growth to verify capacity limits.
- [ ] **Review Connection Limits:** Audit application connection pools and database `max_connections` settings. Ensure pooling is active.
- [ ] **Document Scaling Runbooks:** Create runbooks for vertical scaling, sharding, and emergency capacity expansion.
- [ ] **Schedule Monthly Reviews:** Establish a recurring meeting to review capacity forecasts and approve scaling actions.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Steady, Predictable Growth | Predictive Vertical Scaling | Simplest path; minimizes architectural complexity. | Moderate; costs rise with instance size. |
| Spiky, Unpredictable Traffic | Read Replicas + Auto-scaling | Handles bursts without over-provisioning primary. | Higher; replica costs incurred continuously. |
| Massive Data Volume (>10TB) | Sharding or Distributed DB | Single node limits reached; horizontal scale required. | High; operational complexity and infra costs increase. |
| Cost-Sensitive Workload | Serverless Database | Pay-per-use model optimizes for variable load. | Variable; can be expensive at sustained high load. |
| High Availability Requirement | Multi-AZ with Standby | Ensures failover capacity is always available. | High; standby resources are reserved but idle. |

### Configuration Template

**Prometheus Alerting Rule for Predictive Capacity**

This rule uses `predict_linear` to forecast storage exhaustion and triggers an alert if the threshold is breached within 7 days.

```yaml
groups:
  - name: database_capacity
    rules:
      # Alert if storage will be full in less than 7 days
      - alert: DatabaseStorageExhaustionPredictive
        expr: |
          (
            predict_linear(
              node_filesystem_avail_bytes{mountpoint="/data"}[7d],
              7 * 24 * 3600
            )
          ) < 0
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Database storage exhaustion predicted within 7 days"
          description: "Instance {{ $labels.instance }} is projected to run out of storage on {{ $labels.mountpoint }} within 7 days based on 7-day trend. Current available: {{ $value }} bytes."

      # Alert if IOPS throughput is approaching limit
      - alert: DatabaseIOPSApproachingLimit
        expr: |
          rate(node_disk_io_time_seconds_total{device="nvme0n1"}[5m]) 
          > 0.85 * max_over_time(rate(node_disk_io_time_seconds_total{device="nvme0n1"}[5m])[30d:1h])
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "IOPS throughput approaching historical max"
          description: "IOPS rate on {{ $labels.instance }} is exceeding 85% of the 30-day maximum. Risk of IOPS throttling."

Quick Start Guide

1. Deploy Exporter: Install the database-specific exporter (e.g., postgres_exporter, mysqld_exporter) on your database instances. Ensure metrics are exposed on port 9187.
2. Scrape Configuration: Update your Prometheus scrape_configs to target the exporter endpoints.

   scrape_configs:
     - job_name: 'postgres'
       static_configs:
         - targets: ['db-primary:9187']
   

3. Load Recording Rules: Apply the PromQL recording rules to calculate growth rates and predictions. This reduces query load and standardizes metrics.
4. Verify Dashboard: Import a capacity planning dashboard. Confirm that days_remaining panels are populating correctly and reflect current growth trends.
5. Test Alerting: Simulate a capacity breach by temporarily adjusting the predict_linear window or threshold in a non-production environment to verify alert routing to Slack/PagerDuty.