Secrets management patterns

rUrl: string, maxRetries = 3) { this.sidecarUrl = sidecarUrl; this.maxRetries = maxRetries; }

async fetchSecret(role: string): Promise<SecretPayload> { const cached = this.cache.get(role); const now = Date.now();

if (cached && now < cached.expiresAt) {
  if (now > cached.warnAt) {
    this.triggerAsyncRefresh(role);
  }
  return cached.value;
}

return this.refreshSecret(role);

}

private async refreshSecret(role: string): Promise<SecretPayload> { let lastError: Error | null = null;

for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
  try {
    const response = await fetch(`${this.sidecarUrl}/v1/database/creds/${role}`, {
      headers: { 'Content-Type': 'application/json' },
      signal: AbortSignal.timeout(5000),
    });

    if (!response.ok) {
      const errText = await response.text();
      throw new Error(`Secret fetch failed [${response.status}]: ${errText}`);
    }

    const data = await response.json();
    const payload: SecretPayload = {
      username: data.data.username,
      password: data.data.password,
      lease_id: data.lease_id,
      lease_duration: data.lease_duration,
      rotation_warning: Math.floor(data.lease_duration * 0.75),
    };

    this.cache.set(role, {
      value: payload,
      expiresAt: Date.now() + (payload.lease_duration * 1000),
      warnAt: Date.now() + (payload.rotation_warning * 1000),
    });

    return payload;
  } catch (err) {
    lastError = err as Error;
    const delay = Math.min(100 * Math.pow(2, attempt - 1), 2000);
    await sleep(delay);
  }
}

throw new Error(`Secret refresh failed after ${this.maxRetries} attempts: ${lastError?.message}`);

}

private triggerAsyncRefresh(role: string): void { this.refreshSecret(role).catch(err => { console.error([SecretClient] Async refresh failed for ${role}:, err.message); }); } }

#### Step 3: Integration Pattern
Mount a shared volume or Unix socket between the sidecar and application container. Configure the sidecar to render secrets to `/tmp/secrets/db-creds` or expose via HTTP. The TypeScript client reads from the local endpoint, never contacting the secret engine directly. Database connection pools should be configured to validate connections on checkout and handle `ECONNREFUSED` or authentication failures by triggering a client-side refresh.

## Pitfall Guide

### 1. Caching Beyond Lease Expiration
**Mistake**: Storing secrets in application memory or Redis without strict TTL enforcement.
**Why it fails**: Dynamic credentials expire server-side. Cached values become invalid, causing connection pool exhaustion or silent authentication failures.
**Best practice**: Tie cache expiration to `lease_duration - safety_margin`. Implement client-side renewal before expiry, not after.

### 2. Logging Secret Payloads or Metadata
**Mistake**: Including `username`, `password`, `lease_id`, or rotation timestamps in structured logs or error traces.
**Why it fails**: Log aggregation systems are rarely encrypted at rest or access-controlled with the same rigor as secret engines. Leaked logs become credential dumps.
**Best practice**: Log only secret keys/roles and operation status. Redact or hash sensitive fields. Use audit trails provided by the secret engine instead of application-level logging.

### 3. Static Fallbacks in Production
**Mistake**: Configuring hardcoded credentials or environment variables as fallback when the sidecar or secret engine is unreachable.
**Why it fails**: Defeats dynamic rotation, creates shadow credentials, and violates least privilege. Fallbacks are rarely rotated or monitored.
**Best practice**: Fail fast. Implement circuit breakers that pause non-critical workloads rather than degrading to static credentials. Use health checks to gate traffic.

### 4. Overly Broad IAM/Role Policies
**Mistake**: Granting the sidecar or workload identity access to all secret paths or database roles.
**Why it fails**: Compromised workloads can enumerate or generate credentials beyond their operational scope.
**Best practice**: Apply path-scoped policies. Use least-privilege roles that only permit `read` or `create` on specific dynamic roles. Audit policy drift quarterly.

### 5. Mixing Configuration and Secrets
**Mistake**: Using the same delivery mechanism for feature flags, endpoints, and credentials.
**Why it fails**: Configuration changes should be hot-reloadable without security reviews. Secrets require strict access control, audit trails, and rotation policies.
**Best practice**: Separate delivery channels. Use configmaps/feature flags for non-sensitive data. Route secrets exclusively through the dynamic secret pipeline.

### 6. Skipping Rotation Drills
**Mistake**: Assuming automatic expiration equals effective rotation.
**Why it fails**: Connection pools, long-running workers, and cached DNS/TCP sessions may hold stale credentials. Applications rarely handle mid-operation credential invalidation.
**Best practice**: Schedule monthly rotation tests. Validate that connection pools recreate sessions on auth failure. Implement graceful degradation and retry logic with exponential backoff.

## Production Bundle

### Action Checklist
- [ ] Audit existing secret storage: Identify all environment variables, config files, and static secret stores holding credentials.
- [ ] Define dynamic roles: Map each service to a scoped dynamic credential role with explicit TTL and permissions.
- [ ] Deploy sidecar injection: Configure init/sidecar containers to authenticate via workload identity and render secrets locally.
- [ ] Implement TTL-aware caching: Replace indefinite caches with lease-bound storage and pre-expiry renewal.
- [ ] Enforce fail-fast behavior: Remove static fallbacks and configure circuit breakers for secret engine unavailability.
- [ ] Enable audit logging: Route secret engine access, issuance, and revocation events to a centralized SIEM.
- [ ] Schedule rotation drills: Run monthly credential expiration tests and validate connection pool recovery.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Early-stage startup (<10 services) | Cloud-native static secret store (AWS Secrets Manager/Azure Key Vault) | Low operational overhead, native IAM integration, sufficient for limited blast radius | Low ($0–$50/mo) |
| Regulated enterprise (PCI/SOC2) | HashiCorp Vault + Dynamic DB/Cloud Roles + Sidecar | Full audit trail, policy-as-code, dynamic rotation, compliance-ready | Medium ($200–$800/mo infra) |
| Multi-cloud / hybrid | Vault Enterprise or Crossplane + External Secrets Operator | Unified control plane, avoids vendor lock-in, consistent rotation across clouds | High ($1k–$3k/mo) |
| Ephemeral workloads (batch/AI training) | Short-lived OIDC tokens + temporary service accounts | Credentials auto-revoke on pod termination, zero persistent storage | Low (pay-per-use IAM) |

### Configuration Template

**Vault Agent Config (`vault-agent.hcl`)**
```hcl
vault {
  address = "https://vault.internal:8200"
}

auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role = "app-service-role"
    }
  }
  sink "file" {
    config = {
      path = "/var/run/secrets/vault-token"
    }
  }
}

template {
  destination = "/etc/secrets/db-creds"
  contents = <<EOF
{{ with secret "database/creds/app-readonly" }}
DB_USER={{ .Data.username }}
DB_PASS={{ .Data.password }}
DB_LEASE_ID={{ .Data.lease_id }}
DB_TTL={{ .Data.lease_duration }}
{{ end }}
EOF
  refresh_interval = "5m"
}

Kubernetes Deployment Snippet

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      serviceAccountName: api-sa
      volumes:
        - name: vault-secrets
          emptyDir:
            medium: Memory
      containers:
        - name: vault-agent
          image: hashicorp/vault:1.15
          args: ["agent", "-config=/vault/config/vault-agent.hcl", "-log-level=warn"]
          volumeMounts:
            - name: vault-secrets
              mountPath: /etc/secrets
            - name: vault-config
              mountPath: /vault/config
        - name: app
          image: myregistry/api-service:latest
          env:
            - name: SECRETS_PATH
              value: "/etc/secrets/db-creds"
          volumeMounts:
            - name: vault-secrets
              mountPath: /etc/secrets
              readOnly: true

Quick Start Guide

1. Install and initialize Vault: Deploy a Vault instance (dev mode for testing, HA for production). Enable the database secrets engine and configure your target database connection.
2. Create a dynamic role: Define a role with scoped permissions, default_ttl="30m", and max_ttl="4h". Test credential generation via CLI.
3. Deploy the sidecar: Apply the Kubernetes manifest above. Verify the init container authenticates via Kubernetes ServiceAccount and renders credentials to the shared memory volume.
4. Run the TypeScript client: Point DynamicSecretClient to the local sidecar endpoint or file path. Validate that credentials refresh before TTL expiration and connection pools handle rotation gracefully.