Step 1: PII/PHI Detection and Normalization

Before any cryptographic operation, the system must identify protected fields within the raw interaction payload. This requires a targeted detection pipeline that distinguishes between clinical terminology (which should be preserved) and regulated identifiers (which must be tokenized).

interface DetectedField {
  type: 'NAME' | 'DOB' | 'MRN' | 'SSN' | 'PHONE';
  rawValue: string;
  startIndex: number;
  endIndex: number;
}

function normalizeInput(value: string): string {
  // Strict normalization ensures deterministic hashing
  return value
    .toLowerCase()
    .trim()
    .replace(/\s+/g, ' ')
    .normalize('NFKC');
}

Step 2: Deterministic HMAC-SHA256 Tokenization

The core mechanism relies on HMAC-SHA256 with a secret key. Unlike standard hashing, HMAC requires a cryptographic key, making the output reversible only by someone possessing that key. More importantly, it guarantees that identical inputs always produce identical outputs, enabling cross-session matching.

import { createHmac } from 'crypto';

function generateDeterministicToken(
  fieldType: string,
  rawValue: string,
  secretKey: string
): string {
  const normalized = normalizeInput(rawValue);
  const payload = `${fieldType}:${normalized}`;
  
  const hmac = createHmac('sha256', secretKey);
  hmac.update(payload);
  
  const digest = hmac.digest('hex');
  // Use first 12 hex chars for readability + collision resistance
  return `${fieldType}_TKN_${digest.substring(0, 12)}`;
}

Step 3: Redacted Context Assembly & Storage

Once tokens are generated, the original payload is reconstructed with placeholders. The redacted version is what gets persisted. The database schema should never include columns for raw PHI.

interface MemoryRecord {
  recordId: string;
  agentId: string;
  redactedContext: string;
  tokenMap: Record<string, string>; // Maps token placeholders to full tokens
  retentionTTL: number; // Seconds
  createdAt: Date;
}

async function storeClinicalMemory(
  rawPayload: string,
  detectedFields: DetectedField[],
  secretKey: string,
  retentionDays: number
): Promise<MemoryRecord> {
  let redactedContext = rawPayload;
  const tokenMap: Record<string, string> = {};

  // Process fields in reverse order to maintain string indices
  const sortedFields = [...detectedFields].sort((a, b) => b.startIndex - a.startIndex);

  for (const field of sortedFields) {
    const token = generateDeterministicToken(field.type, field.rawValue, secretKey);
    const placeholder = `[${field.type}_REF]`;
    
    redactedContext = 
      redactedContext.slice(0, field.startIndex) + 
      placeholder + 
      redactedContext.slice(field.endIndex);
    
    tokenMap[placeholder] = token;
  }

  const record: MemoryRecord = {
    recordId: crypto.randomUUID(),
    agentId: 'clinical-intake-v2',
    redactedContext,
    tokenMap,
    retentionTTL: retentionDays * 86400,
    createdAt: new Date()
  };

  // Persist to compliant storage layer
  await memoryDb.insert(record);
  await auditLogger.logRedactionEvent({
    recordId: record.recordId,
    piiTypes: detectedFields.map(f => f.type),
    tokenCount: Object.keys(tokenMap).length,
    timestamp: record.createdAt
  });

  return record;
}

Step 4: Deterministic Retrieval

When a new session begins, the system tokenizes the incoming identifier using the exact same pipeline. It queries the memory store for matching tokens. Because the process is deterministic, the retrieved context aligns perfectly with prior sessions.

async function retrievePatientHistory(
  identifierType: 'NAME' | 'MRN',
  identifierValue: string,
  secretKey: string
): Promise<MemoryRecord[]> {
  const searchToken = generateDeterministicToken(identifierType, identifierValue, secretKey);
  
  // Query storage for records containing this token
  return await memoryDb.find({
    'tokenMap': { $elemMatch: { $eq: searchToken } },
    'retentionTTL': { $gt: Math.floor(Date.now() / 1000) }
  });
}

Architecture Rationale

• HMAC over SHA-256 alone: Standard hashing is vulnerable to rainbow table attacks. HMAC requires a secret key, ensuring that even if the database is compromised, tokens cannot be reverse-engineered without the key.
• Reverse-order string replacement: Prevents index shifting bugs when multiple PII fields exist in a single payload.
• Token map separation: Keeps the redacted context clean while maintaining a structured lookup for audit and retrieval.
• Infra-level TTL: Retention policies are enforced at the storage layer, not the application layer. This guarantees compliance even if the app crashes or fails to run cleanup jobs.

Pitfall Guide

1. Inconsistent Input Normalization Explanation: If "John Doe" and "john doe " produce different tokens, cross-session retrieval fails. Normalization must be strict and repeatable. Fix: Implement a canonical normalization pipeline that handles case, whitespace, Unicode variants, and locale-specific formatting before hashing.

2. Secret Key Rotation Without Versioning Explanation: Rotating the HMAC key breaks all existing token matches. Historical memory becomes orphaned. Fix: Use key versioning. Store the key version alongside each record. When rotating, migrate tokens in batches or maintain a dual-key lookup window during transition.

3. Over-Redaction of Clinical Vocabulary Explanation: Tokenizing medication names, lab values, or diagnosis codes strips the AI of necessary clinical context. Fix: Configure the PII detector to target only regulated identifiers (names, dates, IDs, contact info). Preserve clinical terminology through allow-listing or NLP-based entity classification.

4. Audit Log PHI Leakage Explanation: Developers often log raw payloads for debugging, accidentally writing PHI to monitoring systems. Fix: Enforce a strict logging policy where only token prefixes, event types, timestamps, and record IDs are logged. Implement automated log scanning to detect and redact accidental PHI exposure.

5. Application-Layer TTL Enforcement Explanation: Relying on cron jobs or app code to delete expired records creates compliance gaps during outages or deployment failures. Fix: Use storage engines with native TTL support (e.g., Redis EXPIRE, MongoDB TTL indexes, or cloud storage lifecycle policies). Delegate expiration to the infrastructure layer.

6. Ignoring BAA Scope in Third-Party Memory Stores Explanation: Even if PHI is tokenized, the memory provider may still be considered a Business Associate if they can access the secret key or reconstruct data. Fix: Ensure the architecture guarantees zero-knowledge storage. The memory provider should never receive the HMAC key. Sign BAAs that explicitly define tokenized data as non-PHI under your control.

7. Single-Tenant Key Architecture Explanation: Using one global key for all patients means a key compromise exposes the entire dataset. Fix: Implement tenant-scoped or patient-scoped key derivation. Use a master key to derive per-tenant keys via HKDF, isolating breach impact and simplifying compliance audits.

Production Bundle

Action Checklist

• Define PII/PHI detection rules: Map exactly which fields require tokenization vs. clinical preservation.
• Implement strict input normalization: Ensure case, whitespace, and Unicode handling are deterministic.
• Configure HMAC-SHA256 with HSM-backed key management: Never store keys in environment variables or code repositories.
• Design redacted storage schema: Ensure zero columns accept raw PHI; use token maps for retrieval.
• Enable infrastructure-level TTL: Delegate retention enforcement to the database or storage provider.
• Build audit event pipeline: Log only token prefixes, event types, and timestamps; scan logs for accidental PHI.
• Validate BAA alignment: Confirm memory providers cannot access HMAC keys or reconstruct original data.
• Test cross-session retrieval: Verify identical inputs produce identical tokens across independent sessions.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Early-stage clinical MVP	Ephemeral sessions + manual context prompts	Fastest deployment; avoids compliance overhead during validation	Low (No tokenization infra)
Regulated chronic care AI	Deterministic tokenization + infra TTL	Required for longitudinal context; meets HIPAA audit standards	Medium (Key management, PII detection pipeline)
Multi-tenant SaaS for health systems	Tenant-scoped HMAC keys + zero-knowledge storage	Isolates breach impact; simplifies BAA negotiations per tenant	High (HSM integration, key rotation automation)
Research/De-identified datasets	One-way hashing without secret key	Removes re-identification risk; complies with Safe Harbor de-identification	Low (No key management, but loses cross-session matching)

Configuration Template

// compliance-config.ts
export const HIPAA_MEMORY_CONFIG = {
  tokenization: {
    algorithm: 'HMAC-SHA256',
    keyDerivation: 'HKDF-SHA256',
    masterKeySource: 'AWS_KMS', // or Azure Key Vault / GCP KMS
    keyVersioning: true,
    normalization: {
      case: 'lowercase',
      whitespace: 'collapse',
      unicode: 'NFKC',
      locale: 'en-US'
    }
  },
  piiDetection: {
    targetFields: ['NAME', 'DOB', 'MRN', 'SSN', 'PHONE', 'EMAIL'],
    preserveClinicalTerms: true,
    confidenceThreshold: 0.85
  },
  storage: {
    engine: 'PostgreSQL',
    ttlEnforcement: 'INFRASTRUCTURE', // Never application-layer
    auditLogging: {
      logRawPayload: false,
      logTokenPrefix: true,
      retentionDays: 2555 // 7 years per HIPAA
    }
  }
};

Quick Start Guide

1. Initialize Key Management: Provision an HMAC key through your cloud provider’s KMS. Export the key ARN/ID to your application configuration. Never embed the raw key in code.
2. Deploy PII Detection Pipeline: Integrate a regex/NLP-based detector that flags names, dates, and IDs. Test it against synthetic clinical payloads to verify clinical terms are preserved.
3. Implement Tokenization & Storage: Use the HMAC-SHA256 function to generate tokens, replace PII in the payload, and store the redacted context with a TTL. Verify the database contains zero raw PHI.
4. Validate Cross-Session Retrieval: Submit identical patient identifiers across separate test sessions. Confirm the system returns matching tokens and retrieves the correct historical context.
5. Enable Audit & TTL Enforcement: Configure storage-level expiration policies. Route all redaction events to a compliant audit log. Run a compliance scan to verify no PHI leakage in logs or backups.