Scalability | Storage Cost Efficiency | |----------|-------------------------|-------------------|-------------|------------------------|-------------------------| | Traditional Monolithic (Sync API + RDBMS) | ~500K | 2.8s | 92% | Linear (requires DB sharding) | Low (DB storage expensive) | | Hybrid Async (Queue + DB Artifacts) | ~2.1M | 340ms | 68% | Moderate (queue consumer scaling) | Medium (mixed storage tiers) | | Proposed Decoupled (FastAPI + Kafka + PySpark + S3) | 8.5M+ | 45ms | 18% | Elastic (Spark/K8s auto-scale) | High (Parquet/S3 tiered) |

Key Findings:

• Decoupling the control and compute planes reduces API latency by ~98% and shifts heavy lifting to horizontally scalable Spark executors.
• Relational database load drops by ~80% when analytical artifacts are routed to object storage, preserving PostgreSQL for metadata, state, and multi-tenant context.
• The architecture achieves optimal cost-performance balance when data versions are treated as immutable snapshots, enabling parallel reprocessing without write conflicts.

Core Solution

The architecture enforces a strict separation between the control plane and compute plane, orchestrated through message-driven execution and object storage contracts.

Architecture & Stack

• Control Plane: FastAPI + PostgreSQL. Handles request validation, process metadata, multi-tenant context, and callback consolidation. Strictly I/O bound.
• Compute Plane: PySpark on Kubernetes. Stateless, short-lived ETL jobs that read/write Parquet datasets. Scales horizontally via K8s pod autoscaling.
• Messaging: Kafka (or equivalent queue service). Decouples API triggers from job execution using a generic CLI-like message contract.
• Storage: S3-compatible object storage. Acts as the source of truth for all analytical data with a deterministic, versioned layout.

Execution Flow

1. Request Validation: API validates input and persists process metadata to PostgreSQL.
2. Message Publication: Publishes a structured message to Kafka containing:
   • entity_id
   • data_version
   • action_type
   • callback_url
3. Launcher Mapping: A generic launcher service consumes the message and resolves the action to:
   • Target Spark container image
   • Resource profile (driver/executors CPU, memory, partitions)
4. Job Orchestration: Launcher submits the Spark job to Kubernetes.
5. Distributed Processing:
   • Reads input datasets (Parquet) from object storage
   • Executes PySpark transformation pipelines
   • Writes outputs back to the versioned storage path
6. Status Reconciliation: Job invokes the callback_url with execution status, metrics, and artifact locations.

Message Contract Example

{
  "entity_id": "cust_8842",
  "data_version": "v2024.10.05",
  "action": "aggregate_revenue",
  "callback_url": "https://api.internal/v1/processes/callback",
  "resource_profile": {
    "driver_cores": 4,
    "executor_instances": 8,
    "executor_memory": "16g"
  }
}

Data Model & Storage Contract

Large analytical data is routed exclusively to object storage using a deterministic layout: {entity_id}/{data_version}/...

This design guarantees:

• Immutability: Each version is a read-only snapshot.
• Reproducibility: Identical inputs + version → identical outputs.
• Isolation: Concurrent runs never overwrite historical data.
• Scalability: Zero relational database pressure from analytical payloads.

Pitfall Guide

1. Sync API Blocking: Never execute heavy transformations within the FastAPI request lifecycle. Keep the control plane strictly I/O bound and delegate computation to async Spark jobs.
2. Analytical Data in RDBMS: PostgreSQL should only store process metadata, user context, and audit trails. Storing millions of transformed rows or large Parquet/CSV artifacts will exhaust I/O and inflate costs.
3. Implicit Storage Contracts: Object storage paths function as shared APIs. Treat {entity_id}/{data_version}/ layouts as versioned contracts with documentation, schema validation, and integration tests.
4. Stateful Spark Jobs: Compute jobs must remain stateless and short-lived. Persist all intermediate/final states to object storage and rely on explicit callbacks for orchestration visibility.
5. Missing Callback Mechanisms: Without a reliable callback endpoint, the control plane cannot track job completion, failure, or artifact location. Implement idempotent callback handlers with retry logic.
6. Static Resource Profiling: Hardcoding Spark driver/executor configurations leads to K8s node starvation or underutilization. Map resource profiles dynamically based on data volume and action complexity.
7. Ignoring Data Versioning: Overwriting storage paths breaks reproducibility and isolation. Enforce immutable versioning at the pipeline level and validate version existence before job submission.

Deliverables

• Architecture Blueprint: Visual mapping of control plane (FastAPI/PostgreSQL), compute plane (PySpark/K8s), messaging layer (Kafka), and storage contract (S3 Parquet layout). Includes data flow diagrams and failure boundary definitions.
• Deployment & Validation Checklist: Pre-flight verification steps covering Kafka topic provisioning, Spark image registry access, K8s RBAC permissions, S3 lifecycle policies, callback endpoint health checks, and storage contract versioning.
• Configuration Templates:
  • Kafka message schema (JSON/YAML)
  • FastAPI async endpoint stub with metadata persistence
  • PySpark Kubernetes launcher configuration (SparkOperator/submit args)
  • S3 directory layout specification with immutability enforcement rules