Full AI Infrastructure Deployment on AWS: Architecture, Pipeline, and Production Setup

ction before it is model-ready. This layer should be automated using AWS Glue or AWS EMR for heavy compute.

• Workflow:
  • Trigger Glue Jobs via EventBridge when new data lands in S3.
  • Perform schema validation, null handling, and feature generation.
  • Split data into train, validation, and test sets.
  • Write processed datasets to s3://<account-id>-ai-processed-data.
• Best Practice: Store feature definitions in a Feature Store (e.g., Amazon SageMaker Feature Store) to ensure consistency between training and inference, preventing training-serving skew.

3. Reproducible Training and Model Registry

Training jobs must be reproducible and tracked. AWS SageMaker provides managed infrastructure for training, hyperparameter tuning, and experiment tracking.

• Implementation:
  • Define a SageMaker Training Job that reads processed data from S3.
  • Execute the training script, which outputs a model artifact.
  • Evaluate metrics against a quality threshold.
  • Register the model in the SageMaker Model Registry with metadata including dataset version, code commit hash, and evaluation metrics.
• Rationale: The Model Registry enforces governance. It prevents the "model_final_v7.joblib" anti-pattern by providing a structured catalog of model versions, their lineage, and their approval status.

Training Script Example: The following script demonstrates a production-style training job using XGBoost. It reads from SageMaker's mounted input channels and writes artifacts to the designated output directory.

import os
import pandas as pd
import xgboost as xgb
import joblib

# SageMaker injects environment variables for paths
train_input = os.path.join(os.environ.get("SM_CHANNEL_TRAIN", "/opt/ml/input/data/train"), "dataset.csv")
model_output_dir = os.environ.get("SM_MODEL_DIR", "/opt/ml/model")

# Load and prepare data
df = pd.read_csv(train_input)
features = ["user_tenure_months", "transaction_volume", "support_tickets"]
target = "churn_label"

X_train = df[features]
y_train = df[target]

# Initialize and train model
model = xgb.XGBClassifier(
    n_estimators=150,
    learning_rate=0.05,
    max_depth=6,
    objective="binary:logistic",
    eval_metric="logloss"
)
model.fit(X_train, y_train)

# Save artifact to SageMaker output path
artifact_path = os.path.join(model_output_dir, "model.joblib")
joblib.dump(model, artifact_path)
print(f"Model saved to {artifact_path}")

4. Containerized Inference Service

For the serving layer, containerization ensures portability and consistency. The inference service should be a lightweight API that loads the model artifact and handles requests.

• Architecture: Package the inference code and model artifact into a Docker image. Push the image to Amazon ECR. Deploy the image to Amazon ECS Fargate or SageMaker Endpoints.
• Rationale: Containers encapsulate dependencies, ensuring the inference environment matches the training environment's requirements. ECS Fargate provides serverless compute, eliminating the need to manage EC2 instances.

Inference API Example: This FastAPI service implements a health check endpoint for load balancer integration and a prediction endpoint with structured input validation.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI(title="ChurnPredictionService", version="1.0.0")

# Load model at startup
_model = joblib.load("/app/artifacts/model.joblib")

class InferencePayload(BaseModel):
    tenure: int
    volume: float
    tickets: int

@app.get("/v1/status")
def readiness_probe():
    """Health check for load balancer integration."""
    return {"service": "churn-predictor", "state": "ready"}

@app.post("/v1/predict")
def generate_prediction(payload: InferencePayload):
    """Generate prediction with error handling."""
    try:
        input_vector = np.array([[payload.tenure, payload.volume, payload.tickets]])
        prob = _model.predict_proba(input_vector)[0][1]
        return {
            "risk_score": round(float(prob), 4),
            "threshold_exceeded": prob > 0.75
        }
    except Exception as exc:
        raise HTTPException(status_code=500, detail=str(exc))

Dockerfile Example: A multi-stage build approach reduces image size and improves security.

FROM public.ecr.aws/docker/library/python:3.11-slim AS base

WORKDIR /srv/app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code and artifacts
COPY src/ ./src/
COPY artifacts/ ./artifacts/

# Expose port and define entrypoint
EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]

5. Infrastructure as Code and CI/CD

Manual configuration leads to drift and errors. Define all infrastructure using Terraform and automate deployments with GitHub Actions.

• Terraform: Manage S3 buckets, ECR repositories, ECS clusters, and IAM roles as code.
• CI/CD: Trigger pipelines on code commits. Build Docker images, run tests, push to ECR, and update ECS services.

Terraform Configuration: Define core resources with tagging and security configurations.

resource "aws_s3_bucket" "ml_artifacts" {
  bucket = "prod-ml-artifacts-${var.env}"
  tags   = { Project = "AI-Platform", ManagedBy = "Terraform" }
}

resource "aws_ecr_repository" "inference_image" {
  name                 = "inference-api"
  image_tag_mutability = "MUTABLE"
  
  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_cloudwatch_log_group" "inference_logs" {
  name              = "/ecs/inference-api"
  retention_in_days = 30
}

CI/CD Pipeline: Automate the build and deployment process.

name: Build and Deploy Inference
on:
  push:
    branches: [release]

env:
  IMAGE_URI: ${{ secrets.AWS_ACCOUNT }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/inference-api

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.CI_ROLE_ARN }}
          aws-region: ${{ secrets.AWS_REGION }}
      
      - uses: aws-actions/amazon-ecr-login@v2
      
      - name: Build and Push
        run: |
          docker build -t ${{ env.IMAGE_URI }} .
          docker push ${{ env.IMAGE_URI }}
      
      - name: Update ECS Service
        run: |
          aws ecs update-service \
            --cluster prod-cluster \
            --service inference-service \
            --force-new-deployment

6. Security and Observability

Security must be baked into the architecture. Use IAM roles for least-privilege access, encrypt data at rest with KMS, and store secrets in AWS Secrets Manager. Deploy compute in private subnets and use VPC endpoints for S3 and ECR access.

Observability requires monitoring latency, error rates, and model drift. Configure CloudWatch Alarms for API errors and latency thresholds. Use SageMaker Model Monitor to detect data drift in production inputs.

Pitfall Guide

Production AI systems fail due to common architectural and operational mistakes. The following guide highlights critical pitfalls and their remedies.

Pitfall	Explanation	Fix
Overwriting Raw Data	Transformations modify the original input files, making it impossible to reproduce results or recover from errors.	Implement immutable raw storage. All transformations should write to new paths/buckets, preserving the source.
Training-Serving Skew	Feature engineering logic differs between training and inference, causing predictions to diverge.	Share feature code between training and inference. Use a Feature Store to ensure consistent feature computation.
Hardcoded Credentials	AWS keys or database passwords are embedded in code or environment variables, risking exposure.	Use IAM roles for service access. Store sensitive configuration in AWS Secrets Manager and retrieve at runtime.
Missing Model Registry	Models are stored as files with ambiguous names, leading to confusion about which version is live.	Implement a Model Registry with metadata tracking. Enforce approval workflows before promoting models to production.
Ignoring Drift	Models degrade as input distributions shift, but no monitoring detects the decline.	Deploy drift detection tools. Monitor feature distributions and prediction confidence in production.
Monolithic Containers	Training and inference share the same container image, bloating the inference image with unnecessary dependencies.	Separate training and inference images. Inference images should be minimal, containing only the model and API code.
Lack of Health Checks	Load balancers cannot detect unhealthy instances, routing traffic to failing services.	Implement /health and /ready endpoints. Configure ALB health checks to verify service status before routing traffic.

Production Bundle

Action Checklist

Ensure your AI infrastructure meets production standards by verifying the following items:

• Immutable Storage: Verify raw data buckets are append-only and never modified by downstream processes.
• Model Registry: Confirm all models are registered with metadata, including dataset version and evaluation metrics.
• Drift Monitoring: Set up automated drift detection for key features and prediction distributions.
• Least Privilege IAM: Audit IAM roles to ensure services have only the permissions required for their function.
• Latency Alerts: Configure CloudWatch alarms for p95 latency and error rates on the inference API.
• Rollback Procedure: Test the ability to rollback to a previous model version within minutes.
• Secrets Management: Replace all hardcoded credentials with references to AWS Secrets Manager.
• Private Subnets: Ensure compute resources are deployed in private subnets with VPC endpoints for AWS services.

Decision Matrix

Select the appropriate serving strategy based on your workload characteristics and team maturity.

Scenario	Recommended Approach	Why	Cost Impact
Standard model, rapid deployment	SageMaker Managed Endpoint	Fully managed, autoscaling, integrated with SageMaker ecosystem.	Medium; pay for managed service overhead.
Custom pre-processing, shared infra	ECS Fargate	Flexibility to include custom logic; shares infrastructure costs with other services.	High; efficient resource utilization, no idle server costs.
Multi-model, GPU sharing	EKS	Advanced orchestration, GPU time-slicing, multi-tenancy support.	Low-Medium; high operational cost, but efficient hardware usage.
Low-frequency, bursty traffic	AWS Lambda	Serverless, scales to zero, pay-per-request.	Variable; cost-effective for low volume, expensive at scale.

Configuration Template

Use this Terraform template to provision an ECS service for inference deployment. This template includes network configuration, security groups, and load balancer integration.

resource "aws_ecs_service" "inference" {
  name            = "inference-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.inference.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = data.aws_subnets.private.ids
    security_groups  = [aws_security_group.inference.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.inference.arn
    container_name   = "inference-api"
    container_port   = 8080
  }

  depends_on = [aws_lb_listener.frontend]
}

resource "aws_security_group" "inference" {
  name_prefix = "inference-sg"
  vpc_id      = data.aws_vpc.main.id

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Quick Start Guide

Deploy a basic AI inference service on AWS in under five minutes using the following steps:

1. Provision Infrastructure: Run terraform apply to create S3 buckets, ECR repository, and ECS cluster.
2. Train Model: Execute a SageMaker training job using the provided script. Register the model in the Model Registry.
3. Build Image: Run docker build -t <account>.dkr.ecr.<region>.amazonaws.com/inference-api . and push to ECR.
4. Deploy Service: Update the ECS service with the new image using aws ecs update-service --cluster prod --service inference --force-new-deployment.
5. Verify: Curl the health endpoint curl https://<alb-dns>/v1/status to confirm the service is running.