CodcompassCodcompass
← Back to Blog
DevOps2026-05-09·80 min read

How to Deploy a Machine Learning Project on AWS Using ECR, ECS Fargate, and EFS.

By Tendong Brain Nkengafac

Architecting Serverless Real-Time Inference Pipelines with AWS Container Services

Current Situation Analysis

Machine learning engineering has matured rapidly in model development, yet production deployment remains a persistent bottleneck. Teams routinely excel at training CatBoost or XGBoost models in isolated environments, but struggle when transitioning to live, stateful inference services. The core friction lies in state management: real-time forecasting requires continuous data exchange between training pipelines, inference engines, and visualization layers. Traditional stateless container architectures break down when multiple services must read and write shared artifacts simultaneously.

This problem is frequently overlooked because cloud providers abstract compute provisioning, leading engineers to assume storage and networking will configure themselves. In reality, container orchestration platforms like ECS Fargate deliberately decouple compute from state. Without a coordinated shared storage strategy, inference containers cannot pass predictions to dashboard services, forcing teams into fragile workarounds like polling S3, embedding state in container memory, or managing custom EC2 fleets.

Technical constraints compound the issue. Fargate tasks are ephemeral by design, meaning local filesystem writes vanish on termination. Network File System (NFS) protocols introduce latency and mount complexity. IAM permissions must be scoped precisely to allow image pulls and log streaming without granting excessive privileges. Industry deployment telemetry consistently shows that over 60% of ML projects stall at the infrastructure layer, not due to algorithmic failure, but because of unaddressed state synchronization and container lifecycle management.

WOW Moment: Key Findings

The architectural shift from managed virtual machines to serverless containers with network-attached storage fundamentally changes how ML pipelines scale. Below is a comparative analysis of deployment strategies for real-time inference workloads.

Approach Provisioning Time Storage Latency Operational Overhead Cost Efficiency
Monolithic EC2 + EBS 15–30 mins <5 ms (local) High (patching, scaling, AMI management) Low (idle compute costs)
ECS Fargate + EFS <2 mins 10–25 ms (network) Minimal (serverless, auto-scaling) High (pay-per-second, no idle nodes)
Lambda + S3 <1 min 50–100 ms (API calls) Medium (cold starts, payload limits) Variable (request-based pricing)

The Fargate + EFS combination delivers the optimal balance for real-time ML dashboards. EFS provides POSIX-compliant shared storage that multiple containers can mount simultaneously, while Fargate eliminates node management entirely. The slight network latency is negligible for second-level inference ticks, and the operational overhead drops to near zero. This architecture enables true serverless ML inference without sacrificing state synchronization.

Core Solution

Deploying a real-time forecasting pipeline requires coordinating four AWS services: Elastic Container Registry (ECR) for image distribution, ECS Fargate for compute, EFS for shared state, and IAM for secure task execution. The following implementation uses bash/zsh syntax, decoupled service naming, and production-hardened configurations.

Step 1: Environment Initialization & Authentication

Begin by establishing session context and authenticating the Docker client against your private registry. Hardcoding credentials is a security anti-pattern; leverage AWS STS to dynamically resolve your account identifier.

export AWS_REGION="us-east-1"
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export PIPELINE_NS="temporal-forecast-pipeline"

aws ecr get-login-password --region "$AWS_REGION" | \
  docker login --username AWS --password-stdin "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"

The get-login-password command generates a temporary authorization token valid for 12 hours. Piping it directly to docker login prevents token leakage in shell history. Verify connectivity by confirming the Login Succeeded response.

Step 2: Container Registry & Image Distribution

Create a dedicated repository and push your optimized container image. Production deployments should never use development Dockerfiles. Strip debugging tools, pin base image versions, and leverage multi-stage builds to minimize attack surface and transfer time.

# Create repository if it does not exist
aws ecr describe-repositories --repository-names "$PIPELINE_NS" --region "$AWS_REGION" || \
  aws ecr create-repository --repository-name "$PIPELINE_NS" --region "$AWS_REGION"

# Build with cloud-specific optimizations
docker build -f Dockerfile.prod -t "$PIPELINE_NS:stable" .

# Tag and push
docker tag "$PIPELINE_NS:stable" "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${PIPELINE_NS}:stable"
docker push "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${PIPELINE_NS}:stable"

Docker's layer caching mechanism automatically resumes interrupted uploads. If a network timeout occurs mid-push, re-executing the docker push command transfers only missing layers, preserving bandwidth and deployment velocity.

Step 3: IAM Task Execution Role

Fargate tasks operate without host credentials. They require an execution role to pull images, write metrics, and stream logs. Construct a trust policy that explicitly permits the ECS task scheduler to assume the role.

cat > ecs-execution-trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ecs-tasks.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name fargate-ml-executor \
  --assume-role-policy-document file://ecs-execution-trust.json

aws iam attach-role-policy \
  --role-name fargate-ml-executor \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

aws iam attach-role-policy \
  --role-name fargate-ml-executor \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

The AmazonECSTaskExecutionRolePolicy grants CloudWatch Logs and Secrets Manager access. The AmazonEC2ContainerRegistryReadOnly policy restricts ECR access to pull operations only. Never attach FullAccess policies to execution roles; principle of least privilege prevents credential escalation if a container is compromised.

Step 4: Cluster & Networking Topology

Fargate clusters are logical boundaries, not physical compute pools. They require VPC configuration to route traffic and enforce network isolation.

# Create cluster
aws ecs create-cluster --cluster-name ml-inference-cluster --region "$AWS_REGION"

# Resolve default VPC and subnet
export VPC_ID=$(aws ec2 describe-vpcs --filters "Name=isDefault,Values=true" --query "Vpcs[0].VpcId" --output text --region "$AWS_REGION")
export SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" --query "Subnets[0].SubnetId" --output text --region "$AWS_REGION")

# Create security group
export SG_ID=$(aws ec2 create-security-group \
  --group-name ml-pipeline-sg \
  --description "Inference & Dashboard Network Policy" \
  --vpc-id "$VPC_ID" --query "GroupId" --output text --region "$AWS_REGION")

# Open required ports
aws ec2 authorize-security-group-ingress \
  --group-id "$SG_ID" --protocol tcp --port 8050 --cidr 0.0.0.0/0 --region "$AWS_REGION"

aws ec2 authorize-security-group-ingress \
  --group-id "$SG_ID" --protocol tcp --port 2049 --cidr 0.0.0.0/0 --region "$AWS_REGION"

Port 8050 serves the Dash visualization layer. Port 2049 handles NFS traffic for EFS mounts. In production, restrict CIDR blocks to known IP ranges or use VPC endpoints to eliminate public internet exposure.

Step 5: Shared Storage & Task Definition

EFS provides the POSIX-compliant filesystem required for inter-container state sharing. Create the filesystem, mount targets, and register a Fargate task definition that references both containers and the shared volume.

# Create EFS filesystem
export EFS_ID=$(aws efs create-file-system \
  --creation-token ml-artifacts-store \
  --performance-mode generalPurpose \
  --query "FileSystemId" --output text --region "$AWS_REGION")

# Create mount target in the target subnet
aws efs create-mount-target \
  --file-system-id "$EFS_ID" \
  --subnet-id "$SUBNET_ID" \
  --security-groups "$SG_ID" --region "$AWS_REGION"

The task definition orchestrates container placement, resource allocation, and volume mounting. Fargate requires explicit CPU and memory reservations. EFS volumes are mounted using the efsVolumeConfiguration block.

Pitfall Guide

Real-time ML deployments fail predictably when infrastructure assumptions clash with cloud realities. The following pitfalls represent the most common production failures, along with proven mitigation strategies.

1. Overly Permissive Security Groups Explanation: Opening ports to 0.0.0.0/0 exposes inference endpoints and NFS mounts to the public internet. Attackers can scan for unauthenticated Dash instances or exploit NFS misconfigurations. Fix: Restrict ingress to specific CIDR blocks, corporate VPN ranges, or use VPC endpoints. Reference security groups by ID instead of IP ranges when containers communicate internally.

2. EFS Performance Mode Mismatch Explanation: EFS offers generalPurpose and maxIO modes. maxIO scales to higher throughput but increases latency by 10–20%. Real-time inference ticking at 1-second intervals suffers under maxIO. Fix: Use generalPurpose for low-latency ML artifact sharing. Reserve maxIO for batch processing or large-scale data ingestion pipelines.

3. Missing Container Health Checks Explanation: Fargate replaces unhealthy tasks automatically, but without health checks, it cannot distinguish between a slow startup and a crashed process. This causes task thrashing and deployment instability. Fix: Define healthCheck in the task definition using curl or wget against the service's readiness endpoint. Set appropriate startPeriod, interval, and timeout values.

4. Docker Image Bloat & Layer Inefficiency Explanation: Development images often include Jupyter, test suites, and verbose logging libraries. Bloated images increase ECR storage costs, slow down Fargate cold starts, and expand the attack surface. Fix: Implement multi-stage builds. Compile dependencies in a builder stage, copy only runtime artifacts to a minimal base image (e.g., python:3.11-slim), and use .dockerignore to exclude .git, __pycache__, and local configs.

5. Hardcoded AWS Credentials in Containers Explanation: Embedding AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in environment variables or Dockerfiles violates security best practices. Keys can be extracted from container metadata or logs. Fix: Always use IAM Roles for Tasks. Containers automatically receive temporary credentials via the ECS agent. Never store long-term keys in container images or task definitions.

6. NFS Mount Timeouts on Cold Starts Explanation: Fargate tasks provisioning for the first time may experience NFS mount delays, causing the container to exit before the filesystem is ready. Fix: Add noresvport and nconnect=8 to EFS mount options in the task definition. Implement retry logic in the application startup script to wait for the mount point before initializing the inference engine.

7. Unconfigured CloudWatch Log Groups Explanation: Fargate tasks stream logs to CloudWatch by default, but without explicit log group configuration, logs accumulate in default groups, making debugging and cost tracking difficult. Fix: Define logConfiguration in the task definition with a custom awslogs-group and awslogs-stream-prefix. Set retention policies to control storage costs.

Production Bundle

Action Checklist

  • Verify IAM execution role has least-privilege policies attached (ECR ReadOnly + ECS Task Execution)
  • Implement multi-stage Docker builds and validate image size under 500MB
  • Configure EFS with generalPurpose performance mode and appropriate mount options
  • Define container health checks with realistic start periods and retry thresholds
  • Restrict security group ingress to known CIDR ranges or VPC endpoints
  • Set CloudWatch log retention policies to prevent unbounded storage costs
  • Test cold-start behavior by terminating tasks and measuring time-to-ready
  • Document rollback procedures using ECR image tags and ECS service deployments

Decision Matrix

Scenario Recommended Approach Why Cost Impact
Real-time inference (<5s latency) ECS Fargate + EFS Serverless compute with POSIX shared storage eliminates state management overhead Moderate (pay-per-second + EFS storage)
Batch model training (hours) EC2 Spot + EBS Predictable workloads benefit from reserved capacity and local disk performance Low (spot pricing + EBS gp3)
Event-driven predictions Lambda + S3 Stateless, request-based scaling ideal for sporadic inference triggers Variable (request count + data transfer)
Multi-tenant ML platform ECS EC2 + ECR + VPC Endpoints Full control over networking, GPU allocation, and tenant isolation High (EC2 baseline + operational overhead)

Configuration Template

Copy this ECS task definition template for Fargate deployments with EFS integration. Adjust CPU, memory, and image URIs to match your workload.

{
  "family": "ml-inference-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "4096",
  "executionRoleArn": "arn:aws:iam::ACCOUNT_ID:role/fargate-ml-executor",
  "taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/fargate-ml-executor",
  "containerDefinitions": [
    {
      "name": "forecast-engine",
      "image": "ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/temporal-forecast-pipeline:stable",
      "essential": true,
      "portMappings": [{"containerPort": 8050, "protocol": "tcp"}],
      "mountPoints": [{"sourceVolume": "ml-artifacts", "containerPath": "/data/artifacts"}],
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:8050/health || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 60
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/ml-inference",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "forecast-engine"
        }
      }
    }
  ],
  "volumes": [
    {
      "name": "ml-artifacts",
      "efsVolumeConfiguration": {
        "fileSystemId": "fs-EXAMPLE123456789",
        "rootDirectory": "/",
        "transitEncryption": "ENABLED",
        "authorizationConfig": {
          "accessPointId": "",
          "iam": "DISABLED"
        }
      }
    }
  ]
}

Quick Start Guide

  1. Initialize Session & Auth: Export region and account variables, authenticate Docker against ECR using aws ecr get-login-password.
  2. Build & Push Image: Run docker build -f Dockerfile.prod and push to your ECR repository using the full registry URI.
  3. Provision Infrastructure: Create the ECS cluster, resolve VPC/subnet IDs, configure security groups for ports 8050 and 2049, and deploy the EFS filesystem with mount targets.
  4. Deploy Task Definition: Register the Fargate task definition with EFS volume configuration, IAM execution role, and health checks. Launch the service and verify container status transitions to RUNNING.