self.step_count = 0 self._register_signal_handler()

def _register_signal_handler(self):
    def _handle_term(signum, frame):
        logger.info(f"Run {self.run_id}: SIGTERM received. Persisting state.")
        self._save_checkpoint(force=True)
        os._exit(0)
    signal.signal(signal.SIGTERM, _handle_term)

def _save_checkpoint(self, force=False):
    tmp_path = f"{self.checkpoint_path}/tmp-{self.run_id}-{os.getpid()}.pt"
    final_path = f"{self.checkpoint_path}/model-{self.run_id}.pt"
    
    state = {
        "step": self.step_count,
        "model_state": self.model.state_dict(),
        "optimizer_state": self.optimizer.state_dict(),
        "run_id": self.run_id
    }
    torch.save(state, tmp_path)
    
    # Atomic promotion
    os.replace(tmp_path, final_path)
    logger.info(f"Run {self.run_id}: Checkpoint promoted to {final_path}")

def train_step(self, batch):
    # Forward/backward pass logic
    self.step_count += 1
    if self.step_count % 500 == 0:
        self._save_checkpoint()

**Orchestrator-Level Retry Configuration (Argo Workflows)**
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ml-training-graph-
spec:
  entrypoint: execute-pipeline
  templates:
    - name: fit-model
      retryStrategy:
        limit: 4
        retryPolicy: "OnTransientError"
        backoff:
          duration: "20s"
          factor: 1.5
          maxDuration: "4m"
      container:
        image: registry.internal/ml-engine:v2.4.1
        command: ["python", "fit_model.py"]
        env:
          - name: RUN_CONTEXT
            value: "{{workflow.uid}}"
          - name: CHECKPOINT_BUCKET
            value: "s3://ml-artifacts/training/{{workflow.uid}}"
        resources:
          requests:
            memory: "16Gi"
            cpu: "4"
          limits:
            memory: "20Gi"
            cpu: "6"
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "echo 'Flushing buffers' > /dev/stdout"]

Architecture Decisions and Rationale

• Why atomic promotion? Object storage eventual consistency means partial writes can be read by downstream steps. Writing to a temporary key and using os.replace or S3 CopyObject ensures downstream consumers only see complete artifacts.
• Why orchestrator retries? Container-level retries obscure failure counts from monitoring systems and complicate lineage tracking. Argo/KFP retry policies maintain execution history, respect backoff limits, and integrate with alerting pipelines.
• Why minimal preStop? The hook runs synchronously within the grace period. Heavy logic delays SIGTERM delivery, risking SIGKILL and corrupted state. Async uploads or buffer flushes are safe; model serialization should happen in the main process.
• Why run-scoped identifiers? Correlating metrics, logs, and artifacts requires a deterministic key. Workflow UIDs or UUIDv4 tokens injected as environment variables guarantee traceability across distributed components.

Pitfall Guide

Production Bundle

Action Checklist

• Validate artifact storage connectivity and atomic promotion pattern before pipeline deployment
• Implement pre-flight idempotency guards on all compute-heavy steps
• Configure orchestrator-level retry policies with exponential backoff and transient error filtering
• Align terminationGracePeriodSeconds with checkpoint serialization time plus a 15-second buffer
• Inject deterministic run_id into container environments, metric labels, and artifact prefixes
• Pin dataset versions, container digests, and hyperparameter manifests to experiment tracking
• Define Prometheus alert rules for stalled progress, checkpoint upload failures, and OOM exits
• Execute chaos scenarios (pod deletion, network latency, storage throttling) in staging before production rollout

Decision Matrix

Configuration Template

# resilient-ml-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ml-execution-graph-
spec:
  entrypoint: main-pipeline
  templates:
    - name: main-pipeline
      dag:
        tasks:
          - name: validate-data
            template: run-validation
          - name: preprocess
            template: run-preprocess
            dependencies: [validate-data]
          - name: train-model
            template: run-training
            dependencies: [preprocess]
          - name: register-artifact
            template: run-registration
            dependencies: [train-model]
            onExit: notify-completion

    - name: run-training
      retryStrategy:
        limit: 3
        retryPolicy: "OnTransientError"
        backoff:
          duration: "25s"
          factor: 1.8
          maxDuration: "6m"
      container:
        image: registry.internal/trainer:v3.1.0
        command: ["python", "execute_training.py"]
        env:
          - name: EXECUTION_ID
            value: "{{workflow.uid}}"
          - name: STORAGE_ENDPOINT
            value: "s3://production-ml-artifacts"
          - name: GRACE_PERIOD_SEC
            value: "100"
        resources:
          requests:
            memory: "24Gi"
            cpu: "8"
          limits:
            memory: "32Gi"
            cpu: "10"
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sync && echo 'Buffer flush initiated'"]

    - name: run-registration
      container:
        image: registry.internal/registry-client:v1.0.2
        command: ["python", "promote_and_register.py"]
        env:
          - name: RUN_CONTEXT
            value: "{{workflow.uid}}"
          - name: REGISTRY_URL
            value: "https://ml-registry.internal/api/v1"

    - name: notify-completion
      container:
        image: registry.internal/webhook-sender:v0.9.4
        command: ["python", "send_status.py"]
        env:
          - name: WEBHOOK_URL
            valueFrom:
              secretKeyRef:
                name: pipeline-secrets
                key: alert-endpoint

Quick Start Guide

1. Deploy the workflow controller: Install Argo Workflows or Kubeflow Pipelines on your Kubernetes cluster. Verify RBAC permissions for artifact storage and registry endpoints.
2. Configure storage and tracking: Set up S3/GCS buckets with versioning enabled. Initialize MLflow or equivalent experiment tracking with a reachable backend database.
3. Inject run context: Modify your training script to read EXECUTION_ID or RUN_CONTEXT from environment variables. Implement atomic checkpoint promotion and signal handling as shown in the core solution.
4. Apply retry and grace policies: Update your workflow YAML with retryStrategy, terminationGracePeriodSeconds, and preStop hooks. Align grace periods with your checkpoint serialization benchmarks.
5. Validate with chaos testing: Run the pipeline in staging. Delete the training pod mid-execution, simulate storage throttling, and verify automatic resume, artifact promotion, and telemetry correlation. Iterate until MTTR falls below 15 minutes.