g control over application components. The version field tracks chart releases; appVersion tracks the bundled application. Keeping them distinct prevents accidental downgrades during infrastructure updates and clarifies whether a release contains template fixes or application code changes.
2. CI/CD Validation Pipeline
Every commit must pass through a deterministic validation sequence. The pipeline should lint syntax, render manifests without cluster access, execute unit tests, and package the artifact.
# .github/workflows/chart-delivery.yml
name: Helm Delivery Pipeline
on:
push:
paths: ['charts/**']
pull_request:
paths: ['charts/**']
jobs:
validate-and-package:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Helm
uses: azure/setup-helm@v3
with:
version: 'v3.14.0'
- name: Lint Chart
run: helm lint ./charts/platform-stack --strict
- name: Dry-Run Render
run: helm template release-test ./charts/platform-stack --debug --values ./charts/platform-stack/values.yaml
- name: Run Chart Tests
run: helm test platform-stack --cleanup
- name: Package Release
run: helm package ./charts/platform-stack --destination ./dist
- name: Upload Artifact
uses: actions/upload-artifact@v4
with:
name: helm-package
path: ./dist/*.tgz
Why this sequence? Linting catches syntax errors before they reach the cluster. helm template --debug validates Go template logic and value injection without requiring a live Kubernetes API server. Packaging creates a versioned .tgz artifact that can be stored in an OCI registry or chart repository, ensuring reproducible deployments across environments.
3. Semantic Versioning & Release Tracking
Chart versions must follow SemVer. Patch releases fix template bugs or value defaults. Minor releases add features or new subcharts. Major releases introduce breaking changes to the values schema. Tag every release in your version control system and maintain a CHANGELOG.md that maps chart versions to application versions and dependency updates.
4. Dependency Locking
Unlocked dependencies cause silent drift. Run helm dependency update during CI and commit the Chart.lock file. This pins exact subchart versions, preventing unexpected upgrades when a parent chart is deployed.
5. Rollback & State Verification
Never assume helm upgrade is idempotent. Always pair upgrades with helm rollback capabilities and post-deployment health checks. Use Helm test hooks to verify service readiness before marking a release as successful.
# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "platform-stack.fullname" . }}-test-conn"
labels:
{{- include "platform-stack.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": hook-succeeded
spec:
containers:
- name: wget
image: busybox
command: ['wget', '--no-check-certificate', '--spider', 'http://{{ include "platform-stack.fullname" . }}-backend:8080/health']
restartPolicy: Never
Why test hooks? They shift validation from manual kubectl checks to automated, cluster-native verification. A failed test blocks promotion to production, catching misconfigured endpoints or missing environment variables before they cause outages.
Pitfall Guide
Production Helm workflows fail when teams apply application development patterns to infrastructure delivery without adaptation. Here are the most common failure modes and how to prevent them.
1. Chart Version vs App Version Conflation
Explanation: Teams increment appVersion when they should bump chart version, or vice versa. This breaks rollback tracking and makes it impossible to distinguish between a template fix and an application update.
Fix: Treat version as infrastructure release tracking and appVersion as bundled software tracking. Update version for every chart change, regardless of application updates.
2. Hardcoding Secrets in Values Files
Explanation: Placing database passwords or API keys directly in values.yaml or committing them to version control exposes credentials and prevents environment-specific overrides.
Fix: Use external secret management (Vault, AWS Secrets Manager, or Kubernetes External Secrets). Reference secrets via environment variables or mounted volumes, and leave placeholders in values.yaml that CI/CD pipelines populate at runtime.
3. Skipping Dry-Run Validation
Explanation: Relying solely on helm install --dry-run during development misses template logic errors that only surface under specific value combinations.
Fix: Run helm template with all environment-specific value files in CI. Fail the pipeline if rendering produces invalid Kubernetes manifests or missing required fields.
4. Monolithic Chart Sprawl
Explanation: Packing frontend, backend, database, and monitoring into a single chart creates tight coupling. Updating one component forces redeployment of the entire stack, increasing blast radius.
Fix: Decompose into subcharts or use Helmâs dependencies field. Deploy stateless services and stateful services separately when lifecycle requirements differ.
5. Ignoring Dependency Locking
Explanation: Failing to commit Chart.lock allows subchart versions to drift between environments. A chart that works in staging may break in production due to an upstream subchart update.
Fix: Run helm dependency update in CI, commit the lock file, and treat it as a dependency manifest equivalent to package-lock.json or go.sum.
6. Assuming Helm Upgrades Are Idempotent
Explanation: Helm tracks release history, but stateful resources (PersistentVolumeClaims, ConfigMaps) donât always reconcile cleanly on repeated upgrades. This causes resource conflicts or data loss.
Fix: Use helm upgrade --install only for initial deployments. For subsequent releases, use helm upgrade with explicit version tracking. Implement pre-upgrade hooks to pause traffic or drain connections when necessary.
7. Missing Post-Deployment Verification
Explanation: A successful helm upgrade exit code only means the API server accepted the manifests. It doesnât guarantee pods are healthy or endpoints are responding.
Fix: Implement readiness probes, liveness probes, and Helm test hooks. Integrate with monitoring systems to verify error rates and latency before marking a release as complete.
Production Bundle
This section provides actionable artifacts for implementing an engineered Helm workflow in production environments.
Action Checklist
Decision Matrix
| Scenario | Recommended Approach | Why | Cost Impact |
|---|
| Single application, small team | Monolithic chart with subcharts | Reduces pipeline complexity while maintaining logical separation | Low CI/CD overhead, moderate maintenance |
| Multi-tenant platform | Split charts per service + shared dependency chart | Isolates failure domains and enables independent release cycles | Higher initial setup, lower long-term incident cost |
| Regulated/Compliance environment | OCI registry storage + signed charts + audit pipeline | Ensures traceability, immutability, and policy enforcement | Increased storage costs, requires signing infrastructure |
| Rapid prototyping | Local helm template + kubectl apply | Fast iteration without pipeline overhead | High drift risk, not suitable for production |
Configuration Template
Use this baseline to bootstrap a production-ready Helm delivery pipeline. It includes schema validation, artifact signing, and environment promotion gates.
# .github/workflows/production-delivery.yml
name: Production Helm Delivery
on:
push:
tags: ['v*']
jobs:
build-and-sign:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Helm
uses: azure/setup-helm@v3
- name: Validate Values Schema
run: |
pip install jsonschema
jsonschema -i ./charts/platform-stack/values.yaml ./charts/platform-stack/values.schema.json
- name: Lint & Dry-Run
run: |
helm lint ./charts/platform-stack --strict
helm template prod ./charts/platform-stack --values ./charts/platform-stack/values.prod.yaml
- name: Package & Sign
run: |
helm package ./charts/platform-stack --destination ./dist
helm plugin install https://github.com/chartmuseum/helm-push
helm cm-push ./dist/*.tgz oci://registry.example.com/charts --sign --key my-signing-key
- name: Promote to Staging
run: |
helm upgrade platform-stack oci://registry.example.com/charts/platform-stack --version ${{ github.ref_name }} --namespace staging --wait --timeout 300s
Quick Start Guide
- Initialize the chart structure:
helm create platform-stack and replace the default templates with your service definitions.
- Define environment-specific values: Create
values.dev.yaml, values.staging.yaml, and values.prod.yaml to isolate configuration per deployment target.
- Add validation gates: Integrate
helm lint and helm template into your CI pipeline. Fail builds on syntax errors or missing required values.
- Package and publish: Run
helm package to generate a versioned artifact. Push to an OCI registry or chart repository with semantic version tags.
- Deploy with verification: Use
helm upgrade --install for initial releases, followed by test hooks and health checks to confirm service readiness before marking deployments complete.