Helm vs Kustomize vs CDK8s

Every Kubernetes team arrives at the same problem about six weeks in: raw manifests stop scaling. The first namespace gets cloned to staging, then prod, then a second region, then a per-tenant variant, and suddenly there are twelve copies of the same Deployment differing by three fields each. Something has to generate the YAML. The question is what, and the answer has been contested for almost a decade because the three leading approaches — Helm, Kustomize, and CDK8s — represent genuinely different philosophies about how configuration should be authored.

The choice matters. It shows up in onboarding time, in the number of times somebody has to debug a yaml.Marshaler error, in how cleanly you can produce a per-environment diff, and in whether a junior engineer can safely change an image tag in production. A wrong choice is recoverable but painful — repository-wide migrations from Helm to Kustomize are multi-quarter projects.

This chapter walks the templating spectrum end to end. What each tool is, what it’s good for, what it’s bad for, and the rule of thumb for when to reach for which. By the end, picking the right tool for a given service should take about thirty seconds.

Outline:

1. The YAML generation problem.
2. The templating spectrum — strings, overlays, code.
3. Helm — Go templates over strings.
4. Kustomize — overlays and strategic merge.
5. CDK8s — real code that emits YAML.
6. Pulumi and the wider IaC overlap.
7. Comparison table, by axis.
8. The rule of thumb.
9. Gotchas from the field.
10. The mental model.

---

108.1 The YAML generation problem

Start with the shape of the pain. A service has a Deployment, a Service, a ServiceAccount, a ConfigMap, maybe a HorizontalPodAutoscaler, an Ingress, and an IAMRole mapping. That’s seven YAML files for one service. Now there are ten services. That’s seventy files. Now there are three environments (dev, staging, prod), and each environment has slightly different resource limits, image tags, replica counts, and hostnames. You are now tracking 210 files, most of which are 95% identical.

You cannot copy-paste. The moment a field changes shape — a new required environment variable, a pod security context field becoming mandatory, a sidecar container — you have to update it in every copy. You will forget one. The forgotten one will ship to production on a Friday.

The fix has to do two things. First, deduplicate the shared structure so you write the common bits once. Second, parameterize the variation so the per-environment deltas are explicit, small, and reviewable. Everything that follows — Helm, Kustomize, CDK8s, Pulumi, plain Jsonnet, the home-grown Python script your previous employer had — is an answer to this one problem.

The three major schools of thought differ on where in the stack the deduplication happens:

• String templating. Treat YAML as text. Inject values via a templating language. Helm is the canonical example.
• Overlay composition. Treat YAML as structured data. Merge a base with patches that override specific fields. Kustomize is the canonical example.
• Code. Give up on YAML as the source of truth. Write programs in a real language that emit the YAML. CDK8s and Pulumi.

Each sacrifices something different, and each is right for different workloads.

108.2 The templating spectrum

A picture in words. Arrange the approaches left-to-right by how abstract the source is:

raw YAML ──▶ Kustomize ──▶ Helm ──▶ CDK8s/Pulumi ──▶ full operator
  (none)     (structure)   (strings)    (code)          (runtime)

On the left, the source material and the output are the same thing — the YAML you write is the YAML that ships. On the right, the source is a program that generates YAML at synth time, and the generated YAML may not even be human-readable. Each step right buys more power at the cost of transparency.

A useful second axis: who can read it without a runbook? Raw YAML: anyone. Kustomize: anyone who’s seen it once. Helm: nobody reads {{- range .Values.env }}{{ $key := .name }}{{ end }} for pleasure. CDK8s: only engineers who know the host language and the constructs library.

The spectrum is also a testing spectrum. Kustomize is trivially testable — run kustomize build and diff the output against a golden file. Helm is testable but has templating edge cases that bite (whitespace, type coercion, nil handling). CDK8s is testable with real unit tests in the host language, which is the most powerful but also the most machinery.

Teams rarely stay in one place on this spectrum. A common evolution: start with raw YAML, outgrow it, reach for Kustomize, discover that Kustomize can’t express some branching logic, reach for Helm for the bits that need parameterization, and eventually adopt CDK8s for the highest-stakes services. Mixing is fine — it’s the healthy end state, not a failure mode.

108.3 Helm — Go templates over strings

Helm is the incumbent. It was the first to reach critical mass, and it remains the default distribution format for third-party software on Kubernetes. If a vendor ships a chart, the chart is almost always a Helm chart.

The Helm model: a chart is a directory containing a Chart.yaml (metadata), a values.yaml (default parameters), and a templates/ directory holding Go template files that emit Kubernetes YAML. You install a chart with helm install my-release ./chart -f values-prod.yaml, optionally passing additional --set overrides. Helm renders the templates, applies the resulting manifests, and records the release in a state secret in the target namespace.

A trivial template fragment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-app.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          {{- if .Values.resources }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          {{- end }}

The {{- ... }} trimming directives, the nindent for whitespace control, the include for shared template partials, and the toYaml for rendering a sub-structure — these are the Helm vocabulary. It’s Go templates with a handful of Kubernetes-specific helpers.

What Helm is good at. Distributing parameterized software. If your chart will be installed by strangers in unknown environments, Helm is the right tool. It has first-class values files, built-in dependency management (subcharts), hooks for ordering (pre-install, post-upgrade), and a revision/rollback mechanism via release history. The ecosystem is enormous — bitnami/charts, the vendor-published charts for every major database and tool, the helm lint and helm test workflows.

What Helm is bad at. String templating over structured data is a category error. You are writing Go template code that emits YAML, which then gets parsed as structured data by the API server. Any bug that produces malformed YAML manifests as a parse error at the worst possible time. Indentation is manual (nindent 12), which means off-by-two whitespace bugs. Type coercion is implicit and wrong — a value like "01" will be interpreted as an integer if you’re not careful. Conditional logic inside nested templates gets unreadable fast. And the state model (a release stored in a Kubernetes secret) is its own problem — state gets out of sync, helm upgrade fails with cryptic errors, and operators learn to reach for helm secrets or full deletion-and-reinstall as a recovery pattern.

The rule of thumb: use Helm when you are distributing a chart to other people, or when you are consuming one from a vendor. Don’t use Helm to template your own internal services unless you have a specific reason. There are better tools for that.

108.4 Kustomize — overlays and strategic merge

Kustomize is the second major answer, and it’s the one that ships inside kubectl itself (kubectl apply -k). The philosophy is the opposite of Helm’s: no templating, no variables, no logic. You write plain Kubernetes YAML and compose it with merge-based overlays.

The model. A base/ directory contains the canonical YAML for a resource:

graph TD
  Base["base/<br/>deployment.yaml<br/>service.yaml"] -->|extended by| Dev["overlays/dev/<br/>replicas: 1, small CPU"]
  Base -->|extended by| Staging["overlays/staging/<br/>replicas: 3, medium CPU"]
  Base -->|extended by| Prod["overlays/prod/<br/>replicas: 10, large CPU"]
  style Base fill:var(--fig-accent-soft),stroke:var(--fig-accent)

Kustomize’s overlay model writes the common structure once in base/ and patches only the per-environment deltas — the diff between environments is always explicit and reviewable as a patch file.

# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: app
          image: ghcr.io/org/my-app:latest
          resources:
            requests: { cpu: 100m, memory: 128Mi }

And an overlays/prod/ directory patches it:

# overlays/prod/kustomization.yaml
resources:
  - ../../base
images:
  - name: ghcr.io/org/my-app
    newTag: v1.4.2
replicas:
  - name: my-app
    count: 10
patches:
  - path: resources-patch.yaml

# overlays/prod/resources-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            requests: { cpu: 2, memory: 8Gi }
            limits:   { cpu: 4, memory: 16Gi }

Running kustomize build overlays/prod emits the merged YAML. The base has no knowledge that the overlay exists; the overlay explicitly names what it’s changing. Everything is plain Kubernetes YAML, readable without tooling.

What Kustomize is good at. Environment overlays for services you own. The common case — “my service is mostly the same across dev/staging/prod but the replica count, resource requests, and image tag differ” — is exactly what Kustomize was designed for. The base YAML is readable. The patches are small and reviewable. The diff between environments is obvious because you can see it in the patch file. kustomize build is deterministic and testable. Integration with ArgoCD and Flux is first-class (Chapter 107).

What Kustomize is bad at. Parameterization that isn’t structural. If you need conditional resources (“deploy the Ingress only when external=true”), Kustomize can do it with components but the ergonomics are awkward. If you need to compute values from other values (“memory should be twice cpu”), you cannot. If you need to generate many similar resources from a list, you’re writing a lot of boilerplate. The strategic merge semantics are also subtle — merging lists vs replacing them, patch target resolution, the difference between JSON patches and strategic merges — and the corner cases produce “why didn’t my patch apply” debugging sessions.

Kustomize’s biggest real weakness is that it cannot generate anything beyond what’s in the base. It’s purely subtractive/mutative. If you need fan-out — “create one Deployment per entry in this list of tenants” — you need something else.

The rule of thumb: use Kustomize for internal service manifests across environments. It’s the default choice for first-party services. Combine it with GitOps (ArgoCD Applications that point at an overlay directory) and you have a clean, reviewable, auditable deployment model.

108.5 CDK8s — real code that emits YAML

CDK8s (Cloud Development Kit for Kubernetes) is the third school. Its pitch: YAML is a bad serialization format to author in; use a real programming language instead. You write TypeScript, Python, or Go that uses a constructs library (cdk8s-plus) to describe Kubernetes resources, then run cdk8s synth to emit YAML, which you then apply via kubectl or pipe into your GitOps system.

A TypeScript example:

import { App, Chart } from 'cdk8s';
import { Deployment, Service } from 'cdk8s-plus-27';

class MyAppChart extends Chart {
  constructor(scope: Construct, id: string, props: MyAppProps) {
    super(scope, id);

    const deployment = new Deployment(this, 'app', {
      replicas: props.replicas,
      containers: [{
        image: `ghcr.io/org/my-app:${props.tag}`,
        resources: {
          cpu: { request: Cpu.millis(props.cpuMillis) },
          memory: { request: Size.mebibytes(props.memMib) },
        },
        envVariables: props.env,
      }],
    });

    if (props.public) {
      deployment.exposeViaService({ serviceType: ServiceType.LOAD_BALANCER });
    }
  }
}

const app = new App();
new MyAppChart(app, 'prod', { replicas: 10, tag: 'v1.4.2', cpuMillis: 2000, memMib: 8192, public: true, env: {...} });
new MyAppChart(app, 'staging', { replicas: 2, tag: 'v1.4.2', cpuMillis: 500, memMib: 2048, public: false, env: {...} });
app.synth();

Notice what this buys. The if (props.public) is real conditional logic. The Cpu.millis and Size.mebibytes constructors are type-safe — you cannot pass "2Gi" where an integer is expected. Shared utilities are just functions. You can write unit tests that assert the synthesized YAML has specific properties. Refactoring is IDE-supported. And the input schema is a TypeScript interface, not an untyped values.yaml.

What CDK8s is good at. Complex, dynamic composition. Multi-tenant platforms where each tenant gets a slightly different set of resources. Internal platforms that generate many similar stacks. Anywhere the configuration is itself computed. Type safety catches a huge class of bugs at author time — wrong field names, missing required fields, type mismatches — that Helm and Kustomize catch at apply time or later.

What CDK8s is bad at. Readability for outsiders. A TypeScript file that emits Kubernetes YAML is opaque to anyone who doesn’t know the host language and the constructs library. Debugging means understanding both the input code and the generated output. The toolchain is heavier: npm install, a build step, a synth step before anything hits the cluster. The community is smaller than Helm’s or Kustomize’s, and the ecosystem of shared constructs is still thin. And the “code that emits data” pattern has its own gotchas — spooky action at a distance when shared constructs mutate state, synth-time errors that look nothing like Kubernetes errors.

The rule of thumb: use CDK8s when you’re building a platform that generates many variants and the variation cannot be expressed cleanly as overlays. Think internal developer platforms, multi-tenant control planes, or anywhere an engineer would otherwise write a Python script that shells out to helm template.

108.6 Pulumi and the wider IaC overlap

CDK8s has a sibling worth mentioning: Pulumi. Pulumi is the same “real code that emits infrastructure” idea, but it operates at the IaC layer (AWS, GCP, Azure) rather than the Kubernetes layer. It has a Kubernetes provider that can apply manifests, templates, or CDK8s charts directly, and its state management is more sophisticated than Helm’s.

Pulumi and CDK8s overlap awkwardly. You can use Pulumi with its native Kubernetes provider and never touch CDK8s. You can use CDK8s to synth YAML and apply it via anything (kubectl, ArgoCD, Pulumi). The Pulumi-plus-Kubernetes-provider approach gives you one tool for everything — cloud infra and cluster resources in the same stack, with the same state backend. For a small team, that’s attractive.

The tradeoff is lock-in. Pulumi state lives in the Pulumi Cloud or a self-hosted backend, and migrating out is non-trivial. Helm state lives in Kubernetes secrets and migrating out is easy. Kustomize has no state at all, which is the best of all worlds from a lock-in perspective.

IaC proper — Terraform, Pulumi, AWS CDK — is the subject of Chapter 110. For the Kubernetes-specific authoring question, the three options stay: Helm, Kustomize, CDK8s.

108.7 Comparison table, by axis

Axis	Helm	Kustomize	CDK8s
Source material	Go-templated YAML	Plain YAML + patches	Host-language code
Type safety	None	None (structural)	Full (at synth)
Readable by strangers	Medium	High	Low
Conditional logic	Yes, gnarly	Limited (components)	Yes, trivial
Loops / fan-out	Yes, gnarly	No	Yes, trivial
Composition model	Subcharts	Overlays	Constructs
State management	Release secrets	None	None
GitOps integration	Good	Excellent	Good (via synth)
Ecosystem size	Enormous	Large	Small
Testability	Limited	Golden-file diffs	Unit tests
Distribution to third parties	Excellent	Poor	Poor
Typical use case	Vendor software	Your services	Internal platforms
Learning curve	Medium	Low	High

Reading this table, the pattern is clear. No tool dominates. Each axis has a winner and a loser, and the right choice depends on which axes you care about. For a typical engineering team shipping first-party services, Kustomize wins on every axis that matters to that team — readability, testability, GitOps integration, learning curve. That’s why it’s the default recommendation. Helm and CDK8s win on axes that matter when the use case is different.

108.8 The rule of thumb

Memorize this:

• Helm, when you’re distributing a chart to other people, or installing someone else’s chart. Vendor software goes here. Internal shared infrastructure (Prometheus stack, cert-manager, ingress controllers) is almost always installed via vendor Helm charts.
• Kustomize, when you’re managing your own services across environments. First-party applications go here. This is the majority of your repository.
• CDK8s (or Pulumi with the Kubernetes provider), when you’re generating many variants programmatically — platform tools, per-tenant deployments, anywhere a shell script over helm template would otherwise appear.

A healthy repository has all three. Kustomize for services. Helm for vendor components (with the Helm chart referenced from a Kustomize overlay that patches values — this is a supported pattern and ArgoCD handles it natively). CDK8s for the internal developer platform that generates manifests from a high-level service spec.

The common anti-pattern: using Helm for everything because it was the first tool the team learned. This produces chart repos full of dense Go templates, and the lesson eventually arrives the hard way — usually when a templating bug breaks production and the room spends an hour debugging whitespace.

108.9 Gotchas from the field

A short list of things that have cost real production time.

Helm and strategic merge on lists. Helm can’t strategic-merge. If a chart exposes extraEnv as a list, appending to it from a values override replaces the list entirely. Charts that get this right expose structured maps, not lists.

Helm hook ordering and pre-install jobs. Helm hooks (pre-install, post-upgrade, etc.) run at specific lifecycle points, but the ordering guarantees are weaker than they look. Hooks run in parallel within a phase unless you set helm.sh/hook-weight. Debugging a hook that ran before its dependency is a special kind of frustrating.

Kustomize patch target resolution. Patches are matched against resources by kind + name + namespace. If you rename a resource in the base, every patch targeting it silently becomes a no-op. Kustomize emits a warning, but the warning is easy to miss in CI.

Kustomize generators and the disableNameSuffixHash. ConfigMap and Secret generators append a hash suffix by default (to force pod rollouts when content changes). This is usually what you want. When it isn’t, turning it off changes the update semantics and can cause missed rollouts.

CDK8s synth nondeterminism. If your CDK8s code reads external state (environment variables, filesystem, network), two synths can produce different YAML. This breaks GitOps. Keep synth pure.

CDK8s and cdk8s-plus version skew. cdk8s-plus-27 targets Kubernetes 1.27. Using it against a 1.30 cluster mostly works, but new fields won’t be available. Track the version of cdk8s-plus you’re using and bump with care.

The universal gotcha: whitespace in generated YAML. Every templating tool has a whitespace edge case. Helm has nindent. Kustomize has list-vs-map merge semantics. CDK8s has the constructs library generating unexpected field orderings. Always round-trip generated YAML through kubectl apply --dry-run=client before trusting it.

108.10 The mental model

Eight points to take into Chapter 109:

1. Three philosophies: string templating (Helm), overlay merging (Kustomize), real code (CDK8s). Everything else is a variation.
2. Helm for distribution, especially vendor-published charts. Don’t use it for internal services unless you have a reason.
3. Kustomize for environment overlays of your own services. The default choice.
4. CDK8s for programmatic generation when the variation cannot be expressed as overlays. Type safety is the real benefit.
5. Mixing is correct, not a failure mode. Kustomize overlays over Helm charts is a first-class pattern.
6. Readability decays from left to right on the spectrum. So does the learning curve.
7. Every tool has whitespace or merge gotchas. Round-trip through kubectl apply --dry-run=client in CI.
8. Helm state (release secrets) is its own hazard. Kustomize and CDK8s have none, which is a feature.

In Chapter 109, the question shifts from “how do you generate the manifests” to “where do they run.” Multi-cluster, multi-region, multi-cell architecture.

---

Read it yourself

• The Helm documentation, especially the chart best-practices guide and the hooks reference.
• The Kustomize reference, kubectl integration documentation, and the strategic-merge patch spec.
• The CDK8s documentation and the cdk8s-plus constructs library.
• Production Kubernetes (Rosso et al., O’Reilly, 2021) — the chapters on application management and delivery.
• Ellen Körbes and Brendan Burns on YAML authoring, various KubeCon talks.
• The ArgoCD documentation on Helm + Kustomize combined sources.
• The Pulumi Kubernetes provider documentation.

Practice

1. Take a single-file Deployment + Service YAML and convert it to a Kustomize base + three overlays (dev, staging, prod). Verify kustomize build produces the expected output for each.
2. Write a Helm chart that deploys the same service, with values-dev.yaml / values-prod.yaml. Compare lines-of-YAML with the Kustomize version.
3. Write the same service as a CDK8s chart in TypeScript or Python. Emit YAML for all three environments.
4. For each of the three versions above, intentionally introduce a typo in a field name. Which tools catch it at author time? Which at apply time? Which never?
5. Show how to consume a vendor Helm chart (e.g., ingress-nginx) from a Kustomize overlay that patches the values. Verify the merge.
6. Construct a case where Kustomize cannot express the required variation, and argue why CDK8s is the right tool.
7. Stretch: Build a CDK8s construct that takes a high-level ServiceSpec (with fields like name, image, replicas, public, dependencies) and emits a full Deployment + Service + ServiceAccount + HPA + NetworkPolicy. Write unit tests asserting the generated resources have the expected fields.