Using Helmfile to Wrangle your Kubernetes Releases

The Problem with Vanilla Helm

Helm is a great tool for templating kubernetes manifests. We use helm a LOT here at Novus and we typically organize our charts as follows:

∮ tree .
.
├── Chart.yaml
├── README.md
├── helmfile.yaml
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ └── configmap.yaml
│ ├── cronscaler.yaml
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── pv.yaml
│ ├── pvc.yaml
│ ├── secrets.yaml
│ ├── service.yaml
│ └── servicemonitor.yaml
└── vars
  ├── dev
  │ ├── secrets.yaml
  │ └── values.yaml
  ├── prod
  │ ├── secrets.yaml
  │ └── values.yaml
  ├── qa
  │ ├── secrets.yaml
  │ └── values.yaml
  ├── sandbox
  │ ├── secrets.yaml
  │ └── values.yaml
  └── values.yaml

As you can see, our charts have a shared values.yaml and environment specific files with corresponding overrides. We release to production very frequently, so rather than update our helm chart constantly we override the container image tag through an environment variable. If we were to just use helm we would need to run the following to upgrade our alpha-app release

∮ helm secrets upgrade --install -f vars/values.yaml -f vars/qa/values.yaml -f vars/qa/secrets.yaml --set image.tag=stable alpha-app .

Oof, thats a lot of typing. Even worse, this command is very prone to user error. The settings in later files override the earlier ones, so if you accidentally mix the order of the files, you could deploy the default configuration instead of the qa one.

Moreover, our application has several layers, each of which has its own chart. If we wanted to spin up a full stack environment for testing, we would need to release our frontend, backend, workers, as well as activemq (our message broker) in separate commands. That's a lot of helm!

Helmfile to the Rescue

Helmfile offers a declarative way to define your helm releases. Since our charts all follow roughly the same directory structure, we can easily define a general purpose helmfile.yaml that we can add to our charts.

helmfile.yaml
---

# Define deployment environments
environments:
    dev:
    qa:
    sandbox:
    prod:


# Define helm releases
releases:
  - name: alpha-app
    namespace: default
    chart: .
    values:
      - ./vars/values.yaml
      - ./vars/{{ .Environment.Name }}/values.yaml
    secrets:
      - ./vars/{{ .Environment.Name }}/secrets.yaml

With this concise configuration file, we have a solution to our first issue! We no longer have to remember that long list of values files, arranged in precise order. Even better, we can now easily switch between qa and prod releases. The long upgrade command above now becomes

 ∮ helmfile -e qa sync

or, in the production environment

 ∮ helmfile -e prod sync

Wow, that's so much easier to remember! Every release is the same as the release name and details are all defined in the helmfile.yaml.

Diff Support out of the Box

Helm has an excellent diff plugin which makes it easy to see exactly what you are changing in an upgrade. Combined with the changes in helm3 you can use it to declaratively set the state of a helm release. Fortunatley, we don't lose that functionality with helmfile!

∮ helmfile -e prod diff --context 4
skipping missing secrets file matching "./vars/prod/secrets.yaml"
Comparing release=alpha-redis, chart=.
redis, alpha-redis-master, StatefulSet (apps) has changed:
...
					limits:
							cpu: 2
							memory: 27Gi
					requests:
-							cpu: 1
+							cpu: 2
							memory: 27Gi
					volumeMounts:
						 - name: start-scripts
							mountPath: /opt/bitnami/scripts/start-scripts
...

‍TIP: use the --context 4 flag on your diff commands so that it only prints the relevant part of the file

Helmfile also provides a handy command called helmfile apply that terraform lovers may find familiar. By running it with the interactive mode flag (-i), you can get the diff of your release and immediately run the release if you like it:

∮ helmfile -e prod -i apply --context 4
skipping missing secrets file matching "./vars/prod/secrets.yaml"
Comparing release=alpha-redis, chart=.
redis, alpha-redis-master, StatefulSet (apps) has changed:
...
					limits:
						cpu: 2
						memory: 27Gi
					requests:
-						cpu: 1
+						cpu: 2
						memory: 27Gi
					volumeMounts:
					 - name: start-scripts
			 		   mountPath: /opt/bitnami/scripts/start-scripts
...
Affected releases are:
	alpha-redis (.) UPDATED

Do you really want to apply?
	Helmfile will apply all your changes, as shown above.

[y/n]:

Multiple Releases

But what if we only want to deploy 1 or two of our charts? Again, helmfile has the answer: labels. By putting our releases into one helmfile.yaml and giving them labels we can release the whole stack or only the portions we need.

# Define deployment environments
environments:
  dev:
  qa:
  sandbox:
  prod:

# Define helm releases
releases:
- name: activemq-artemis
	labels:
		tier: message
  chart: ../activemq-artemis/
  namespace: default
  missingFileHandler: Info
  values:
		- "../activemq-artemis/vars/values.yaml"
		- "../activemq-artemis/vars/{{ .Environment.Name }}/values.yaml"
  secrets:
		- "../activemq-artemis/vars/{{ .Environment.Name }}/secrets.yaml"
- name: aloha
  labels:
  	tier: frontend
  chart: ../aloha/
  namespace: default
  missingFileHandler: Info
  values:
		- "../aloha/vars/values.yaml"
		- "../aloha/vars/{{ .Environment.Name }}/values.yaml"
  secrets:
    - "../aloha/vars/{{ .Environment.Name }}/secrets.yaml"
  set:
  	- name: image.tag
      value: {{ env "ALOHA_TAG" | default "stable" | quote }}
- name: alpha-app
  wait: true
  timeout: 600
  labels:
  	tier: backend
  chart: alpha-app/
  namespace: default
  missingFileHandler: Info
  values:
  	- "alpha-app/vars/values.yaml"
    - "alpha-app/vars/{{ .Environment.Name }}/values.yaml"
  secrets:
  	- "alpha-app/vars/{{ .Environment.Name }}/secrets.yaml"
  set:
  	- name: image.tag
      value: {{ env "ALPHA_TAG" | default "stable" | quote }}
- name: analytics-worker
  labels:
  	tier: backend
  chart: analytics-worker/
  namespace: default
  missingFileHandler: Info
  values:
    - "analytics-worker/vars/values.yaml"
    - "analytics-worker/vars/{{ .Environment.Name }}/values.yaml"
  secrets:
    - "analytics-worker/vars/{{ .Environment.Name }}/secrets.yaml"
  set:
  	- name: image.tag
      value: {{ env "ALPHA_TAG" | default "stable" | quote }}
  

With the above configuration, we can control 4 different releases from a single helmfile.

 ∮ helmfile -e qa -l tier=backend sync

This command runs the upgrade on just the last two (backend) releases.

TIP: I also defined the image tags in the release using {{ env "ALPHA_TAG" | default "stable" | quote }}. This lets me easily set those tags using an env variable and not clutter the helmfile command with extra --set flags. This is especially handy in the CI/CD server since we are already passing around environment variables there for settings.

Some More Cool Features 😎

• If you have a lot of releases, it can be useful to separate them into multiple files and put them all into a helmfile.d/ directory. This is supported out of the box.
• You can define shell commands to run inside of the helmfile templates. We use it to set an environment variable that lets us know what configuration was used to release:

 set:
	- name: vars.normal.ALPHA_CONFIG_SHA
	value: {{ exec "git" (list "rev-parse" "--short" "HEAD") | trim | default "Unknown" | quote }}

• If you are using multiple versions of helm on your system, you can define which helm binary you want to use using -b /path/to/helm. This can also be defined directly in the helmfile.yaml
• helmfile also supports release dependencies for multiple releases. By defining needs: on a release, you can have that release wait for another to finish before proceeding. We use this to spin up pods in the right order so that they are not trying to form connections with not yet existing other services.