Helm has long been most-used package manager for Kubernetes. It filled a gap for Kubernetes at a critical time back in 2016 when Google got involved in the project. We needed to make it easy to install off-the-shelf applications on Kubernetes to enable an ecosystem similar to what was developing around Docker Hub and what Mesosphere was developing around DCOS.

Operating systems had packages, and so did configuration management tools: Puppet Forge, Ansible Galaxy, and Chef Supermarket. So packages, and the community chart repo, for Kubernetes made sense.

Speaking of Ansible, it used Jinja YAML templates. So did Google Deployment Manager. Lots of CI systems at the time also used YAML. YAML and templates were believed to be more familiar and accessible to operators than, say, Python code, which Google Deployment Manager also supported. Kubernetes (and Docker and Etcd) was written in Go, so Go templates made sense from that perspective, as there wasn't a Jinja implementation in Go.

But since then, many alternative tools have been developed, focused mainly on the configuration generation functionality of Helm, as an alternative to YAML templates. It's not hard to find posts mentioning the complexity of Helm templates. I've written about it. In particular, templates for off-the-shelf packages can be extremely large and complex due to the desire to enable pretty much every field of every resource in each package to be customized.

There have been fewer tools targeting Helm's package management functionality, though there have been some. Glasskube is a recent one, which does address some key package problem areas, notably dependency management. But it wraps Helm and adds its own patching mechanism, so it doesn't really address the configuration complexity problem.

I recently wrote about how I converted a Helm chart to an AI agent skill. I liked a number of aspects of that experience, but it was challenging to understand what the agent would do.

So I created my own package installer based on configuration as data, with a goal of making the installation process simpler, more understandable, more automated, and more aligned with ConfigHub. It's open source and can be used without ConfigHub.

With help from AI, I was able to produce the bulk of the initial implementation in one week. It helped to be treading a relatively well paved path.

I already wrote a post about what a package is in configuration as code tools, so I won't rehash that here.

But what is a package installer? It's a tool that installs application packages on a target platform. The reason I didn't call it a package manager is because, while I do want to provide dependency management similar to what package managers for other systems do, I want to manage the lifecycle of the installed configuration differently (discussed more below).

I took inspiration from install wizards, like InstallShield, and from the experience of installing apps on my laptop. The experience is very simple, with at most a few high-level decisions about which components or features to install, and where to install the application, and with default values for all choices. Changes to detailed settings are made after installation rather than before or during the installation process. Post-installation changes can also be made to Debian package conffiles, specifically for the application configuration part of the package.

Enabling post-installation changes to be made to Kubernetes configuration such that they won't be clobbered upon upgrade requires managing the rendered configuration differently. Obviously, if the whole configuration is regenerated from scratch, naively, any changes to the previously generated configuration would be dropped. Managing configuration as data enables post-installation customization by storing the rendered, customized configuration, and by merging changes from rendered manifests.

The same approach enables decoupling variant management and operational changes from installation. Packages don't need to parameterize replicas, cpu and memory resources, environment variables, labels, annotations, and other standard Kubernetes fields because those can just be changed with later, with functions, AI agents, hand editing, or merging from the live state.

In the installer, choices of which "components" to install, where to install the application (i.e., the Kubernetes Namespace), and any other information that needs to be provided at install time are gathered by a "wizard", which can be used interactively or non-interactively (including driven by an AI agent), and are recorded in KRM files an out/spec directory.

apiVersion: installer.confighub.com/v1alpha1
kind: Selection
metadata:
  name: argocd-selection
spec:
  package: argocd
  packageVersion: 3.4.2
  base: cluster-install
---
apiVersion: installer.confighub.com/v1alpha1
kind: Inputs
metadata:
  name: argocd-inputs
spec:
  package: argocd
  packageVersion: 3.4.2
  namespace: argocd
  values: {}

I also took inspiration from Autoconf, curl | bash installers, Puppet Facter, Ansible Facts, and Nvidia AICR. These tools automatically gather information, such as the architecture of the target system, and use it to configure the installation process. That's a kind of automation that has been lost in the pursuit of hermetic and idempotent configuration generation, which itself is a problem that was created by repeatedly regenerating configuration using configuration as code formats.

The installer currently supports a local "fact" collection step, which can collect details about the workload being deployed and/or facts about the target cluster the application will be deployed to. It's local (for now, at least) so that it can use the context and credentials of the user. The facts are similarly recorded.

apiVersion: installer.confighub.com/v1alpha1
kind: Facts
metadata:
  name: confighub-worker-facts
spec:
  package: confighub-worker
  packageVersion: 0.1.0
  values:
    bridgeWorkerID: 6590e916-caee-4a3e-8096-2faeda20d04e
    configHubURL: https://hub.confighub.com
    image: ghcr.io/confighubai/confighub-worker:v0.1.46

Once choices, inputs, and facts are recorded, then the configuration is rendered using the recorded information, which makes it repeatable and idempotent unless inputs are changed deliberately.

I decided to use Kustomize to perform the rendering. The installer generates a kustomization.yaml file and uses the Components feature of Kustomize to combine the selected bases and components, which was a good fit for that part of the process. That also makes it possible to use existing kustomizations in the installer.

Also, while Kustomize is best known for patching configuration, it also has the ability to transform configuration programmatically. The installer uses the kustomize image transformer to facilitate overriding images specifically, because that's frequently needed for image mirroring, rollbacks, rolling out CVE fixes, and so on.

However, kustomize has a modest number of other built-in transformers, and I'm not aware of any popular transformer plugins or KRM functions. I wanted to enable more complex and robust configuration changes. So the installer transformer command serves as a Kustomize transformer plugin (as a native executable rather than as a container) that can invoke any built-in ConfigHub function, including functions written in Starlark, CEL, and yq. It is invoked automatically by the rendering process for transformers and validators specified in the Package spec.

  transformers:
    - toolchain: Kubernetes/YAML
      whereResource: ""
      description: Set the namespace on every namespaced resource and on RBAC subjects.
      invocations:
        - name: set-namespace
          args: ["{{ .Namespace }}"]

I also wanted to be able to validate the rendered configuration. At the moment, that's done in the installer after kustomize rendering completes.

  validators:
    - toolchain: Kubernetes/YAML
      description: Default validators applied at the end of every render.
      invocations:
        - name: vet-schemas # runs kubeconform
        - name: vet-merge-keys # checks for duplicate merge keys, e.g. envs
        - name: vet-format # yaml linter

All of these steps can be invoked via a single installer setup command. Installation with default bases, components, and inputs, if any, is concise:

% installer setup --work-dir argo-install --namespace argocd \
    --pull ${installer_dir}/packages/argocd --non-interactive

A record of what the installer did, functions invoked with expanded arguments and resource files generated, is also written. For now, this is mostly for debugging and auditing.

apiVersion: installer.confighub.com/v1alpha1
kind: FunctionChain
metadata:
  name: argocd-function-chain
spec:
  package: argocd
  packageVersion: 3.4.2
  groups:
    - toolchain: Kubernetes/YAML
      invocations:
        - name: set-namespace
          args:
            - argocd
      description: Set the namespace on every namespaced resource and on RBAC subjects.
---
apiVersion: installer.confighub.com/v1alpha1
kind: ManifestIndex
metadata:
  name: argocd-manifest-index
spec:
  package: argocd
  packageVersion: 3.4.2
  files:
    - filename: clusterrole-argocd-application-controller.yaml
      slug: clusterrole-argocd-application-controller
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      name: argocd-application-controller
...
    - filename: statefulset-argocd-argocd-application-controller.yaml
      slug: statefulset-argocd-argocd-application-controller
      apiVersion: apps/v1
      kind: StatefulSet
      name: argocd-application-controller
      namespace: argocd

Similar to Kustomize, installer has an edit command for modifying the Package spec, so that the YAML doesn't have to be edited by hand in an editor, though it can be.

Secrets, if any, are output to a different directory out/secrets from the other manifests, which are output to out/manifests. There's more work to do on secret support in the future. The rendered manifests could then be uploaded to ConfigHub using installer upload , or applied directly with kubectl or your favorite GitOps tool.

For those of you who use kpt or want to try kpt, I added a kpt guide. The role of kpt in the guide is to manage post-installation changes. Like ConfigHub, kpt can merge fully rendered, WET Kubernetes YAML, so it can be used to preserve changes after re-rendering with kustomize for upgrades and changed installation choices. That requires committing (and pushing and tagging) the rendered manifests and copying them and propagating updates with kpt pkg commands.

Why not just use kustomize or kpt? Neither tool was designed to be a full-blown installer. For example, they don't have interactive wizards, component selection, fact collection, and so on.

Additionally, package management was explicitly out of scope for kustomize. kpt is git-centric (ConfigHub does not store configuration in git), doesn't support OCI-based packages (issue), and doesn't do dependency management. I also wanted to compose packages rather than statically nest them. I'll write another post about installer's package functionality, and another about kpt and its relation to kustomize.

So, the installer is DRY-ish (DAMP? — need a good backronym for that), because installers of off-the-shelf packages involve choices, computation, and, sometimes, non-hermetic processes. Luckily, with configuration as data (and ConfigHub), configuration can be authored using whatever mechanism is most effective, such as an install wizard or AI agent or GUI or importing it from a cluster, and can be modified directly in its fully rendered form after that, over time.

Decoupling the stages of customization (installation, post-installation, variant creation, operational changes, etc.), keeping the configuration fully rendered (WET) at every step, enabling tools to operate on the configuration, and maintaining relationships between variants can simplify configuration management compared to a monolithic template or generator that needs to incorporate all needs in one generation pass. Future posts will explore this approach in more depth.

I didn't have space in this post to cover all of the installer's features (e.g., application configuration), but hopefully this gave you a sense of why I created it and how it's different.

Have you tried to use kustomize or kpt for package installation as an alternative to Helm? If you've tried Glasskube, what did you think? Have you tried another package manager? What would you like to see in a new Kubernetes package installer / manager? Give the installer a try and let me know what you think.

You are also welcome to try out ConfigHub, which is now in preview.

Feel free to reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

If you found this interesting, you may be interested in other posts in my Kubernetes series.