Managing kubeconfig Files

Note: There is an updated version of this functionality described in a later post.

If you deal with Kubernetes, you are most likely familar with the kubeconfig file, typically located at ~/.kube/config. This file typically contains the clusters, users, and contexts (combinations of clusters and users) that you use in order to connect to our Kubernetes environments (typically with kubectl).

As well as using the default file at ~/.kube/config, there is also the option to specify a kubeconfig file using either the --kubeconfig argument, or the KUBECONFIG environment variable. In fact, the kubeconfig load order is (taken verbatim from the kubectl config docs:

The loading order follows these rules:

1. If the –kubeconfig flag is set, then only that file is loaded. The flag may only be set once and no merging takes place.
2. If $KUBECONFIG environment variable is set, then it is used a list of paths (normal path delimitting rules for your system). These paths are merged. When a value is modified, it is modified in the file that defines the stanza. When a value is created, it is created in the first file that exists. If no files in the chain exist, then it creates the last file in the list.
3. Otherwise, ${HOME}/.kube/config is used and no merging takes place.

While having all of the contexts you may need in one file is nice, it is difficult to maintain, and seldom the default case. Multiple tools which provide you with access credentials will provide a fresh kubeconfig to use. While you can merge the configs together into ~/.kube/config, it is manual, and makes removing contexts more difficult (having to explicitly remove the context, cluster, and user). There is an open issue in Kubernetes tracking this. However by keeping each provided config file separate, and just loading all of them, removal is much easier (just remove the file). To me, this seems like a much more manageable approach.

I prefer to keep all individual config files under ~/.kube/configs, and by taking advantage of the multiple-path aspect of the $KUBECONFIG environment variable option, we can make this happen. I threw together a quick zsh hook which dynamically sets the $KUBECONFIG environment variable before each command, but reading the contents of ~/.kube/configs and building the delimitted list.

The snippet is as follows:

function set-kubeconfig {
  # Sets the KUBECONFIG environment variable to a dynamic concatentation of everything
  # under ~/.kube/configs/*
  # Does NOT overwrite KUBECONFIG if the KUBECONFIG_MANUAL env var is set

  if [ -d ~/.kube/configs ]; then
    if [ -z "$KUBECONFIG_MANUAL" ]; then
      export KUBECONFIG=~/.kube/config$(find ~/.kube/configs -type f 2>/dev/null | xargs -I % echo -n ":%")
    fi
  fi
}

You’ll notice that I included an escape hatch; by setting the $KUBECONFIG_MANUAL variable, KUBECONFIG will not be automatically updated. This allows you to still set it to an arbitrary location if necessary.

As well, we hooked this into a zsh precmd hook on terminal setup:

add-zsh-hook precmd set-kubeconfig

While this is done using a zsh hook, Now our default kubeconfig will include any file thrown under ~/.kube/configs, and removing a context is as easy as deleting the file!

As a potential update, using the KUBECONFIG_MANUAL environment variable is tedious, and can get confusing. I’ll likely be updating the logic to not automatically set KUBECONFIG if it has been changed by any other means (ie, set manually).

Hopefully this works out for you, or was at least helpful. If you have any questions, don’t hesitate to shoot me an email, or follow me on twitter @nrmitchi.