We Contribute: Keeping the DNSimple cert-manager Webhook Alive

In a time where automation applies to pretty much all facets of IT, from securing supply chains all the way to networking, managing TLS certificates is also a part of this.
In times where the industry consensus on certificate validation periods is shifting towards shorter TTLs, this essentially becomes non-negotiable anyway.

Now, if we were still living in the „good old“ days of certificates with lifetimes of several years, one could simply order a certificate through the web interface of a certificate authority and call it a day. Alas, those days are over (and for good reason), so we must now think of ways to automate this process – because who would want to click around in a GUI every few months.

Luckily, both certificates and automating them is made very accessible by projects such as Let’s Encrypt and cert-manager. To verify ownership of a domain, certificate authorities usually want you to solve one of two ACME challenges: HTTP01 and DNS01. At Puzzle, we are exclusively solving DNS01 challenges, meaning that a CA such as Let’s Encrypt must be able to resolve a specific TXT-record in a target domain to be able to verify ownership, after which it will resume the certificate issuance.

DNS as code

Many DNS providers now offer APIs that allow users to manage DNS zones programmatically instead of clicking around in a web interface. One such provider is DNSimple which we have been using for a long time now.

This brings us back to cert-manager, an arguably indispensable component of many Kubernetes clusters. In short, it enables the management of certificates as Kubernetes objects.

Cert-manager also supports ACME and is able to solve ACME challenges by itself.
However, in the case of DNS01, cert-manager can only do this type of challenge with a limited set of providers by default.
To extend support to other providers, users must install additional webhooks, most of which are community efforts. Thankfully, someone did create a webhook for DNSimple some time ago.

We contribute

Unfortunately, this webhook – created and maintained by one person – started receiving fewer and fewer commits and was at risk of eventually becoming incompatible with newer versions of cert-manager.
A common fate for projects with a single maintainer, because sooner or later life and/or work will inevitably force us to rethink how and where we spend our time.
Some projects will simply take a backseat.

One of Puzzle’s core values is “We contribute”, and in its spirit we saw an opportunity to adopt the project because we also rely on this software and have an interest in keeping it maintained and compatible with future releases. Many thanks go out to Arne Diekmann for bringing the webhook to life!

Now, a team of engineers who all have an interest of keeping this project running are taking turns at maintaining the code. The project is largely feature complete, so maintenance primarily consists of keeping dependencies up to date and addressing occasional issues reported by users.

To keep track of dependencies we’ve opted to use Renovate bot as it is Puzzle’s dependency updater tool of choice and the one we are most comfortable with.
To quickly check the compatibility of potential dependency updates, we’ve worked on creating robust E2E tests that cover everything from building the webhook to issuing a certificate in a real Kubernetes cluster using multiple versions of Kubernetes and cert-manager.

Lessons learned

To increase development velocity, we’ve learned the importance of robust E2E testing and doing tests with multiple versions of the software. For instance, we test against 3 versions of Kubernetes and 2 versions of cert-manager respectively. Although there is not really that much code to test, it’s most likely the (kubernetes client) go packages that introduce issues;

obs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        k8s-version: ["1.33", "1.34", "1.35"]
        cm-version: ["1.18.6", "1.19.4"]

With such a testing framework in place, we can now afford to let Renovate open patch PRs and automerge them if the tests pass.

However, merging package updates isn’t enough. One must also eventually ship those updates. Because the code footprint is quite small and the “biggest” maintenance task is dependency updates, we have opted to automate the release process as well – at least to a degree:
Every month, a CI job checks whether the only PRs merged since the last release are automerged Renovate dependency updates. If so, it creates a new image tag and Helm chart version to ship the update to users.
Otherwise, one of our engineers must take manual action to create an appropriate release.
(As will be done anyway if a bigger feature or bugfix is introduced either by us or a contributor.)

In the end, we’re happy to have adopted this project and to put Puzzle’s „We contribute“ philosophy into practice in this way.