Tasdik Rahman

tail -f /var/log/life*

Hosted with ♥ on Github

© 2015-2022 All rights reserved.

Introducing Kingsly — The Cert Manager

22 Apr 2020 #security #x509 #devops #ruby #rubyonrails #letsencrypt #oss Introducing Kingsly — The Cert Manager

This was originally published under Gojek’s engineering blog by me, this post is a repost.

There’s one thing all devices connected to the Internet have in common — they rely on protocols called SSL/TLS to protect information in transit.

SSL/TLS are cryptographic protocols designed to provide secure communication over insecure infrastructure.

Any communication over the public internet should be encrypted, for which we need SSL certificates. There are many cases for public communication in GOJEK as well. Some of them are listed below:

While the industry is moving towards a zero trust network, certificate management has been a big pain for us.

This post details how we built Kingsly, GOJEK’s open source certificate management tool.

Managing SSL/TLS certs until yesterday

A certificate is a digital document that contains a public key, some information about the entity associated with it, and a digital signature from the certificate issuer.

In other words, it’s a shell that allows us to exchange, store and use public keys. With that, certificates become the basic building block of the Public Key Infrastructure

We use letsencrypt heavily at GOJEK, to generate SSL/TLS certificates for numerous use cases, which can range from putting them behind our HAproxy Loadbalancers, envoy proxies, IPSec VPN’s and a lot of other places.

As of the end of 2018, the whole setup of renewing the certificates (it’s basically regenerating the certificate in the case of letsencrypt) installed at different places, was a manual process.

All letsencrypt SSL certificates, including renewals, are valid for no more than 90 days from their issue date. Thirty days before the certificate expires you will begin receiving renewal notices.

Seeing the above, someone from our team would renew these certificates manually and do the needful.

Although, this served our purpose for the time being, it was becoming difficult and time-consuming for one person to manage this piece of infrastructure with the increasing number of IPSec VPN’s.

Managing your own PKI infrastructure is a hard problem in itself and leaving it to manual processes always leaves room for human error.

The Kernel team, which I am part of, focuses on solving infrastructure problems, improving systems resiliency and developer productivity.

Hence the motto: Productivity through Automation

Problems with our current approach

We have faced a lot of issues around cert management:

We need a solution to this problem — something very basic without a high learning curve.

The features we wanted in our tool

Enter Kingsly

It all started last December, when our team was contemplating what to ship for our internal company hackathon.

This tool was a perfect candidate for the night as it was both:

By the end of the night, we had it in a working condition and generating SSL certificates was as simple as doing a

$ curl -X POST http://kingsly.host/v1/cert_bundles \
  -u admin:password \
  -H 'Content-Type: application/json' \
  -d '{
        "top_level_domain":"your-domain.com",
        "sub_domain": "your-sub-domain"
    }'

and the response from the Kingsly server would be:

'{
  "private_key":"-----BEGIN RSA PRIVATE KEY-----\nFOO...\n-----END RSA PRIVATE KEY-----\n",
  "full_chain":"-----BEGIN CERTIFICATE-----\nBAR...\n-----END CERTIFICATE-----\n"
}'

A simple JSON response which can be digested by a client the way it wants.

We had a framework for the product. Now what?

We didn’t want this to end up as a hackathon project that remains a desolate creature in one’s source code repository. So we decided to extend Kingsly and make it into something which fit the initial set of requirements we had in mind for it.

Moar features!!

After the cert generation part, the next feature which we wanted was automatic renewals of those certs.

The client (the IPSec client which we built) would keep polling the Kingsly server, get the certificates, and see if the current cert inside the IPSec VPN was different from the one which the server returned. If not, the client would replace the certs in the IPSec VPN box with the ones it received from the Kingsly server. It would then continue with the flow of tasks needed for the IPSec VPN to pick up the new certs, something which used to be done manually by a human.

Who gets to request generation of certs?

This was still an unsolved problem in the whole equation. We would need to devise a way in which we would be able to deny requests from clients.

One thing which we were sure about was: authentication and authorization should not be handled by the application.

A very simple solution was to put the Kingsly web server behind an HAproxy. Here, we would have a frontend rule pointing to a backend having an ACL with a list of IPs which would be allowed through.

This checked the initial problem of only allowing the clients which were whitelisted.

But the other problem would be maintaining the updated list of all the clients being allowed. On top of that, we were trying to automate a process. Hence, settling for something which eventually needed manual intervention didn’t quite fit our vision. So, we started exploring other possibilities and stumbled upon IAP, which fit the bill for our use case.

Identity Aware Proxy (IAP) is the GCP provided solution for user as well as service authentication.

IAP is nothing but Oauth 2.0 implementation over a proxy.

For authentication with IAP, some required headers need to be added to every request which goes out of the client box. We created a proxy service, which adds the authentication details to the request. For service-to-service authentication, it uses OAuth 2.0 using creds of service accounts.

Why use IAP for auth?

There’s more to come

As the saying goes, there’s no silver bullet in software and Kingsly checks off the list of requirements we had from a tool.

Kingsly is open source! Check the links below 🖖

Kingsly was also presented by me over at DevConf, India