Enhancing the Security of Our Customer Identity Platform Through Automated Key Rotation

Static secrets are evil. Whether secret keys hard-coded in source code, tokens without expiry or plaintext API keys referenced in configuration files, static secrets are ticking time bombs. The same is true for cryptographic key material in the context of JSON Web Tokens (JWTs) and OpenID Connect (OIDC).

At Zalando, our customer authentication experience team takes protecting our customers' data and their digital identities seriously. Part of our toolbox is an OpenID Connect (OIDC)-based identity provider (IdP). A key aspect of this system's security is the regular rotation of cryptographic keys, which we've automated to ensure the ongoing safety of our platform.

This article aims to shed light on why we rotate cryptographic keys, how the periodical JWK rotation process works, and what it means for customers.

What are JSON Web Keys (JWKs)?

JSON Web Keys (JWKs) are an essential part of the JSON Object Signing and Encryption (JOSE) standards family and the backbone of token-based authentication and authorization frameworks like OIDC. JWK standardises the representation and management of cryptographic keys (RFC 7517). Its JSON data structure allows the exchange of public keys in a web-native format.

Identity providers (IdPs) like ours commonly use JWKs to distribute public key material via well-known and specified URIs. Clients can use the key material to e.g. verify digitally signed JSON Web Tokens (JWTs) issued by the IdP. These tokens contain information about users and their access rights, and their integrity is crucial for preventing unauthorized access.

Why is Key Rotation Important?

Rotation in context of secrets, passwords or cryptographic key material describes the periodical replacement of old with new. This process model is one of the four phases of the secret lifecycle and addresses the threat of undetected key compromise and reduces the window of vulnerability for potential exploits.

If a signing key’s private part is compromised, anyone could forge fake tokens. These tokens could then be used to impersonate users and access sensitive data. Essentially, all tokens signed with the leaked key would become untrustworthy. Regularly rotating cryptographic keys hence is a fundamental security practice. It is paramount that identity providers store long-lived key material securely and rotate regularly.

Our Approach to JWK Rotation

The key rotation process for our identity provider here at Zalando is built around four major principles:

• Automation: New keys are generated and old keys are retired automatically, eliminating manual intervention and ensuring consistency.
• Scheduled Rotation: Keys are rotated on a regular basis to minimize the window of vulnerability.
• Secure Key Management: Our keys are securely stored and managed using industry best practices to protect them from unauthorized access.
• Seamless Rotation: Planned rotations are transparent to clients and do not result in any kind of access revocation or token invalidation.

The robust and automated key rotation process we’ve implemented, follows a careful, phased approach to ensure a smooth transition and minimize disruption for our clients and customers:

First, a new key pair is generated. We then publish the public key portion of this new pair on our JWK endpoint, making it available to our clients. To avoid any immediate disruptions, we incorporate a grace period, allowing clients ample time to fetch the latest set of JWKs – cache control headers matter! After this period, the new key is being elected as the new active signing key. The previous active key is being retired, meaning it's no longer used for signing new tokens, but its public key remains available on the JWK endpoint to ensure that previously issued tokens can still be verified.

Finally, once a retired key surpasses the maximum lifetime of any token it might have signed, we remove its public key from the JWK endpoint. To determine when it's safe to remove a key, we need to know which key signed which token and how long those tokens are valid. Our JWTs include a key ID that tells us exactly which key was used to create them. We also control how long each token lasts before it expires.

With this information, we can easily calculate when a key can be safely deleted. We simply take the time the key was retired, add the maximum token lifespan, and add a little extra time just to be safe. At that point, any token signed with that key will have expired, so it's safe to remove the key from our public list.

Here's the simple formula:

Time of key retirement + Maximum token lifespan + Extra safety time = Time to drop the public key

This ensures that we don't accidentally remove a key that's still needed to verify valid tokens.

Conclusion

Protecting our customers' data is a top priority at Zalando. Automated key rotation using JWKs is just one of the many ways we demonstrate our commitment to security. We believe that this approach balances security with operational stability, allowing us to rotate keys effectively while minimizing any impact on our clients. By regularly rotating our cryptographic keys, we ensure that our customer identity platform remains resilient against potential threats without compromising on experience.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Software Engineer!