At the time of this writing, we have a universe of Mac applications — that are identified and version-inventoried — within the fleet of little over 3,000 Mac devices in Zalando from which a subset — selected either by their importance, frequency of updates or size of the install base — are part of a so-called software lifecycle.

However, in July 2019, when a vulnerability was discovered in Zoom (long before becoming the mainstream video conference app during the COVID-19 pandemic), Information Security requested the immediate deployment of the latest patch to every device that had the app installed and a report of the progress of this task.

The report and the patch were not a challenge in themselves — this was already part of what we were doing with core applications such as Google Chrome, or Chat — but the process was nothing more than a set of manual and repetitive chores that could be streamlined.

So this defined a set of goals:

• Procure patches and updates in a proactive way
• Test them and then deploy to our users as soon as possible after their release
• Keep detailed information about the patch levels of key applications
• Automate, as much as possible, all these tasks

Our tools

JAMF Patch Management

The Mac Management Platform in use in Zalando, called JAMF Pro, provides Patch Management functionalities that are great at detecting the patch level of devices and deploying the appropriate versions; however, getting this functionality to work properly has the following requirements.

A source of patch definitions

The first thing the system needs is the so-called definition of the title¹ including dates, versions, OS requirements, etc. in a JSON format. JAMF (the company behind JAMF Pro) offers a web service with a basic set of titles, but of course, that doesn’t cover all our core applications. Fortunately, it’s also possible to configure additional sources of patch definitions, either local or from third parties.

Installation packages

Each vendor has different locations to provide their installers; additionally, for the management platform to be able to install applications (or its updates), they need to be uploaded to distribution points in a PKG format, which is not always what the vendor provides.

AutoPkg

An open source tool developed by the community of Mac admins around the world, called AutoPkg, provides a framework to automate many of the tasks surrounding patch management. The steps taken through the process are defined on plist-format files called recipes, which AutoPkg follows.

Recipes

The community of AutoPkg users has generated recipes that cover a broad range of applications and that are updated regularly; nevertheless, for security reasons, AutoPkg requires manual inspection of downloaded recipes or the creation of local copies, before allowing an automated execution. AutoPkg recipes have a parent-child relationship which brings modularity and also the chance of having different results depending on the child recipe that was executed.

Processors

Each step of a recipe is executed by a Python piece of code called processor. AutoPkg includes dozens of these processors — each of them with a specific functionality — but also has the ability to run custom processors, coded by users, to provide functionality not covered by the standard ones.

Our solution

The combination of JAMF Patch Management and AutoPkg was the right one to accomplish our goals, but this doesn’t work for our needs just out of the box and then it evolved into three different projects.

Cookbook

The name was obvious for the project aiming to standardize and manage our AutoPkg recipes.

For improved modularity of the process, each application that we have introduced into the software lifecycle has its own set of recipes:

• Download from the vendor
• Create a package
• Sign the package²
• Upload to the distribution points

In addition to the recipes, we created three custom processors to:

• Announce in a Google Chat group the availability of a new version, packaged and uploaded to our system
• Generate the JSON patch definition and upload it to our own definition server, for titles not covered by JAMF
• Update information in our reporting tool, LineUp

Finally, for better organization of the workload, Cookbook is a git repository. We work locally, push our changes to the repository and then after merging, we pull on a server called Apple Packaging Station that runs AutoPkg on a regular schedule with help from a third party tool called AutoPkgR.

LineUp

When we first created a report about the deployment of the patch of Zoom, we pulled the information from our platform directly into a Google Spreadsheet and then used Google Data Studio to generate a chart.

This may seem okay for a one-shot requirement, but in reality this happens often throughout the year and becomes hard to maintain or scale. So then we opted for a custom database (hosted in Zalando’s shared Postgres cluster) queried with Grafana, which offers great visualization capabilities.

But then, with a proper database structure already holding the data, the next logical step was to add a custom visualization tool and provide it with its own API to update the information. This is when LineUp was born.

At the beginning, we were just looking for a simple mechanism to show information from the database without requiring a client application or the user to run SQL queries, and even the simplest web development frameworks, once connected to a database, have power to do much more than this. We selected Django as our framework and after developing these simple views, we decided to leverage its capabilities and come up with detailed views for each Mac application, creating a module to use JAMF’s API to get up-to-date information about them.

Then, while working on this, it was natural to expand the scope and include the inventory of applications running in the Windows and Ubuntu platforms and to do so, we developed a module to query Zalando’s asset management platform.

PackageChanger

After each scheduled execution of our AutoPKG recipes we end up with a set of packages uploaded to the distribution points, notifications about them in our Chat group, and the JAMF server aware of these new versions of applications. Now it’s time to test the updates and release them if they are working properly.

This became a new tedious process which is done in JAMF’s web UI. Each update implies going to a set of screens to associate the new version with a package, assign that version to a group of testers and later, release the version to the rest of the users as well as setting this version as the baseline installer for new devices.

To simplify these steps, we created PackageChanger, a command line tool that, through JAMF’s API, let’s us work with packages and versions in a faster and simpler way than using a web UI.

To work with the API we selected Ruby-JSS — a Ruby library developed by the Mac admins at Pixar Animation Studios — which to this day is the most comprehensive and well documented library to interact with it.

Our next steps

The work done so far has improved significantly the way we make updates available, especially for key applications, and has provided us with ways to have real-time information during first few hours after a software vulnerability is disclosed. We are still missing, nevertheless, some refinements to have a completely streamlined software lifecycle.

User interaction

Patch management from JAMF offers us two ways to deploy patches: automatic push or through the Self Service application notifying the user when updates are available. The latter would be optimal, but the notification mechanism does not work and leaves us with our user base unaware of patches. On the other hand, pushing updates has proven to be a source of discomfort for users, especially because updated applications need to be closed and reopened and it’s really difficult to find a convenient moment to do this.

As a response, we are working on an alternative notification mechanism, so we can continue to offer updates through Self Service, but making users aware of them with enough frequency and convenience so that they install them in a comfortable and timely manner.

Quality gate

Before generally releasing a patch we deploy it to a small subset of devices whose owners are considered testers. This allows us to know if the installer works and if the application runs as expected after the update.

These tests may be enough for simple applications — such as Google Chat — but fall short for specialized or complex ones — such as Tableau Desktop — where only a trained user would be able to tell if the new version is ready to be deployed to the user base.

The next improvement in this direction would be a quality gate, in which additional tests for releases are described and a bigger set of testers can go through them, decide if they are passed successfully, and then approve collectively the deployment of a patch.

Increased selection of titles

The initial set of applications covered by patch management was selected because of the obvious level of use the get within Zalando: Google Chrome, Chat, Backup and Sync, etc.

Afterwards, when LineUp provided us with information about the number of installations of each application, we had a roadmap of sorts to know which applications should be covered next. For example, we discovered that over one third of the Mac fleet has Docker installed on them, so we decided to start offering it in Self Service and provide patch management so that we can be sure our user base has easy access to this tool.

Here, the next step is part of a continuous improvement cycle, in which we will keep adding applications to the automated lifecycle.