FontOps: Font Development At Scale
Two years ago, I took over the development of (and later, the technical programme management) of the Noto project, a library of nearly 250 font families spanning over 150 writing systems. Noto provides “fallback” fonts for language support on Android, iOS, MacOS and Linux - billions of devices across the planet, and for many writing systems, Noto’s fonts are the only fonts available. How on earth can we manage and maintain such a diverse and important catalogue?
One of the ideas I wanted to bring to the project was what I now jokingly call “fontops” - inspired by the concept of devops, “fontops” automates as much of the font production, release, testing and quality assurance processes as possible, so that these processes can operate at scale. For 250 fonts, manual processes don’t scale. Instead, everything has to be done with the knowledge that it’ll need to be done tens or hundreds of times. Getting a computer to do it is the only way. Anything that you need to do more than once should be automated so that it can done any number of times.
I want to explain some of the ways that these principles of automation and scale drive how the Noto project operates today.
Development at scale
To organise the development of the library, each writing system was given its own GitHub repository under the notofonts organisation. For some writing system repositories, such as notofonts/arabic, we have a number of font families representing sans, serif and auxiliary styles; for others, like notofonts/pahwah-hmong there is a single font family. This ensures that issues are easy to track, and that the project is broken down into manageable chunks.
So the first step was to create a template project which sets up a standard directory structure for how the information for each script and its font sources would be stored.
It was extremely important for me that the font binaries were built through a continuous integration server, as provided through GitHub Actions - and in fact, that these automated builds were the final deliverables that become the font releases. The benefit of continuous integration / build automation for font projects is that it means that the font is always built in a “neutral” way. Python virtual environments allow us to somewhat isolate the build toolchain for a font, but even so one might end up with situations where the particular combination of software on my machine builds a font that is subtly different to one built with the particular combination of software on your machine. Having the build server emit the font binaries removes that uncertainty - the GitHub Actions runner is the definitive build environment we can all agree on.
Thankfully, the Google Fonts team had already established their googlefonts-project-template, a template repository with a standard directory structure which sets up an automated font build through continuous integration. Push a new version of the font source, and an action runs to build and archive the font binaries. Very nice! Except for two problems…
First, Noto delivers to a number of “downstream” projects, who all want slightly different things. Android wants the smallest possible binary files - no hinting, no extraneous glyphs, reduced variable font axes; Google Fonts requires a “basic Latin” set to be added to its deliverables; other users want more “standard” builds. So our build process needs to create all these different build targets: hence, notobuilder, a custom version of the Google Fonts Builder.
But second, what if you need to change the way the build works? When maintaining a handful of repositories, you can easily update your Python dependency requirements.txt or make a tweak to the GitHub Action workflow file. But for over 150 repos? Once again, anything that you need to more than once needs to be automated. So I wrote a script to apply changes to all repositories in the project, but also centralized the build process, including the Python requirements and Github actions in the notobuilder repository. Change things there, and every repository gets the changes.
After a few months of testing using the Noto Sans Test repository, we were ready to go, and another script created and populated each of the new repos and transferred their issues across. Now adding a new script to Noto is as simple as clicking on the “Use This Template” button on the template repository.
Push a change to the source file, and a new version is built automatically and then published to each script’s development site along with fontbakery tests and proof sheets (which we’ll talk about in a minute!).
Deployment at scale
Building the fonts is one thing, but I wanted to take a similar devops approach to releases. For many software projects, creating a git tag and pushing this to Github sets off another continuous integration process, which creates a release and publishes it to some software repository - pypi for Python, crates.io for Rust, and so on. Could we do a similar thing for Noto fonts? Push a git tag and have a font update appear on Google Fonts?
As it turns out, getting fonts into Google Fonts isn’t quite as easy as simply uploading them to a web site! But it certainly was possible to arrange the Github actions workflow such that pushing a tag builds the fonts, bundles them up, creates a release, and runs the Google Fonts Packager to automatically create a pull request for the new binaries.
Although each script now has releases in their own individual repositories, many users still wanted a single distribution point to pick up all of the families. To achieve this, another Python script runs each night and walks through each repository, gathering information about new releases, open issues, and so on, and storing them in a big JSON file - a kind of “Noto API” which can be used by downstream distributors. It also pulls these latest releases into its own central repository, tagging the updated files by release so that they can be easily found and previous versions can be retrieved.
Another script builds that into a “Noto dashboard” web site, and another Github action tags and releases the whole repository once per month so that Linux distributions and other users can grab the latest “Noto monthly release”.
I believe in the “release early, release often” philosophy of software releases, and being able to issue two quick command lines to produce a font release makes it easy to produce releases at a significant velocity; so far this month we have had 25 family releases, and it’s only the 14th! But of course the only reason why I’m comfortable pushing out releases at such a velocity is because of the quality assurance which happens on each commit.
Quality assurance at scale
Between development and release, though, there has to be testing and quality assurance - and once again this needs to happen at scale, so has to be automated.
I divide font QA into two distinctly different areas: there’s the technical quality assurance, which refers to the integrity of the font binaries, ensuring that the tables contain expected values and so on, and this is handled by Fontbakery; there’s also the visual side of quality assurance, the kind of things that require a human eye to notice. But even in the visual side we want to automate this testing as much as possible, producing automated reports which help reviewers to understand what has changed in the font and how it performs.
Again, this is all driven by continuous integration as much as possible. For every commit to the source repository, Fontbakery is run on the built binary artifacts and a report is included in the actions outputs; it’s also added to the development web site for each script.
Noto fonts often have much more complex shaping rules than other fonts, and many of the issues we recieve are about shaping problems. So one aspect of quality assurance is making sure that we don’t get regressions of these reported issues; we use test-driven development, again driven through the Fontbakery reports, to ensure that shaping rules produce expected outputs given sample texts highlighted in the issues.
But you can’t automate everything, and there are visual checks as well. Another part of the continuous integration system calls a Python module called notoqa. Again, running on each commit, we grab the last release of a given family and use diffenator2 to look for any regressions changes in the font tables, changes in glyph outlines or differences between shaped strings of text. It’s a really amazing tool for helping to see visually what’s changed in the font without having to spend your time looking at what hasn’t changed in a font.
Finally, we like to keep track of how things are moving along in the project as a whole. Using the “Noto API” JSON file mentioned above we also build an analytics site showing where the open issues remain and where we should be focusing our efforts.
It’s by no means a perfect system and there is still room for improvement, but I’m proud of the automation infrastructure that has been put in place across the Noto project - “fontops” has made a vast and diverse collection of font families into a manageable system. Having this build-test-release cycle made it possible for us to drive some really significant quality improvements across the project last year, closing out hundreds of issues and produce hundreds of new releases. You certainly wouldn’t want to be doing all of that by hand…