10 years of building Apache Kafka
A New Year, a new build system.
The Apache Kafka build system has evolved many times over the years. There has been a concerted effort to modernize the build in the past few months. After dozens of commits, many of conversations with the ASF Infrastructure team, and a lot of trial and error, Apache Kafka is now using GitHub Actions.
This article is the first in a series about modernizing Apache Kafka’s build.
A long time ago, a fledgling open source project named Apache Kafka was in the Apache Software Foundation’s incubator. This project was written in Scala which itself was a relatively new language on the scene. As with most Scala projects, the build tool of choice was sbt — the Scala Build Tool.
As Kafka grew, so did the reasonably small Scala code base. In fact, the code base did not remain purely Scala for very long. Following its graduation to a TLP (Top Level Project), Kafka saw the addition of a fairly significant chunk of Java code. This code was KafkaProducer which still exists today.
commit 269d16d3c915d09f650ae32aa81542bd8522ca68
Author: Jay Kreps <jay.kreps@gmail.com>
Date: Thu Jan 23 12:41:11 2014 -0800
KAFKA-1227 New producer!With the introduction of Java to the code base, the utility of sbt really started to come into question. Not to mention the thing was outrageously slow to compile.
At this point, along comes an elephant. No, not the yellow one who also lives under Apache’s roof, but a bluish-green elephant known as Gradle. Gradle was fairly far along its 1.x release line which included support for Java and Scala. Gradle had recently introduced the Gradle Daemon which allowed for incremental compilation. This worked really well for Java, and could be a massive time saver for developers.
A recently minted committer to Apache Kafka took it upon himself to give this new build system a try. After a lot of tweaking and help from other committers, we landed KAFKA-1171.
commit 52614ad65e698a9860ab22b2ce66bae773263b36 (HEAD)
Author: Jun Rao <junrao@gmail.com>
Date: Fri Feb 7 13:48:04 2014 -0800
kafka-1171; Gradle build for Kafka; patched by David Arthur, Joe Stein, Chris Freeman and Jun Rao; reviewed by Guozhang Wang, Joel Koshy and Neha Narkhede(hey that’s me!)
Fast forward a decade, and Kafka is now primarily written in Java and has a rather large (and rather complicated) Gradle build. Since it’s inception, the Gradle build has been modified over 700 times by dozens of authors. As experienced software engineers know, the build is code. It must be tested, improved, documented, and generally requires same level of attention as any production software.
The way we build (i.e., compile and package) Apache Kafka is one thing. How we test it is quite another. Kafka uses a fairly standard approach to Continuous Integration (CI). We compile and test every commit made to our mainline branch. This is done to ensure a new change has not caused a regression or flat out broken things.
From the early days up until a few months ago, Kafka has used the ASF-hosted Jenkins to run our automated tests. Jenkins has enabled us to deliver extremely high quality and robust software across the decade of Kafka releases. While Jenkins has been a crucial part of Kafka’s history, it doesn’t mean that we always loved it. We started experiencing more and more instability from our CI environment as the size and complexity of Kafka grew.
Our Jenkins dashboards were a constant sea of failed builds. Something needed to change.
Starting in August of 2024, I started working with the Kafka community to bring about a new CI system based on GitHub Actions.
commit 3d868aacf2b893cbe658a6f7a1caa787be746ff3
Author: David Arthur <mumrah@gmail.com>
Date: Mon Aug 19 16:46:20 2024 -0400
MINOR add GH action for PR builds (#16895)
For now this will only run on PRs with branch names starting with "gh-"
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>The response from the community was overwhelmingly positive. There was apparantly some serious pent-up demand for build improvements. In a little over a month, we had reached feature parity and said goodbye to Jenkins.
commit 264131cdaaef3f4696942f26534b3f61f3a2a162
Author: David Arthur <mumrah@gmail.com>
Date: Wed Sep 25 16:31:03 2024 -0400
MINOR: Disable Jenkins build (#17273)
Reviewers: David Jacot <djacot@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>The full story of moving away from Jenkins and adopting GitHub Actions is worthy of its own post.
We have seen many improvements to project as a result of the move to GitHub Actions. This includes not only the stability of the CI environment, but also the way test and overall developer quality of life. It truely feels like a new chapter in the project.
Stay tuned for more posts like this, and more build improvements in 2025.