- All web-, worker- and run-containers are now powered by the latest generation instances featuring more and faster CPUs as well as SSD storage.
- Push performance has been improved by up to 58% for applications with a large number of dependencies.
Most of you will be familiar with the infamous XKCD compiling comic. For applications with a large number of dependencies, the build process, which is part of the push to the cloudControl platform, most recently felt like an equally good excuse to slack off.
We ourselves, as well as our users, were quite unhappy with this. After all, PaaS is about making development more efficient. Not about creating new time sinks in the process.
To improve our push performance, we first set out to analyze bottlenecks in the build process. The build process, in a nutshell, works like this: A push to the Git repository triggers a hook that starts a worker container, copies the current Git state into the worker, runs the buildpack to download all dependencies and build the application, copies back the dependency cache, uploads the built image to S3, and then if all steps were successful, accepts the push.
Naturally, the build process is quite I/O intensive both regarding downloading dependencies from various sources as well as building the application images and caching the dependencies for subsequent builds. Like all cloudControl containers, the worker containers used to run the builds have a non-persistent file system. Therefore on each build, the application source, assets and dependencies have to be copied into the worker, and afterwards the built image and the dependency cache have to be copied back out of the worker.
The biggest bottleneck we identified was how we handled the dependency cache. Previously, we used rsync to sync the dependency cache directory before destroying the container. The reasoning being, only copying the changes should speed up the process. However, rsync turned out to have a huge performance penalty on pushes with a cold cache while also not providing significant benefits on subsequent pushes with a warm cache compared to simply tarballing the whole directory every time and using said tarball again on the next run.
Furthermore, since AWS provides new instance types featuring more and faster virtual CPUs as well as SSD backed storage, we evaluated switching all cloudControl hosts running web-, worker- or run-containers to the new faster instances. While this will be a nice performance improvement for a wide variety of applications running on the cloudControl platform, the I/O heavy build process is perfect to show the improvements coming from faster CPUs and SSD storage.
As a real world benchmark to measure the improvements, we used our own website, which is a fairly standard Ruby application using a Rails-typical large number of dependencies, and also the Rails asset pipeline.
We did 10 runs each for the original build process using the old instance type, for the improved build process using the old instance type, and finally for the improved build process using the new instance type.
As you can see in the graph below, push performance improved by a tremendous 58% for cold dependency caches through the improved build process and faster CPUs and SSD storage, but also for warm caches the push performance improved by 42%.
Click on the image for a larger version.
While the build process improvements primarily helped the dependency cache handling, the faster instances help with both the cloudControl specific part of the build process and also the language and framework related tasks, e.g. the asset precompilation. For our website, the latter improved from taking up to 100 seconds to taking around 60-70 seconds.
We hope the faster CPUs and SSD storage help speed up your applications, and the improved build process makes day to day work with the cloudControl platform even better.
We’d love to hear from you in the comments, if you can share some performance benefits from your own applications.