Travis CI added a feature to cache dependencies between builds for their paying customers. In projects where resolving and fetching dependencies is a slow operation–e.g. in any Ruby project–this can shave a significant amount off total build time, resulting in faster feedback from CI for the developer. This post explores a DIY method of providing such a cache to open source projects as well.
This is not a new idea. Michał Czyż famously posted a tip how to Speed up
Travis-CI build preparation on Coderwall. Two projects on GitHub,
bundle_cache and travis_bundle_cache, implement the pattern of caching
the results of
bundle install to Amazon S3. However, both projects depend on
the aws-sdk library, which in turn depends on Nokogiri and JSON libraries that
have native extensions to be compiled. As a result, installing the library that
is supposed to speed up your build time is still slow, and this is
WAD by Manfred Stienstra is another Ruby solution, but it doesn’t depend
on aws-sdk and is a standalone script that you can vendor in your project.
This is great because it frees you from having to
gem install anything.
However, I wanted to go a step further and explore whether we need Ruby at all, or can the whole process be handled by a simple shell script and utilities available on a stock Unix system.
The result is the
cached-bundle script whose entire core logic can be seen
below. It delegates the Amazon S3 upload logic to a separate
The cache key is constructed from the Ruby version and MD5 sum of
Gemfile.lock. If any of these change, it’s considered a cache miss and gem
dependencies will be fetched and installed normally.
The dependencies of these scripts are:
- An Amazon S3 bucket in the default region
- Amazon access credentials via environment variables (see below)
Gemfile.lockchecked into version control
To enable caching of Bundler dependencies, add the scripts to the
directory of your project and add this to
secure: value is obtained by means of the offical
travis CLI tool:
$ travis encrypt AMAZON_SECRET_ACCESS_KEY="..."
That’s it! The caching of gem dependencies this way resulted in a >1 minute speedup per build in a project with a relatively small gem bundle.
s3-put is useful for more than just caching dependencies. ruby-build, for
instance, uses Travis CI and this script to keep its Ruby download mirror
up to date whenever someone adds a new version of Ruby to the project.