Speeding up Travis CI builds

Travis CI added a feature to cache dependencies between builds for their paying customers. In projects where resolving and fetching dependencies is a slow operation–e.g. in any Ruby project–this can shave a significant amount off total build time, resulting in faster feedback from CI for the developer. This post explores a DIY method of providing such a cache to open source projects as well.

Update: Since this post was published, Travis CI enabled caching dependencies for open source projects as well.

This is not a new idea. Michał Czyż famously posted a tip how to Speed up Travis-CI build preparation on Coderwall. Two projects on GitHub, bundle_cache and travis_bundle_cache, implement the pattern of caching the results of bundle install to Amazon S3. However, both projects depend on the aws-sdk library, which in turn depends on Nokogiri and JSON libraries that have native extensions to be compiled. As a result, installing the library that is supposed to speed up your build time is still slow, and this is unnacceptable.

WAD by Manfred Stienstra is another Ruby solution, but it doesn’t depend on aws-sdk and is a standalone script that you can vendor in your project. This is great because it frees you from having to gem install anything.

However, I wanted to go a step further and explore whether we need Ruby at all, or can the whole process be handled by a simple shell script and utilities available on a stock Unix system.

The result is the cached-bundle script whose entire core logic can be seen below. It delegates the Amazon S3 upload logic to a separate s3-put script:

cache_name="${TRAVIS_RUBY_VERSION}-${gemfile_hash}.tgz"
fetch_url="http://${AMAZON_S3_BUCKET}.s3.amazonaws.com/${TRAVIS_REPO_SLUG}/${cache_name}"

if download "$fetch_url" "$cache_name"; then
  tar xzf "$cache_name"
fi

bundle "$@"

if [ ! -f "$cache_name" ]; then
  tar czf "$cache_name" vendor/bundle
  script/s3-put "$cache_name" "${AMAZON_S3_BUCKET}:${TRAVIS_REPO_SLUG}/${cache_name}"
fi

The cache key is constructed from the Ruby version and MD5 sum of Gemfile.lock. If any of these change, it’s considered a cache miss and gem dependencies will be fetched and installed normally.

You can fetch the cached-bundle and s3-put scripts, which combined weigh less than 70 lines of code.

The dependencies of these scripts are:

An Amazon S3 bucket in the default region
Amazon access credentials via environment variables (see below)
Gemfile.lock checked into version control
openssl
curl

To enable caching of Bundler dependencies, add the scripts to the script/ directory of your project and add this to .travis.yml:

install: script/cached-bundle install --deployment
env:
  global:
  - AMAZON_S3_BUCKET=my-bucket
  - AMAZON_ACCESS_KEY_ID=MYACCESSKEY
  - secure: "..."

…where secure: value is obtained by means of the offical travis CLI tool:

$ travis encrypt AMAZON_SECRET_ACCESS_KEY="..."

That’s it! The caching of gem dependencies this way resulted in a >1 minute speedup per build in a project with a relatively small gem bundle.

s3-put is useful for more than just caching dependencies. ruby-build, for instance, uses Travis CI and this script to keep its Ruby download mirror up to date whenever someone adds a new version of Ruby to the project.