Software Engineering at Google Chapter #18 - Build Systems and Build Philosophy (3 of 3)

  • External dependencies are risky because the build system needs to constantly check to see if they have changed. If the build system doesn’t check then artifacts will be built with old or non-current versions of the dependency
  • Artifact-based build systems such as Bazel track changes to external dependencies by using hashes. This means the build system does not need to re-download the external dependency for each build - it only downloads it when it changes (as determined by a change in hash)
  • Using hashes also means that if we build an old version of the software it will be built using the same external dependencies as it did originally (as opposed to pulling down the newest, and perhaps incompatible version of the external dependency)
  • Distributed builds are scaled across multiple systems in order to reduce build time. Essentially, parts of the build are done in parallel.
  • Google’s system of distributed builds consists of...
    • Masters that oversee and coordinate the build job across multiple systems and send the results back to the developer
    • Workers that execute parts of the build and send the results back to the master
    • Order of Operations where the master determines which build tasks must be done in which order
    • Remote caching for both workstations and servers. These caches hold both dependencies (such as libraries) and build artifacts (the end result of a build)
  • Google recommends using an artifact based build system from day one because that is where you will inevitably end up as your code base and company grow
  • Smaller build targets means more parallelization, less frequent rebuilds (due to caching), and finer grained testing (and retesting)
  • Try to use remote caching of built output instead of caching external dependencies in order to reduce overhead (only rebuild what needs to be rebuilt vs pulling in and building against the same dependences for each rebuild of your project)
  • In the build system you should intentionally limit engineers power and flexibility in order to increase productivity via consistency
  • Dependencies should be deps should be versions explicitly and manually
  • Consider pinning or explicitly stating dependency versions in your build files (eg: don't import package foo but import package foo v1.2.3)
  • The tradeoff for version pinning is convenience (pull down the latest version) vs consistency (always build with v1.2.3)
  • One should mirror external dependencies (eg: NPM packages) internally so that they can be analyzed for security and consistency
  • Another approach is to store dependencies with source code so you know exactly what will be used in each build



Thank you for your time and attention.
Apply what you've learned here.
Enjoy it all.