Modeling Continuous Delivery under Bazel

The term "CD" is ambiguous. Some engineers use it to mean "Continous Deployment", in which changes are automatically released, e.g. into a "dev" environment.

Aspect recommends that Continuous Delivery is modelled as the step of the pipeline where built artifacts are uploaded from the build machine to a well-known repository location. This could be a container image registry like Docker Hub, a blob store like AWS S3, or even a database.

This makes a clear separation of responsibilities between CI/CD and Deployment:

• The CI/CD pipeline, should upload only the artifacts that are:
  • configured with BUILD.bazel files: product engineers don't need to worry about setting up CD
  • green: it can prove that all relevant tests are passing
  • changed from a previous build
• The deployment system
  • locates and "promotes" artifacts to the next environment, such as "dev", "staging", or "prod".

Build vs. Buy

The recommendations in this guide can be applied in two ways:

• DevInfra teams may wish to implement and operate a custom system for their organization, or
• Use Aspect Workflows, which provides this feature out-of-the-box.

What is "deliverable"

A deliverable artifact is one that contains both the binary or files to push, as well as the "pushing" logic that knows how to perform the upload. It might also send a message to the deployment system to trigger an auto-deployment of the new artifact.

In Bazel terms, this means a deliverable should be an executable program that can be bazel run.

Container Images

The rules_oci oci_push or rules_docker container_push rules can both be executed with bazel run to push a Docker image to a registry like Docker Hub.

Therefore these rules are considered "deliverable".

Git Push

Sometimes artifacts belong in a separate code repository. For example, an SDK built from the API definitions in a monorepo needs to be published.

See an example git_push executable here: https://github.com/aspect-build/bazel-examples/tree/main/git_push.

S3 upload

See s3_sync.

Which targets to deliver

A bazel query expression is the most convenient way to locate deliverable targets. Users may choose a tagging scheme for their workspace (i.e. "all targets with tags = ['artifact']"), or deliver well-known rule kinds (i.e. oci_push), or both.

Which changes to deliver

To optimize time and money, it's best to deliver only "changed" targets. This avoids wasted time and resources uploading the same artifact repeatedly. It also means that release engineers won't have to sort through a massive list of duplicates when choosing a release.

There are two approaches for choosing "changed" targets:

1. Predict the changes based on a version control delta. For example you could git diff between the hash being delivered and the "prior successful" delivery hash, then use a tool like bazel-diff or target-determinator to produce a list of targets that might be affected by those changes.
2. Determine empirically based on what is actually different. This requires determinism, so it must only use unstamped build results (--nostamp). In most cases a green CI run just completed, so these unstamped outputs are easily available.

Aspect recommends following the second approach because the first has some downsides:

• bazel-diff is incorrect and will sometimes miss affected targets, so they aren't delivered.
• target-determinator is slow and may hurt the "service level indicator" of time between pushing a hotfix and being able to release that fix.
• It will over-deliver, because sometimes a source change doesn't actually factor into whether the release binary changes, such as for a comment-only change.

The rest of this section provides more details about approach 2 (determine empirically).

To determine whether that executable target should be delivered on a particular commit, it is first hashed. This uses the aspect outputs command with a special pseudo-mnemonic "ExecutableHash", for example:

$ aspect outputs 'attr("tags", "\bdeliverable\b", //...)' ExecutableHash
//cli:release h1:cj8OUC3l3fIr3Zxnffk6y7gukLOJmiWRCAQoqadg66Y=
//workflows/rosetta:release h1:kjHVajw+Nta2kh3Epcd32DkZxTE1NHA8b5N7hCNFNSM=

Some lookup database is required to store previously delivered hashes. If the hash value matches one previously seen, then delivery of that target may be skipped.

Debugging changes to deliver

The aspect outputs command may be run locally to understand whether a given change to a source file will result in a new executable. Sometimes the result may be surprising. For example, if a comment in a .go source file is changed, the compiler produces the same .a file as a result, so the hash seen on the uploader executable is unchanged.

Another scenario that won't change the executable is when some production configuration is changed. For example, Helm charts may be used to deploy to Kubernetes. If these aren't included inside the image, then changes to these files won't cause a new delivery.

Perform the delivery

Run each deliverable target with stamping enabled. This can be done in a script which reads the targets from a manifest file such as cat $delivery_manifest | xargs -N1 bazel run --stamp.