Skip to main content

Migrate to bzlmod

Background: Bazel's dependency story​

Historically, Bazel came from Blaze, the monorepo build tool developed within Google's monorepo, "google3". google3 is fully self-contained, even bootstrapping compilers from source. It has no "third-party" dependencies, aside from those that engineers lovingly vendor into the third_party folder. For this reason, Blaze has no affordances for fetching external packages. The google3/WORKSPACE file is short and meaningless to Googlers as there's very little repo-wide configuration.

However when releasing Bazel, it was clear that other users needed external packages. As much as Google is opinionated, asking everyone to vendor their dependencies in every language is a non-starter.

Thus, concepts such as "repository rules" were invented to give the beginning of Bazel's analysis phase the ability to fetch archives over HTTP, and perform post-install steps on the code that was fetched. As evidence of this lineage, note that Bazel's evaluation model doesn't mention the "fetch" phase; this is because Blaze has no such concept.

However, the implementation of the "fetch" phase in the WORKSPACE file was fatally flawed: it has no dependency resolution step. In fact it doesn't understand transitive dependencies at all. To workaround this omission, developers of Bazel rulesets created macros that wrap the repository_rules, such as my_rules_dependencies(). Calling this from the end-user's WORKSPACE file causes those dependencies to fetch. This doesn't work well because the first fetch wins. If my_rules_dependencies() brings you rules_python@0.1.0 then you get errors later in the build about rules_python not working the way you expect, and it's nearly impossible to discover where version 0.1.0 comes from.

The solution to these problems is to replace most or all of the content of your WORKSPACE file with a new file introduced for Bazel 6, MODULE.bazel. Starting with Bazel 7, the new file is enabled by default, and in Bazel 8 the team plans to disable reading WORKSPACE unless you supply an opt-in flag; however that flag will be removed in Bazel 9.

This means every Bazel user is forced to make this migration eventually.

more reading

For more background, see our blog post from before Bazel 6 was released.

Common migration path​

  1. Upgrade to latest Bazel first. It's possible to do bzlmod on Bazel 6 and then upgrade Bazel after. However we recommend this order because bzlmod bugfixes have been landing, so the bzlmod-first migration might need extra workarounds.
  2. At first, disable new Bazel 7 flags. Makes the Bazel 7 upgrade less risky and "do one thing at a time". These include:
    • Add --noenable_bzlmod
    • Remove --noexperimental_check_external_repository_files which is broken with BwoB
    • Add --@io_bazel_rules_docker//transitions:enable=false if using rules_docker, issue
    • Add --noincompatible_sandbox_hermetic_tmp issue
  3. Land .bazelversion upgrade and wait a few days to "bake" since there may be developer machines or CD machines that have hard-coded Bazel version.
  4. Remove --noenable_bzlmod and try analyzing (bazel build --nobuild //...). Bazel will create a MODULE.bazel file. Follow guidance below to make the minimal changes to get this green.
  5. Rinse-and-repeat to burn down the content of WORKSPACE and replace with MODULE.bazel.

Detailed migration notes

Bazel 7 minimum versions​

Rules added Bazel 7 support at different times. Here's a guide to the minimum versions you need:

https://github.com/aspect-build/rules_js/releases/tag/v1.31.0 Because first release to contain https://github.com/aspect-build/rules_js/commit/dbafe9edf84160b97dc113b299460d219b322068 "fix: allow for Bazel 7 flag rename"

https://github.com/bazelbuild/rules_apple/releases/tag/3.0.0-rc2 Because notes mention Replaced apple_common.multi_arch_split with new transition, this should not be a breaking change but is required for bazel 7.x support. Please file any issues you find!

https://github.com/bazelbuild/apple_support/releases/tag/1.6.0 https://github.com/bazelbuild/rules_swift/releases/tag/1.8.0 because first release to say "This release is compatible with 5.x LTS, 6.x LTS, and bazel 7.x rolling releases"

https://github.com/grpc/grpc/releases/tag/v1.61.0 Because first release to contain https://github.com/grpc/grpc/commit/45aecbe3555c1da36cfbfaff0622646923d5137c "Added Bazel 7 to the support bazel versions"

MODULE version resolution​

Modules and versioning are one of the main benefits of bzlmod. However, the version resolution is different than WORKSPACE and can lead to different versions being resolved. With WORKSPACE and the use of "maybe" the first version declared is used, with MODULE the highest version declared is used.

See bazel bzlmod migration and referenced MVS version resolution strategy.

When the resolved version differs from the version declared locally in the MODULE this will be visible with a WARNING, for example:

WARNING: For repository 'bazel_features', the root module requires module version bazel_features@1.2.0, but got bazel_features@1.9.1 in the resolved dependency graph.

Cases where this can lead to issues:

  • when patches are applied to modules the patches may break when the version of that module is updated by another ruleset

MODULE.bazel.lock issues​

The lockfile caches fetched data and ideally remains enabled, however should not be checked in to git due to a number of pending bazel issues.

Some issues may require deleting the lockfile and bazel regenerating it, for example:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'SINGLE_EXTENSION_EVAL:ModuleExtensionId{bzlFileLabel=@@rules_go~//go:extensions.bzl, extensionName=go_sdk, isolationKey=Optional.empty}' (requested by nodes 'BZLMOD_REPO_RULE:@@rules_go~~go_sdk~io_bazel_rules_nogo')
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null value in entry: bazel_features=null
    at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
    at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54)
    at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:341)
    at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:450)
    at com.google.devtools.build.lib.bazel.bzlmod.Module.getRepoMappingWithBazelDepsOnly(Module.java:67)
    at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.getExtensionUsagesById(BazelDepGraphFunction.java:233)
    at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.getExtensionUsagesById(BazelDepGraphFunction.java:221)
    at com.google.devtools.build.lib.bazel.bzlmod.SingleExtensionEvalFunction.tryGettingValueFromLockFile(SingleExtensionEvalFunction.java:258)
    at com.google.devtools.build.lib.bazel.bzlmod.SingleExtensionEvalFunction.compute(SingleExtensionEvalFunction.java:165)
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
    ... 7 more

Runfiles directory change​

The runfiles directory layout of binary targets has changed when bzlmod is enabled. If a source repository is directly referencing in the runfiles directory the paths need to change.

Ideally runfiles libraries or env vars would be used to avoid hardcoding paths.

rules_docker must propagate repo_mapping so binaries can use runfiles library​

rules_docker requires the following patch:

diff --git a/lang/image.bzl b/lang/image.bzl
index 2eca59e..e617006 100644
--- a/lang/image.bzl
+++ b/lang/image.bzl
@@ -172,6 +172,9 @@ def _app_layer_impl(ctx, runfiles = None, emptyfiles = None):
         if filepath(ctx, f) not in available and layer_file_path(ctx, f) not in available
     }
 
+    if ctx.attr.binary:
+        file_map[_runfiles_dir(ctx)+"/_repo_mapping"] = ctx.attr.binary[DefaultInfo].files_to_run.repo_mapping_manifest
+
     # emptyfiles(dep) can be `depset` or `list`. Convert it to list only if needed.
     emptyfiles_list = emptyfiles(dep).to_list() if type(emptyfiles(dep)) == "depset" else emptyfiles(dep)
     empty_files = [

Missing dependencies​

Pre-bzlmod repository rules were global, for example all http_archive repositories are global. If one ruleset does not declare a dependency in a *_deps() macro it may not be detected if fetched elsewhere. As rulesets are moved to MODULE these missing dependencies might be exposed and require additional http_archive calls to fetch transitive dependencies.

For example grpc does not declare its dependency on rules_ruby and normally inherits this dep from protobuf. If protobuf is then moved into MODULE.bazel a http_archive(name = "rules_ruby") must be added to declare the missing transitive dependency.

rules_python​

Different behaviour with --enable_bzlmod​

It behaves differently by sensing the --enable_bzlmod flag and opting into different behavior. This flag assumes rules_python has been moved to MODULE.bazel. Until rules_python has been moved to MODULE.bazel this line should be patched to BZLMOD_ENABLED = False

Can be observed by syntax errors reported due to different python interpreter being selected by toolchain resolution.

API changes​

The MODULE.bazel API is different then the WORKSPACE API and must be migrated at the same time as migrating rules_python to MODULE.bazel.

Python version used by pip​

API changes with bzlmod rules_python may cause the python version to change, which may cause the python version used by pip to change, which may cause pip install failures such as gcc or link errors.

See "specifying python_version" below.

Specifying an exact python_version​

It's not desirable to change the Python version at all, even to a later micro version on the same minor.

The fix is in versions >0.31.0, see b9f39bf0

@rules_python_internal​

When switching to bzlmod @rules_python_internal-not-found errors appeared. This may be rules_python#1543. The following workaround appears to work:

internal_deps = use_extension("@rules_python//python/private/bzlmod:internal_deps.bzl", "internal_deps")
internal_deps.install()
use_repo(internal_deps, "rules_python_internal")

wheel mods additive_build_content​

The pip.whl_mods(additive_build_content) can be used to add BUILD content to pip packages. These BUILDs are in the @pip_name repo and will not have access to repositories loaded from the main repository.

stackb/rules_proto​

bzlmod​

When --enable_bzlmod is enabled the root BUILD must be patched to change @// to @build_stack_rules_proto//. See https://github.com/stackb/rules_proto/issues/369

gazelle version compatibility​

Also this breaking change forces upgrading gazelle to >=0.35: https://github.com/stackb/rules_proto/issues/367

protobuf (com_google_protobuf):​

protobuf C compilation errors​

When adding --enable_bzlmod one of the first errors was a protobuf cc error when compiling "well known types".

Example:

bazel-out/k8-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/type_proto/google/protobuf/type.pb.h:886:5: error: 'GOOGLE_DCHECK' was not declared in this scope; did you mean ‘ABSL_DCHECK'

Moving protobuf to MODULE.bazel seems to workaround this issue. Most likely due to the protobuf bzlmod patches on BCR.

protobuf C linking errors​

If multiple versions of protobuf have been loaded by different rulesets (via http_archives) there may be link errors.

For example:

In file included from bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/field_mask_util/google/protobuf/util/field_mask_util.h:40,
                 from external/com_google_protobuf/src/google/protobuf/util/field_mask_util.cc:31:
bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/field_mask_proto/google/protobuf/field_mask.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
   17 | #error This file was generated by an older version of protoc which is
      |  ^~~~~

This protobuf compilation error is an example of duplicate repositories which must be patched to remove, see section below.

Duplicate repositories in MODULE + WORKSPACE​

Due to bzlmod-loaded repos not being in //external:* tools such as maybe do not detect the bzlmod-loaded repos. See https://github.com/bazelbuild/bazel/issues/15377#issuecomment-1115248739

This is sometimes mitigated because http_archive calls will "shadow" bzlmod modules, but only when the names align and not when bazel_dep(repo_name) is used. See https://github.com/bazelbuild/bazel/issues/21818

Duplicate copies of repos will frequently lead to issues such as compilation errors (example: two versions of protobuf being linked into a common binary) or structs/providers/... duplicated and not being "equal" between the MODULE and WORKSPACE versions.

To avoid duplicate verions of rulesets any module loaded with bzlmod can often no longer be loaded with http_archive and similar in WORKSPACEs, including transitive deps in *_deps() calls of other repos.

This requires removing http_archives from *_deps() macros of rulesets not yet moved to MODULE.bazel, often requiring patches.

For example rulesets like grpc fetch protobuf and stackb/rules_proto a few times more and more). When protobuf is moved to MODULE.bazel these must be patched to not fetch duplicate versions of protobuf.

go_repository deps​

As a special case of Duplicate repositories, a go_repository call in MODULE + WORKSPACE for the same package leads to go errors. The workaround is adding those transitive go deps to your own go.mod so the transitive ones get shadowed instead of duplicated, see https://bazelbuild.slack.com/archives/CDBP88Z0D/p1712280817398069 In practice, you should create a transitives.go file with "underscore"-named unused imports, so the Go tooling like go mod tidy still works.

native.bind() in repository rules​

native.bind() has no equivelent in bzlmod: https://bazel.build/external/migration#bind-targets

Any rulset, wether loaded via WORKSPACE or MODULE, can not use native.bind() when --enable_bzlmod.

If a repository has bind(name = "x", actual = "@a//:b") all usage of x must be changed to @a//:b, this often requires a patch (for example grpc makes use of this referring to protobuf binary targets).

See grpc patch for example: https://github.com/grpc/grpc/pull/36157

Last updated on by Jason Bedard