Fetching External Dependencies

To work with the code in this repository, we need the toolchains and external dependencies that it relies on. Typically that means third-party dependencies, but the same techniques are useful for internal dependencies managed outside of your Bazel workspace.

By the end of this section, you should be able to run bazel fetch to download these for the language you pick.

If you're stuck, ask the instructor or teaching assistants for help!

Concepts

Definitions

A "Bazel module" is a Bazel project that can have multiple versions, each of which publishes metadata about other modules that it depends on.

"Starlark" is a dialect of Python used to configure Bazel, as well as some other tools.

A "Starlark module" is a different concept, representing a .bzl file we can load from.

Introduced in Bazel 6.0, "bzlmod" is the package manager for Bazel modules. It is semantically similar to numerous other dependency management systems, though extensive discussion went in to making it sufficiently generic.

Read more in its documentation: https://bazel.build/build/bzlmod#modules

Exercise: bazel fetch

Let's find a Bazel-level dependency to add. Go to the Web UI for the Bazel Central Registry: https://registry.bazel.build

Search for whatever interests you. It should be something new.

Use the button to copy the text from the "Install" code block and paste it in MODULE.bazel.

Now you can ask Bazel to fetch that package:

% bazel fetch @some_module//path:all

Or fetch everything, which will fetch just the added dependency if you've made no other changes:

% bazel fetch ...

Bazel creates a MODULE.bazel.lock file as well, which captures the specific versions of dependencies including transitive dependencies, making the module resolution reproducible. This file should be checked in.

note

Bazel also has a sync command, but this is rarely useful and not covered here.

Language dependencies

Bzlmod delegates to other language-specific package managers.

• Python: pip
• JavaScript: pnpm
• Java: Coursier / Maven / Gradle
• and so on

:::exception C++ doesn't really have a popular "package manager", so the Bazel Central Registry is accidentally becoming one. :::

In most cases, under Bazel we'll still use the canonical files for declaring these dependencies, though under Bazel they should always be "pinned" for reproducible builds.

Why? We want to preserve interoperability with existing tools as much as possible, such as editors and static analysis tools, and these understand the idiomatic files for the language.

note

• "pinned" means direct and transitive dependency versions are always exactly specified
• can include integrity hashes for supply chain security

The steps to do this vary a bit between languages, but they all have the following rough outline:

1. You may leave the developer's constraints alone, but do use semantic versioning ranges. For example:
   • our logger/frontend/package.json allows any version of http-server
   • our requirements.txt allows any version of requests
2. Pin transitive dependencies to a constant version
   • These are generally written to a separate "lock" file.
3. Mirror that dependency list into Starlark
   • This allows Bazel to manage the dependencies itself.
4. Add code to expose external repositories for use by BUILD targets
   • The instructions for each language should tell you how to do this.

caution

In practice, you'll find that not all rules do a good job of documenting bzlmod usage yet. You can get a hint by finding the tests for a ruleset.

On https://registry.bazel.build, click the "View registry source" link for a module, and open the presubmit.yml file. You'll find a path to some sub-folder where a test lives. These are executable examples, so they give us a clue how the module is used.

For example,

bcr_test_module:
    module_path: 'e2e/bzlmod'

Then you'd navigate to the /e2e/bzlmod folder in the ruleset repository, and there will be something that is guaranteed to work.

Exercise: add some language-specific dependencies

Bazel's reproducibility can only be as good as the information it's given. Each external package manager has a feature to pin the dependencies.

The /README.bazel.md in the bazel-examples repo explains how to manage third-party package dependencies. You should be sure to have similar documentation for developers in your repository.

The goal is to produce the following files, for the languages you care about:

• go.mod -> go.sum
• package.json -> pnpm-lock.yaml
• requirements.txt -> requirements_lock.txt
• Java sources -> maven_install.json
• Package.swift -> Package.resolved

You'll have to read the documentation for the ruleset you use to figure out an approach to do this, and also add to the MODULE.bazel file by searching the Registry and following ruleset install instructions.

Configuring the downloader

Bazel's downloader is full-featured, and you can use it to block undesired network access, fetch via your corporate proxy or artifact repository, and more. Beware though, that some ruleset use the Bazel downloader while others do not.

Read more

https://blog.aspect.build/configuring-bazels-downloader

Eager fetches

Developers shouldn't need to fetch things they don't use. A developer working in one language shouldn't be blocked waiting to download toolchains for some other language. Most of the time developers will not have a reason to fetch everything, but rather just let fetches happen automatically as needed.

load is eager

The load statement in Starlark happens eagerly during the Loading phase, and causes things to be eagerly fetched.

This de-optimization is easily introduced, and typically is only diagnosed when developers complain about "slow initial builds".

In WORKSPACE

These happen for every single build regardless of the dependency graph or which targets the user requests. Bazel must evaluate the complete WORKSPACE and MODULE.bazel files to understand what third-party dependencies exist for the build. Let's say the WORKSPACE file contains this content:

WORKSPACE.bazel

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
   name = "my_deps",
   requirements_lock = "//path/to:requirements_lock.txt",
)

load("@my_deps//:requirements.bzl", "install_deps")
install_deps()

Because the highlighted line has a load statement, the my_deps repository is requested at loading time, and so the pip_parse implementation will run. If it uses a hermetic python interpreter, then that interpreter must be built or fetched for any build.

In BUILD.bazel

In this example, a BUILD file loads from @npm:

BUILD.bazel

load("@npm//@bazel/typescript:index.bzl", "ts_project")

package(default_visibility = ["//visibility:public"])

ts_project(
    name = "a",
    srcs = glob(["*.ts"]),
    declaration = True,
    tsconfig = "//:tsconfig.json",
    deps = [
        "@npm//@types/node",
        "@npm//tslib",
    ],
)

filegroup(name = "b")

Even if a developer only asks Bazel to build the filegroup named b, the load statement means that the @npm repository must be fetched.

Read more

https://blog.aspect.build/avoid-eager-fetches

Exercise: Fetch

Let's verify we have one language working, by using bazel fetch to get one of the dependencies.