Installing on Buildkite

Setup Buildkite queues

Each runner group declared in the Workflows terraform module is associated with an agent queue in a Buildkite agent cluster via the queue input. You must create these queues in Buildkite. Read Buildkite's documentation for more details.

  bk_runner_groups = {
    default = {
      queue                     = "aspect-default"
      agent_idle_timeout_min    = 120
      max_runners               = 15
      resource_type             = "default-runner"
      ...
    }
    small = {
      queue                     = "aspect-small"
      agent_idle_timeout_min    = 720
      max_runners               = 4
      resource_type             = "small-runner"
      ...
    }
  }

We recommend using a dedicated runner group with a small, cost-effective instance type and a long idle timeout to minimize the wait times on the Setup Aspect Workflows step (described below) for developers coming online at the beginning of the work day as available runners are scaling up.

Configure a pipeline

Create a new Pipeline for your Workflows build or choose an existing one and configure it to your preference. If you use GitHub, follow these steps to integrate your repository with Buildkite.

Click Steps and Convert to YAML Steps if not already converted, then paste the following:

steps:
    - key: aspect-workflows-setup
      label: ':aspect-build: Setup Aspect Workflows'
      commands:
          - 'rosetta steps | buildkite-agent pipeline upload'
      agents:
          queue: aspect-small

This YAML auto-generates all of the Workflows tasks declared in your Aspect configuration file.

To use a different runner group for the "Setup Aspect Workflows" step, replace queue: aspect-small with the name of the runner group's queue in the Terraform module declaration.

The preceding code snippet assumes the Aspect Workflows configuration file is located at .aspect/workflows/config.yaml. You can override this by passing a --config flag to the rosetta steps command:

commands:
    - 'rosetta steps --config path/to/config | buildkite-agent pipeline upload'

tip

You can use the Buildkite provider for Terraform to set up your pipeline programmatically.

Populate secrets

Per-runner group secrets

The Workflows Terraform module creates three secrets per runner group. You can use the same secret values across the runner groups.

Secret	Description
1. aw_bk_agent_token__<RunnerGroup>_XXXXXX	Connects runner agents to Buildkite.
2. aw_bk_api_token__<RunnerGroup>_XXXXXX	For querying metrics from the Buildkite REST API.
3. aw_bk_git_ssh_key__<RunnerGroup>_XXXXXX	SSH key to authenticate repository access.

1. Buildkite agent token

The agent token allows Workflows runners to connect to Buildkite. Under your organization, click the Agents Tab -> Agent Tokens, click New Token, and add "Aspect Workflows runners" to the description. Click Create Token. Copy and paste this token into the secret named aw_bk_agent_token__<RunnerGroup>_XXXXXX.

Alternatively, Terraform can supply the value.

An output from the Workflows Terraform module exposes the AWS Secrets Manager Secret ID. The ID is named bk_agent_token_secret_ids["runner group name"] where "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then you can configure the secret with:

resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_agent_token_secret_ids["default"]
  secret_string = "my-value"
}

You should supply the secret_string value using whatever mechanism you already use for managing secrets.

2. Buildkite API access token

The API token allows Aspect to monitor the agents created for anomalies. Under Personal Settings go to the API Access Tokens tab and click the New API Access Token button. Give it a name such as "Aspect Workflows API Token" and select your organization from the dropdown. Under REST API Scopes, Aspect needs the "Read Agents" permission. Once created copy and paste this token into a secret named aw_bk_api_token__<RunnerGroup>_XXXXXX.

Alternatively, Terraform can supply the value.

An output from the Workflows Terraform module exposes the AWS Secrets Manager Secret ID. The ID is named bk_api_token_secret_ids["runner group name"] where "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then you can configure the secret with:

resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_api_token_secret_ids["default"]
  secret_string = "my-value"
}

You should supply the secret_string value using whatever mechanism you already use for managing secrets.

3. Git SSH key

If you use a private repository, set up an SSH key that Workflows runners can use to check out your repository.

Under your pipeline's GitHub settings, select the SSH radio button. Then, upload the private key to a secret named aw_bk_git_ssh_key__<RunnerGroup>_XXXXXX. If you use GitHub, we recommend using a Deploy Key with read-only access.

Alternatively, Terraform can supply the value.

An output from the Workflows Terraform module exposes the AWS Secrets Manager Secret ID. The ID is named bk_git_ssh_key_secret_ids["runner group name"] where "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains:

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then you can configure the secret with:

```hcl
resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_git_ssh_key_secret_ids["default"]
  secret_string = "my-value"
}

You should supply the secret_string value using whatever mechanism you already use for managing secrets.

tip

The Workflows module exposes the names of the secrets it creates as an outputs. If you use a tool like Vault, you can wire in secret values automatically via Terraform.

GitHub API token secret

A number of Workflows features require read-only access to the GitHub API. For example, the "Format" task uses a GitHub token to fetch the changed files in a PR.

Create a fine-grained Personal Access Token (PAT) and grant the read permission for Pull Requests, scoped to any repositories that are tested by Workflows.

info

You may need to enable the use of PATs in your organization's settings.

Next, copy the token value into Secrets Manager.

1. Navigate to Your Cloud Console > Secrets Manager > Secrets,
2. Locate the key in the following format aw_gh_api_token__XXXXXXXXXXXXXXXX
3. Set the value to the fine-grained token GitHub provided.

Terraform can also supply this secret.

resource "aws_secretsmanager_secret_version" "gh_api_token" {
  secret_id     = module.aspect-workflows.github_token_secret_id
  secret_string = "my-github-token"
}

Configure workflows

Add a Workflows configuration file to your repository.

(Optional) Setup warming

Warming periodically caches the results of repository rule executions to speed up the first build on a cold runner and reduce the chances of dependency fetching failures. Bazel does not natively cache the results of repository rules.

To enable warming, set up a new Buildkite warming runner group in your Workflows terraform configuration, create the corresponding "aspect-warming" queue in your Buildkite agent cluster, and then enable warming on your CI runner group.

  bk_runner_groups = {
    warming = {
      queue                  = "aspect-warming"
      agent_idle_timeout_min = 1
      max_runners            = 1
      resource_type          = "default-runner"
    }
    default = {
      queue                     = "aspect-default"
      agent_idle_timeout_min    = 120
      max_runners               = 15
      resource_type             = "default-runner"
      warming                   = true
    }
  }

Create a new warming pipeline in Buildkite with the following YAML steps.

env:
  ASPECT_WORKFLOWS_BIN_DIR: /etc/aspect/workflows/bin
steps:
  - label: ":fire: Create warming archives"
    commands: |
      echo "--- :aspect-build: Workflows environment"
      ${ASPECT_WORKFLOWS_BIN_DIR}/configure_workflows_env
      echo "--- :stethoscope: Agent health check"
      ${ASPECT_WORKFLOWS_BIN_DIR}/agent_health_check
      echo "--- :bazel: Create warming archive"
      rosetta run warming
      ${ASPECT_WORKFLOWS_BIN_DIR}/warming_archive
    agents:
      queue: aspect-warming

Finally, go to your warming pipeline's settings, click Schedules, then New Schedule, and configure the cron interval.

For example, to run warming every four hours between 8 AM and 11 PM Eastern Standard Time (EST), use 0 08-22/4 * * * America/Toronto.

(Optional) Custom Buildkite Agent hooks

The Buildkite Agent allows you to declare Agent hooks which it runs at various points throughout the lifecycle of an agent and of jobs.

To supply your own Agent hook, add the hook to the S3 or GCS bucket named aw-buildkite-agent-hooks-XXXXXXXXX under the global/ folder. Runners find the hooks during bootstrap and install them for the Buildkite Agent to use.

To supply an agent hook that applies only to a specific runner group, add the hook to the runner-group/<runner-group-name> folder instead. Runner group specific hooks will be appended to any global hooks in the global/ folder of the same kind.

info

Workflows automatically generate pre-bootstrap, pre-checkout, post-checkout, and pre-exit hooks run before any user supplied pre-bootstrap, pre-checkout, post-checkout, and pre-exit hooks.

The Workflows generated pre-bootstrap and pre-exit hooks are used to gather telemetry for the different stages of the build.

The Workflows generated pre-checkout hook configures the SSH key set in the aw_bk_git_ssh_key secret for the runner group, starts the ssh-agent process, and sets the BUILDKITE_BUILD_CHECKOUT_PATH environment variable. The Workflows generated post-checkout removes the SSH key set in the pre-checkout hook and shuts down the ssh-agent process. User supplied pre-checkout and post-checkout hooks must not to conflict with these settings.

tip

You can alternatively try Aspect Workflows' native hooks solution, which allows you to execute code before and after tasks.

(Optional) Customizing the Buildkite Agent configuration

Workflows automatically generates a Buildkite Agent configuration file on runners with minimal configuration.

To customize the Buildkite Agent configuration further, upload a user buildkite-agent.cfg file to the Workflows Buildkite Agent hooks bucket global/ folder. Runners fetch the user buildkite-agent.cfg from the Buildkite Agent hooks bucket during bootstrap and use it when generating the Buildkite Agent configuration for the runner.

As with hooks, you can upload a custom Buildkite Agent configuration that applies only to a specific runner group to the runner-group/<runner-group-name>/ folder in the Buildkite Agent hooks bucket. If you specify a global and runner group user Buildkite Agent configurations, settings in the runner group configuration take precedence over conflicting settings in the global configuration.

info

The Workflows generated buildkite-agent.cfg file configures the following settings in the Buildkite Agent configuration:

• token
• name
• build-path
• hooks-path
• spawn
• disconnect-after-job
• disconnect-after-idle-timeout

The preceding configuration settings takes precedence over conflicting values set in user supplied buildkite-agent.cfg files.

Customizing the git clone depth

To speed up the Preparing working directory step, you can configure a shallow clone in the Buildkite Agent configuration. For large repositories this can make a measurable impact on clone times on new runners.

For example, to specify a clone depth of 100, set the following in either the global or per-runner group Buildkite Agent configuration in the Workflows Buildkite Agent hooks bucket.

git-clone-flags="-v --no-checkout --depth=100"
git-fetch-flags="-v --prune --depth=100"