Bare Metal Images with bencher run

Bare Metal Runners execute benchmarks packaged in OCI Image Format. When using the bencher run CLI subcommand, specify an Image with the --image option to enable remote, bare metal execution.

When Continuous Benchmarking, that is benchmarking in CI, using bare metal runners can drastically reduce the amount of noise in your benchmark results. Further, Bencher Bare Metal can be run outside of CI using the exact same bare metal runners. This allows both developers and agents to check the performance of their code changes without having to go through an entire CI workflow. See the Bare Metal Workflow for a full overview.

--image <IMAGE>

Set the OCI image reference for remote Bare Metal Runner execution (e.g. my-image:tag, registry.bencher.dev/my-image:tag, or for Bencher Self-Hosted instances localhost:6610/my-image:tag). Only the OCI registry for the Bencher API server is supported. When --image is set, the benchmark command runs on a remote Bare Metal Runner instead of locally.

🐰 Shared Bare Metal Runners do not have network access. That is, you cannot pull down any dependencies while executing your benchmarks. Your Image must be entirely self-contained.

--entrypoint <ENTRYPOINT>

Override the container entrypoint for executing the benchmark command. This option requires the --image option to be set.

--env <KEY=VALUE>

Set environment variables in KEY=VALUE format. May be specified multiple times. This option requires the --image option to be set.

--job-timeout <SECONDS>

Request a maximum Job execution time in seconds. However, the following Job timeouts are always enforced with the provided concurrency limits:

This option requires either the --image option or the --job option to be set.

--job-poll-interval <SECONDS>

Set the poll interval in seconds when waiting for Bare Metal Job completion. This option requires either the --image option or the --job option to be set. This option conflicts with the --detach option.

--detach

Detach after submitting the Bare Metal Job, without waiting for completion. This option requires the --image option to be set. This option conflicts with the --job-poll-interval option.

--job <UUID>

Optional: Attach to a Bare Metal Job. The majority of the time, this option is used when a separate bencher run invocation has already used the --detach flag. Several other bencher run options pick up and function as though you had never detached. This allows you to await Bare Metal Jobs asynchronously using callbacks (per bencher run invocation webhooks).

See Pull Requests on Bare Metal for an example using GitHub Actions.

--callback-url <URL>

Optional: Specify a callback URL for a detached Bare Metal Job. The URL must use https. The organization for this project must have an active Bencher Plus plan.

To specify request headers, use the --callback-header option. To specify the request body, use the --callback-body option. Otherwise, the bencher run API endpoint response body is used by default.

This option requires the --image option and the --detach flag to be set. This option conflicts with the --github-actions option.

--callback-header <NAME: VALUE>

Optional: Add a header to the callback request in NAME: VALUE format. The header NAME has a max size of 256 bytes, and the header VALUE has a max size of 8 KiB. This option may be specified multiple times (max 16).

The following headers are not supported:

• Connection
• Content-Length
• Host
• Keep-Alive
• Proxy-Connection
• TE
• Trailer
• Transfer-Encoding
• Upgrade

This option requires the --callback-url option to be set.

--callback-body <JSON>

Optional: Set the JSON body of the callback request. This JSON value can use any of the following template variables in a string:

The max size for this JSON value template is 64 KiB. Template expansion is single-pass, and any unknown template variables will fail the expansion.

This option requires the --callback-url option to be set. If the --callback-body option is not specified, the bencher run API endpoint response body ("{{ report }}") is used by default.

For example, to post a message to a Slack incoming webhook:

bencher run \
  --project project-abc4567-wxyz123456789 \
  --image project-abc4567-wxyz123456789:latest \
  --detach \
  --callback-url "$SLACK_WEBHOOK_URL" \
  --callback-body '{"text": "{{ project.name }}: Bencher Job {{ job.status }}", "blocks": [{"type": "section", "text": {"type": "mrkdwn", "text": "*{{ project.name }}*: Bencher Job {{ job.status }}\n<https://bencher.dev/console/projects/{{ project.slug }}/reports/{{ report.uuid }}|View the report>"}}]}' \
  bencher mock

🐰 Congrats! You have learned all about Bare Metal Images! 🎉

Keep Going: Branches & Start Points ➡