Writing Benchmarks in Rust

Learn how to write benchmarks and measure the performance of your Rust code.

Supported Benchmarking Crates

CodSpeed offers compatibility layers for several popular benchmarking crates:

divan (recommended)

The most convenient way to run your Rust benchmarks

criterion.rs

A benchmarking crate inspired by the criterion haskell library

libtest (bencher)

The libtest (unstable) benchmark runner

You should use the divan benchmarking framework due to its extensive features, such as running type-generic benchmarks. Its popularity is growing rapidly within the Rust ecosystem.

If you’re already using criterion.rs or bencher, consider their respective plugins, as they require minimal adjustments to work with CodSpeed.

How does CodSpeed work with Rust benchmarks?

Rust, being a compiled language, has CodSpeed integrations that differ from those for interpreted languages. The CodSpeed benchmarking process for Rust occurs at both build time and runtime.

To facilitate this, CodSpeed provides:

1. A cargo-codspeed cargo subcommand: used regardless of the benchmarking crate.
2. Compatibility layers for popular benchmarking crates: chose the appropriate one based on your project’s needs.

cargo-codspeed

To integrate CodSpeed with your Rust codebase, use the cargo subcommand: cargo-codspeed. This tool allows you to run CodSpeed benchmarks without modifying the behavior of the standard cargo bench command.

Creating benchmarks with cargo-codspeed is the same as with the supported APIs. So if you already have benchmarks written with one of these, only a minor import change is required.

Due to a limitation of cargo, we currently do not support build.rustflags in .cargo/config.toml file when using cargo-codspeed.

However, you can use [target.'cfg(all())'] instead of [build] as a workaround which does the same and is compatible.

[build]                                       
[target.'cfg(all())']                         
rustflags = ["--cfg", "your_feature_flag"]

Installation

To check your benchmarks with CodSpeed, you first need to install the cargo-codspeed CLI tool:

cargo install cargo-codspeed --locked

This tool can then be used directly within cargo:

terminal

$ cargo codspeed
Cargo extension to build & run your codspeed benchmarks

Usage: cargo codspeed <COMMAND>

Commands:
  build  Build the benchmarks
  run    Run the previously built benchmarks

Options:
  -h, --help     Print help information
  -V, --version  Print version information

Usage

No matter which benchmarking crate you’re using, the cargo-codspeed command is used to help you build and run the benchmark in a CodSpeed environment.

terminal

$ cargo codspeed build
    Finished release [optimized] target(s) in 0.12s
    Finished built 1 benchmark suite(s)
$ cargo codspeed run
   Collected 1 benchmark suite(s) to run
     Running example
Using codspeed-bencher-compat v1.0.0 compatibility layer
NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.
Checked: benches/example.rs::a (group: benches)
Checked: benches/example.rs::b (group: benches)
        Done running bencher_example
    Finished running 1 benchmark suite(s)

Use --measurement-mode / -m to select the CodSpeed instrument:

• simulation (default): Runs benchmarks once on a simulated CPU for consistent measurements.
• walltime: Measures wall-clock time for real-world scenarios.
• memory: Benchmarks are run once using memory profiling to track heap allocations and memory usage.

See the cargo-codspeed reference for more information.

Advanced build options

By default, cargo codspeed build will build all the benchmark executables of your workspace. But you can also be more specific with the following options:

Cargo Workspaces

If you’re using CodSpeed within a workspace you can use the -p flag to specify the crate to run the build command on:

cargo codspeed build -p my_package

Build only specific benchmark executables

With the following folder structure:

benches/
  ├── bench1.rs
  └── bench2.rs

To build only bench1, you can pass its name as the --bench flag:

cargo codspeed build --bench bench1
# Repeat this argument to build multiple benchmark executables
cargo codspeed build --bench bench1 --bench bench2

Feature flags

If you’re using feature flags in your benchmark suite, you can use the --features flag to specify the features to enable:

cargo codspeed build --features my_feature

Advanced run options

By default, cargo codspeed run will run all the built benchmarks (of the latest cargo codspeed build ... command you ran).

To run only a subset of the built benchmarks, you can do the following:

# Run all the benchmark executables of the `my_package` crate
cargo codspeed run -p my_package
# Run only the `bench1` benchmark executable
cargo codspeed run --bench bench1
# Run only benches containing `foo` in their name
cargo codspeed run foo
# Run only benches matching the `foo.*bar` regex
cargo codspeed run "foo.*bar"
# You can combine these options
cargo codspeed run -p my_package --bench bench1 foo

Building multiple instruments

When using multiple measurement modes, you need to build your benchmarks with all the required instruments. Use the -m flag to specify each mode:

# Build with both simulation and memory instruments
cargo codspeed build -m memory -m simulation

The resulting binaries are compatible across these modes, so you only need to build once. At runtime, cargo codspeed run will execute the benchmarks for each mode specified in your CI workflow.

To combine measurement modes like simulation and memory, check out the documentation on running multiple instruments serially.

Continue by choosing a benchmarking crate

divan (recommended)

The most convenient way to run your Rust benchmarks

criterion.rs

A benchmarking crate inspired by the criterion haskell library

libtest (bencher)

The libtest (unstable) benchmark runner