Running Benchmarks in GitLab CI

Learn how to setup CodSpeed and run benchmarks within your GitLab CI workflow

Running benchmarks in CI environments presents unique challenges due to the inherent noise and variability of shared cloud infrastructure. Standard hosted runners can exhibit significant performance variance.

Read our detailed post on how CI noise affects benchmark consistency.

CodSpeed instruments are designed to mitigate these challenges and gather accurate performance data even in noisy environments. The easiest way to get started running benchmarks in GitLab CI is to use the CodSpeed Runner directly.

For now, only the following OS and versions are supported on the runners:

• Ubuntu 22.04 and later
• Debian 12 and later

1. Create the benchmarks job

Create a new job to run the benchmarks for your repository.

For example, you can add this job to your existing pipeline by adding a section in your .gitlab-ci.yml file with the following content:

.gitlab-ci.yml

workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

codspeed:
  stage: test
  image: <your-docker-image> # for example: python:3.12 or rust:1.82
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    # ...
    # Setup your environment here:
    # - Configure your Python/Rust/Node version
    # - Install your dependencies
    # - Build your benchmarks (if using a compiled language)
    # ...

    # Use the latest CodSpeed version
    - curl -fsSL https://codspeed.io/install.sh | bash
    # Or set a specific version. Refer to https://github.com/CodSpeedHQ/codspeed/releases for available versions. For example:
    # - curl -fsSL https://codspeed.io/4.2.1/install.sh | bash
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode simulation -- "<Insert your benchmark command here>"

Make sure to run the job on merge_request_event. This is required to have reports on merge requests correctly working.

Learn more about baseline report selection.

Sample configurations

Python

.gitlab-ci.yml

workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
codspeed:
  stage: test
  image: python:3.12
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL https://codspeed.io/install.sh  | bash
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode simulation -- pytest tests/ --codspeed

More info on how to setup Python benchmarks.

Rust

.gitlab-ci.yml

workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
codspeed:
  stage: test
  image: rust:1.82
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - curl -fsSL https://codspeed.io/install.sh  | bash
    - source $HOME/.cargo/env
    - cargo install cargo-codspeed --locked
    # Build the benchmark target(s)
    - cargo codspeed build
  script:
    # Run the benchmarks
    - codspeed run --mode simulation -- cargo codspeed run

More info on how to setup Rust benchmarks.

Node.js

.gitlab-ci.yml

workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
codspeed:
  stage: test
  image: nodejs:22
  id_tokens:
    # OpenID Connect token to authenticate with CodSpeed
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - npm install
    - curl -fsSL https://codspeed.io/install.sh  | bash
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode simulation -- node -r esbuild-register benches/bench.ts

More info on how to setup Node.js benchmarks.

CodSpeed only currently supports docker images that are based on Ubuntu or Debian.

2. Check the results

Once the workflow is created, your merge requests will receive a performance report comment and will also receive some additional checks:

Merge Request Setup Comment with detected benchmarks

Merge Request Status Check with performance results

3. Next Steps

Now that everything is up and running (and hopefully green 🎉), you can start enhancing your workflow to get the most out of CodSpeed.

Configure GitLab CI for CodSpeed

Learn how to configure authentication methods and advanced options for GitLab CI

Explore the Performance Metrics

Understand the performance metrics generated by CodSpeed

Enforce Performance Checks

Make sure you or team members never merge unexpected performance regressions

Explore Profiling

Get detailed flame graphs and performance traces for your benchmarks

Shard the execution of your benchmarks

Run your benchmarks in parallel to speed up your CI