Configuring GitLab CI for CodSpeed

Learn how to configure GitLab CI to run benchmarks with CodSpeed.

Authentication

In order to upload benchmark results to CodSpeed, the GitLab CI job needs to authenticate with CodSpeed. There are two supported methods for authentication: OpenID Connect (OIDC) and static CodSpeed tokens.

OIDC (Recommended)

CodSpeed recommends using OpenID Connect (OIDC) for authentication.

Using this method, a token is generated on-the-fly during the workflow run. This token is then used to authenticate securely with CodSpeed without needing to store long-lived credentials, but grants no additional permissions to the workflow.

Configure your GitLab CI job to use the CODSPEED_TOKEN as an OIDC token by adding the following section to your job definition:

id_tokens:
  CODSPEED_TOKEN:
    aud: codspeed.io

For more information on setting up OIDC in GitLab CI, refer to the GitLab documentation on OIDC.

CodSpeed token (Legacy)

While we recommend using OpenID Connect (OIDC) for improved security, you can use a static CodSpeed token for authentication.

Retrieve your CodSpeed token from your repository settings on CodSpeed

Be mindful that a token is scoped to a specific repository. Make sure that you are on the correct repository settings page when copying the token.

Then, create a new CI/CD variable in your repository with the name CODSPEED_TOKEN and the value of your token.

Set the CODSPEED_TOKEN variable as "Masked", but not "Protected", because it must be available on all branches.

Advanced

Running benchmarks in parallel CI jobs

With GitlabCI, you can split your benchmarks across multiple jobs.

For example with pytest:

.gitlab-ci.yml

workflow:
  # run on merge requests and pushes on the default branch
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

codspeed-1:
  stage: test
  image: python:3.12
  id_tokens:
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL https://codspeed.io/install.sh | bash
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode instrumentation -- pytest tests/ --codspeed --test-group=1 --test-group-count=2

codspeed-2:
  stage: test
  image: python:3.12
  id_tokens:
    CODSPEED_TOKEN:
      aud: codspeed.io
  before_script:
    - pip install -r requirements.txt
    - curl -fsSL https://codspeed.io/install.sh | bash
    - source $HOME/.cargo/env
  script:
    - codspeed run --mode instrumentation -- pytest tests/ --codspeed --test-group=2 --test-group-count=2

CodSpeed will only supports emitting results from your benchmarks if you split them within a single CI workflow.

If you run benchmarks in multiple CI workflows, CodSpeed will not be able to aggregate the results correctly, and you may see incomplete or missing data in your CodSpeed reports.

When running benchmarks across multiple jobs, all jobs must use the same authentication method (either all OIDC, all static tokens, or all tokenless). Mixing authentication methods will prevent CodSpeed from aggregating the results correctly.

Learn more about benchmark sharding and how to integrate with your CI provider.