> ## Documentation Index
> Fetch the complete documentation index at: https://codspeed.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Configuring CircleCI for CodSpeed
> Learn how to configure CircleCI to run benchmarks with CodSpeed.
## Authentication
In order to upload benchmark results to CodSpeed, the CircleCI job needs to
authenticate with CodSpeed. There are two supported methods for authentication:
OpenID Connect (OIDC) and static CodSpeed tokens.
### OIDC (Recommended)
CodSpeed recommends using
[OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/) for
authentication.
Using this method, a token is generated on-the-fly during the workflow run. This
token is then used to authenticate securely with CodSpeed without needing to
store long-lived credentials, but grants no additional permissions to the
workflow.
On CircleCI, this works by default: a job that sets no `CODSPEED_TOKEN`
authenticates with an OIDC token, and there is nothing to configure for it. If a
job cannot use OIDC, authenticate it with a [CodSpeed token](#codspeed-token).
### CodSpeed token
Some jobs cannot use OIDC and need a static CodSpeed token instead:
* Pull requests opened from a fork, whose token names the fork rather than your
repository.
* Pipelines triggered by a custom webhook, whose token names no repository at
all.
* Jobs running on an image that does not ship the `circleci` CLI, which the
CodSpeed CLI needs to request the token.
Retrieve your CodSpeed token from your repository settings on CodSpeed:
Be mindful that a token is scoped to a specific repository. Make sure that you
are on the correct repository settings page when copying the token.
Then add it as a
[project environment variable](https://circleci.com/docs/set-environment-variable/#set-an-environment-variable-in-a-project)
or in a [context](https://circleci.com/docs/contexts/), with the name
`CODSPEED_TOKEN`. The CodSpeed CLI reads it from the job environment.
## Project settings
### Pipeline trigger
CodSpeed recommends the **PR opened or pushed to, default branch and tag
pushes** event for the benchmarks pipeline: it builds the default branch,
recording the baselines, and builds pull requests, which is what CodSpeed
reports on. See
[the setup guide](/docs/integrations/ci/circleci#2-create-the-benchmarks-pipeline)
for how to set it.
### Running the benchmarks in an existing pipeline
You can add the benchmarks to a pipeline you already have. Its trigger covers
every job in it, so restricting the benchmarks to pull requests restricts the
rest too.
CircleCI does not rebuild a branch it has already built, so a branch built
before you changed the trigger receives no performance report until you push a
new commit to it.
## Legacy GitHub OAuth projects
CircleCI has
[two types of GitHub integration](https://circleci.com/docs/guides/integration/using-the-circleci-github-app-in-an-oauth-org/#two-types-of-github-integration):
the GitHub App, which the [setup guide](/docs/integrations/ci/circleci) follows, and
the GitHub OAuth app it replaces. CodSpeed supports both.
An OAuth project holds a single pipeline, and its configuration has to live in
`.circleci/config.yml`. The benchmarks job goes in that file, alongside your
other jobs, rather than in a dedicated one:
```yaml .circleci/config.yml theme={null}
version: 2.1
jobs:
tests:
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install -r requirements.txt
- run: pytest tests/
benchmarks:
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install -r requirements.txt
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- pytest tests/ --codspeed
workflows:
ci:
jobs:
- tests
- benchmarks
```
The job needs no upload token: with `CODSPEED_TOKEN` unset, it authenticates
with an [OIDC token](#oidc-recommended). Its [trigger](#pipeline-trigger) is the
one that pipeline already has, so
[the same tradeoff applies](#running-the-benchmarks-in-an-existing-pipeline).
If you use an OAuth project with a trigger that builds **every push** and the
pull request is opened after the branch has already been built, the CodSpeed
report may not appear on the pull request. In that case, push a new commit to
the branch to get the report.
To bypass these restrictions, CodSpeed recommends installing the CircleCI GitHub
App, which can be installed alongside an existing GitHub OAuth pipeline, and
giving the benchmarks a pipeline of their own as the
[setup guide](/docs/integrations/ci/circleci) does.
## Advanced
### CLI version
The examples pin the CLI version in the install URL, which is what CodSpeed
recommends: the tools an instrument needs are pinned to it, so a version that
moves under you can
[shift the measurements](/docs/instruments/cpu/regression-causes). All versions are
listed on the [releases page](https://github.com/CodSpeedHQ/codspeed/releases).
To install the latest version on every job instead, drop the version from the
URL:
```yaml .circleci/codspeed.yml theme={null}
steps:
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
```
The second command puts `codspeed` on the `PATH` of the steps that follow, since
every `run` step starts a fresh shell. Refer to the CircleCI documentation on
[setting an environment variable in a shell command](https://circleci.com/docs/set-environment-variable/#set-an-environment-variable-in-a-shell-command)
for more details.
### Running benchmarks in parallel CI jobs
Splitting a benchmark suite across several jobs cuts the time a run takes. On
CircleCI, there are two ways to do it: a matrix of jobs, or the `parallelism`
key.
CodSpeed only supports emitting results from your benchmarks if you split them
within a single CI workflow.
If you run benchmarks in multiple CI workflows, CodSpeed will not be able to
aggregate the results correctly, and you may see incomplete or missing data in
your CodSpeed reports.
A CircleCI pipeline can hold several workflows, so keep every benchmark job in
the same one.
#### Matrix jobs
A matrix declares one job per shard. Reach for it when the shards differ by more
than a number, for example when each one runs a different benchmark command.
For example with `pytest`. The `--test-group` options come from
[`pytest-test-groups`](/docs/benchmarks/python#running-benchmarks-in-parallel-ci-jobs),
which the benchmarks job needs installed alongside your other dependencies:
```yaml .circleci/codspeed.yml theme={null}
version: 2.1
jobs:
benchmarks:
parameters:
shard:
type: integer
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install -r requirements.txt
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- \
pytest tests/ --codspeed --test-group=<< parameters.shard >> --test-group-count=2
workflows:
benchmarks:
jobs:
- benchmarks:
matrix:
parameters:
shard: [1, 2]
```
CodSpeed aggregates the results of every job of the workflow into a single
report:
#### The `parallelism` key
`parallelism` runs a job on several identical containers. They all run the same
steps, so the split comes from `CIRCLE_NODE_INDEX`, numbered from `0`, and
`CIRCLE_NODE_TOTAL`. The example shifts the index by one, since
`pytest-test-groups` numbers its groups from `1`:
```yaml .circleci/codspeed.yml theme={null}
version: 2.1
jobs:
benchmarks:
parallelism: 2
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install -r requirements.txt
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- \
pytest tests/ --codspeed \
--test-group=$((CIRCLE_NODE_INDEX + 1)) \
--test-group-count=$CIRCLE_NODE_TOTAL
workflows:
benchmarks:
jobs:
- benchmarks
```
CodSpeed records each container as its own part of the run.
A job that does not split its benchmarks runs the whole benchmark suite on every
container. CodSpeed does not support the same benchmark running several times in
a run, and the pull request receives this comment instead of a performance
report:
CircleCI can also split a list of file names across the containers itself, which
fits benchmark commands that take files as arguments. Refer to the CircleCI
documentation on
[test splitting and parallelism](https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/).
Learn more about
[benchmark sharding and how to integrate with your CI provider](/docs/features/sharded-benchmarks).
### Benchmarks in several languages
Benchmarks written in several languages run in one job per language, in the same
workflow. CodSpeed aggregates their results into a single performance report.
For example, with Python and Rust benchmarks:
```yaml .circleci/codspeed.yml theme={null}
version: 2.1
jobs:
python-benchmarks:
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install -r requirements.txt
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- pytest tests/ --codspeed
rust-benchmarks:
docker:
- image: cimg/rust:1.82
steps:
- checkout
- run:
name: Install the CodSpeed CLI
command: |
curl -fsSL https://codspeed.io/v5.1.0/install.sh | bash
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$BASH_ENV"
- run: cargo install cargo-codspeed --locked
# Build the benchmark target(s)
- run: cargo codspeed build
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- cargo codspeed run
workflows:
benchmarks:
jobs:
- python-benchmarks
- rust-benchmarks
```
### Caching the installed instruments
The CodSpeed CLI installs the tools its instruments need, such as valgrind for
CPU simulation, on every job. Install them in a step of their own with
`codspeed setup`, pointed at a directory you cache, and later jobs restore them
instead of installing them again:
```yaml .circleci/codspeed.yml theme={null}
steps:
- restore_cache:
keys:
- v1-codspeed-instruments-{{ arch }}
- run:
name: Install the CodSpeed instruments
command:
codspeed setup --mode simulation --setup-cache-dir ~/.cache/codspeed
- save_cache:
key: v1-codspeed-instruments-{{ arch }}
paths:
- ~/.cache/codspeed
- run:
name: Run the benchmarks
command: |
codspeed run --mode simulation -- pytest tests/ --codspeed
```
Only the setup step takes `--setup-cache-dir`, since the instruments it installs
are already in place when the benchmarks run. Caching right after it, rather
than after the benchmarks, means a benchmark failure does not cost the next run
the install.
The tools are pinned to the CLI version, so include that version in the cache
key when you install a pinned CLI rather than the latest one.
### Executors
Both the `machine` and the `docker` executor are supported:
* `machine`, with an Ubuntu 22.04 or later image, for example
`ubuntu-2404:current`.
* `docker`, with an image based on Ubuntu 22.04 or later, or Debian 12 or later.
CircleCI's [`cimg` images](https://circleci.com/developer/images) qualify.
Keep a benchmark job's executor stable over time, so that a run and the baseline
it is compared against are measured in the same
[runtime environment](/docs/instruments/cpu/regression-causes#ci-runner-variability).
Two jobs can use different executors, as
[benchmarks in several languages](#benchmarks-in-several-languages) do.
CircleCI has full support for the [CPU simulation](/docs/instruments/cpu) and
[memory](/docs/instruments/memory) instruments. The [walltime](/docs/instruments/walltime)
instrument will run on a CircleCI executor, but not produce reliable results
without a dedicated runner: run it on a
[macro runner](/docs/integrations/ci/circleci/macro-runners) instead.