> ## Documentation Index
> Fetch the complete documentation index at: https://codspeed.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Writing Benchmarks in Scala

> Create benchmarks for your Scala codebase using Java Microbenchmark Harness

> [!NOTE]
>
> The JVM integration is still in early development, and **only the [walltime
> instrument](https://codspeed.io/docs/instruments/walltime.md) is currently supported**.
>
> If you have any feedback, please reach out to us via
> [Discord](https://discord.gg/MxpaCfKSqF) or
> [email our support](mailto:contact@codspeed.io).

Integrating CodSpeed into your Scala project works through a fork of
[JMH](https://github.com/openjdk/jmh) (Java Microbenchmark Harness). You write
standard JMH benchmarks in Scala and swap in the CodSpeed JMH fork as a
dependency. When running in CI with CodSpeed, the results are automatically
collected and reported.

JMH generates its measurement harness from the compiled `@Benchmark` classes, so
no annotation processing setup is needed. Scala only changes how benchmark
classes have to be declared, and which generator a Maven build has to run.

> [!NOTE]
>
> This page covers Gradle and Maven. If you benchmark with
> [sbt-jmh](https://github.com/sbt/sbt-jmh), let us know on
> [Discord](https://discord.gg/MxpaCfKSqF) so we can prioritize it.

## Installation

CodSpeed provides a fork of JMH that collects walltime results and sends them to
CodSpeed.

**Gradle**

Add the CodSpeed JMH fork as a Git submodule:

```bash theme={null}
git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
git submodule update --init --recursive -- third-party/codspeed-jvm
```

Then include it as a composite build in your `settings.gradle.kts`, with
dependency substitution to redirect JMH dependencies to the CodSpeed fork:

```kotlin settings.gradle.kts theme={null}
includeBuild("third-party/codspeed-jvm/jmh-fork") {              // [!code ++]
    dependencySubstitution {                          // [!code ++]
        substitute(module("org.openjdk.jmh:jmh-core"))           // [!code ++]
            .using(project(":jmh-core"))              // [!code ++]
        substitute(module("org.openjdk.jmh:jmh-generator-annprocess")) // [!code ++]
            .using(project(":jmh-generator-annprocess")) // [!code ++]
    }                                                 // [!code ++]
}                                                     // [!code ++]
```

We recommend using the
[JMH Gradle Plugin](https://github.com/melix/jmh-gradle-plugin) as it handles
benchmark compilation and provides the `jmh` task. It generates the harness from
the compiled Scala classes in `src/jmh/scala`, which need the Scala library
added to the `jmh` configuration:

```kotlin build.gradle.kts theme={null}
plugins {
    scala
    id("me.champeau.jmh") version "0.7.3"// [!code ++]
}

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(21))// [!code ++]
    }
}

dependencies {
    implementation("org.scala-lang:scala3-library_3:3.3.6")
    jmh("org.scala-lang:scala3-library_3:3.3.6")// [!code ++]
}
```

**Maven**

Add the CodSpeed JMH fork as a Git submodule and publish it to your local Maven
repository:

```bash theme={null}
git submodule add https://github.com/CodSpeedHQ/codspeed-jvm.git third-party/codspeed-jvm
git submodule update --init --recursive -- third-party/codspeed-jvm
cd third-party/codspeed-jvm
./gradlew -p jmh-fork publishToMavenLocal
```

We're planning to publish to Maven Central in the future. If you need this,
please reach out via [Discord](https://discord.gg/MxpaCfKSqF) or
[email](mailto:contact@codspeed.io).

Then replace the JMH dependencies in your `pom.xml` with the CodSpeed fork.
Maven resolves from `~/.m2/repository` by default. Scala projects drive the JMH
bytecode generator, so the fork has to be substituted in two places: the
`jmh-core` dependency and the generator used by `exec-maven-plugin`:

```xml pom.xml theme={null}
<properties>
    <jmh.version>1.37</jmh.version>// [!code --]
    <jmh.version>0.2.1</jmh.version>// [!code ++]
</properties>

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>// [!code --]
        <groupId>io.codspeed.jmh</groupId>// [!code ++]
        <artifactId>jmh-core</artifactId>
        <version>${jmh.version}</version>
    </dependency>
</dependencies>

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <dependencies>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>// [!code --]
            <groupId>io.codspeed.jmh</groupId>// [!code ++]
            <artifactId>jmh-generator-bytecode</artifactId>
            <version>${jmh.version}</version>
        </dependency>
    </dependencies>
</plugin>
```

> [!NOTE]
>
> Your Maven project must be configured to compile Scala sources, run the JMH
> generator over the compiled classes, and package an executable benchmark JAR.
> The `jmh-scala-benchmark-archetype` from JMH generates a project with all of
> this wired up:
>
> ```bash theme={null}
> mvn archetype:generate -DinteractiveMode=false \
>   -DarchetypeGroupId=org.openjdk.jmh \
>   -DarchetypeArtifactId=jmh-scala-benchmark-archetype \
>   -DarchetypeVersion=1.37 \
>   -DgroupId=com.example -DartifactId=scala-benchmarks -Dversion=1.0
> ```

> [!WARNING]
>
> The archetype pins Scala 2.11.8 and `scala-maven-plugin` 3.2.2, which cannot
> compile against JDK 21. Update both, and point the dependency at the matching
> standard library:
>
> ```xml pom.xml theme={null}
> <dependency>
>     <groupId>org.scala-lang</groupId>
>     <artifactId>scala-library</artifactId>// [!code --]
>     <artifactId>scala3-library_3</artifactId>// [!code ++]
>     <version>${scala.stdLib.version}</version>
> </dependency>
>
> <properties>
>     <scala.stdLib.version>2.11.8</scala.stdLib.version>// [!code --]
>     <scala.stdLib.version>3.3.6</scala.stdLib.version>// [!code ++]
>     <scala.mavenPlugin.version>3.2.2</scala.mavenPlugin.version>// [!code --]
>     <scala.mavenPlugin.version>4.9.2</scala.mavenPlugin.version>// [!code ++]
> </properties>
> ```

## Creating benchmarks

Write your benchmarks using standard JMH annotations:

```scala FibBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.Benchmark

class FibBenchmark {

  @Benchmark
  def fib(): Long = fib(30)

  private def fib(n: Int): Long =
    if (n <= 1) n.toLong else fib(n - 1) + fib(n - 2)
}
```

> [!WARNING]
>
> Benchmarks must live in a `class`, not an `object`. JMH subclasses the
> benchmark class to generate the measurement harness, and an `object` compiles
> to a final class, which fails the build with `Benchmark classes should not be
>     final`.

> [!NOTE]
>
> JMH benchmarks must return their result or use `Blackhole.consume()` to
> prevent the JVM from eliminating dead code. All examples on this page return
> the computed value.

## Testing the benchmarks locally

To run the benchmarks with CodSpeed locally, first install the `codspeed`
runner:

```bash theme={null}
curl -fsSL https://codspeed.io/install.sh | sh
```

Then run your benchmarks with CodSpeed:

**Gradle**

\$ codspeed run --mode walltime -- ./gradlew jmh
►►► Running the benchmarks
\# JMH version: 0.2.1
\# VM version: JDK 21.0.9, OpenJDK 64-Bit Server VM, 21.0.9+10
\# Warmup: 3 iterations, 1 s each
\# Measurement: 3 iterations, 1 s each
\# Benchmark: bench.FibBenchmark.fib

\# Run progress: 0.00% complete, ETA 00:00:18
\# Warmup Iteration   1: 286.841 ops/s
...

Benchmark          Mode  Cnt    Score    Error  Units
FibBenchmark.fib  thrpt    9  305.213 ± 44.639  ops/s

**Maven**

\$ mvn package -q
\$ codspeed run --mode walltime -- java -jar target/benchmarks.jar
►►► Running the benchmarks
\# JMH version: 0.2.1
\# VM version: JDK 21.0.9, OpenJDK 64-Bit Server VM, 21.0.9+10
\# Warmup: 3 iterations, 1 s each
\# Measurement: 3 iterations, 1 s each
\# Benchmark: com.example.MyBenchmark.fib

\# Run progress: 0.00% complete, ETA 00:00:18
\# Warmup Iteration   1: 279.817 ops/s
...

Benchmark         Mode  Cnt    Score   Error  Units
MyBenchmark.fib  thrpt    9  284.008 ± 4.488  ops/s

## Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This
allows CodSpeed to automatically run benchmarks and warn you about regressions
during development.

> [!TIP]
>
> If you want more details on how to configure the CodSpeed action, you can check
> out the [Continuous Reporting section](https://codspeed.io/docs/integrations/ci.md).

Here is an example of a GitHub Actions workflow that runs the benchmarks and
reports the results to CodSpeed on every push to the `main` branch and every
pull request:

**Gradle**

```yaml .github/workflows/codspeed.yml icon="github" theme={null}
name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: recursive
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: walltime
          run: ./gradlew jmh
```

**Maven**

```yaml .github/workflows/codspeed.yml icon="github" theme={null}
name: CodSpeed Benchmarks

on:
  push:
    branches:
      - "main" # or "master"
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: codspeed-macro-arm64-graviton-ubuntu-22-04
    permissions: # optional for public repositories
      contents: read # required for actions/checkout
      id-token: write # required for OIDC authentication with CodSpeed
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: recursive
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
      - name: Install CodSpeed JMH fork
        run: cd third-party/codspeed-jvm && ./gradlew -p jmh-fork publishToMavenLocal
      - name: Build benchmark JAR
        run: mvn package -q
      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v5
        with:
          mode: walltime
          run: java -jar target/benchmarks.jar
```

## Advanced usage

JMH provides many features for writing expressive benchmarks. Below is a
selection that can be useful in CodSpeed benchmarks.

### Parameterized benchmarks

Use `@Param` to run the same benchmark with different input values. Scala passes
the values as an `Array`, and the annotated field has to be a `var`, since JMH
assigns it before the benchmark runs and rejects final fields:

```scala ParamBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.{Benchmark, Param, Scope, State}

@State(Scope.Benchmark)
class ParamBenchmark {

  @Param(Array("10", "20", "30"))
  var n: Int = 0

  @Benchmark
  def fib(): Long = fib(n)

  private def fib(k: Int): Long =
    if (k <= 1) k.toLong else fib(k - 1) + fib(k - 2)
}
```

### Shared state

Use `@State` to share setup logic across benchmarks and control the scope of the
state object:

```scala StateBenchmark.scala theme={null}
package bench

import org.openjdk.jmh.annotations.{Benchmark, Scope, Setup, State}

@State(Scope.Benchmark)
class StateBenchmark {

  private var numbers: Vector[Int] = Vector.empty

  @Setup
  def setup(): Unit = {
    numbers = (0 until 1000).toVector
  }

  @Benchmark
  def sum(): Int = numbers.sum
}
```

JMH annotations behave the same in Scala as in Java. For a deeper dive, see the
dedicated guide:

> **[How to Benchmark Java with JMH](https://codspeed.io/docs/guides/how-to-benchmark-java-with-jmh.md)**
>
> An in-depth guide to writing JMH benchmarks: project setup, annotations,
> parameters, common pitfalls, and CodSpeed CI integration.

## Compatibility

* **JDK 21 or later** is required.
* Benchmark classes must not be final, since JMH subclasses them to generate the
  measurement harness.
* All standard JMH annotations are supported.
* Only the [Walltime instrument](https://codspeed.io/docs/instruments/walltime.md) is supported.
* CodSpeed uses a custom mode for all benchmarks to collect statistically
  significant results. Any `@BenchmarkMode` annotations in your code are
  ignored.

If you run into issues or require certain features, please
[open an issue](https://github.com/CodSpeedHQ/codspeed-jvm/issues) or
[join our Discord](https://discord.com/invite/MxpaCfKSqF) to get help.

## Next steps

> **[Example repository](https://github.com/CodSpeedHQ/codspeed-jvm)**
>
> The CodSpeed JVM repository with example JMH benchmarks.

> **[How to Benchmark Java with JMH](https://codspeed.io/docs/guides/how-to-benchmark-java-with-jmh.md)**
>
> An in-depth guide to writing JMH benchmarks with CodSpeed.

> **[Walltime instrument](https://codspeed.io/docs/instruments/walltime.md)**
>
> Learn more about the Walltime instrument and how to use it.

> **[Dive into performance changes](https://codspeed.io/docs/features/profiling.md)**
>
> Learn more about profiling and how to read flamegraphs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.