Benchmarks

Benchmarks measure the execution time of your scripts with dedicated functions, helping you identify performance bottlenecks and ensure your code meets performance requirements.

Quick Start

Create a benchmark file with functions prefixed with bench:

tests/benchmark/example_bench.sh

#!/usr/bin/env bash

# @revs=100 @its=5
function bench_my_function() {
  my_function_under_test
}

Running

./bashunit bench tests/benchmark/

Annotations

Control benchmark behavior with comment annotations placed before the function:

Annotation	Description	Default
@revs=N	Number of revolutions (function calls per iteration)	1
@its=N	Number of iterations (separate processes)	1
@max_ms=N	Maximum allowed average time in milliseconds (decimals allowed, e.g. 1.5)	-

Basic benchmark

# @revs=1000 @its=5
function bench_string_operations() {
  local result="${text//foo/bar}"
}

With threshold

# @revs=10 @its=3 @max_ms=50
function bench_api_call() {
  curl -s "$API_URL/health" > /dev/null
}

TIP

Each iteration runs in a separate process, providing isolated timing measurements. Higher @its values give more reliable averages.

Running Benchmarks

Run benchmarks using the bench command:

Run all benchmarks

./bashunit bench tests/benchmark/

Run specific file

./bashunit bench tests/benchmark/string_bench.sh

With simple output

./bashunit bench --simple tests/benchmark/

If no file is provided, bashunit uses BASHUNIT_DEFAULT_PATH to locate all *bench.sh files.

A run that finds nothing to measure exits non-zero, the same way bashunit test does. A typo in the path or in the prefix would otherwise leave a green CI job that benchmarked nothing, which is easy to miss when the whole output is numbers. bench gives the same two answers test does:

$ bashunit bench bnech/
Error: no such path: 'bnech/'.

$ bashunit bench holds_no_bench_function.sh
No benchmarks found

A path that is not on disk is a wrong invocation and is named. A file or directory that exists but holds no bench_ function is an empty selection, and keeps reporting No benchmarks found.

Output Formats

Simple Output

Shows progress dots during execution, followed by a summary table:

Command

./bashunit bench --simple

Output

.........

Benchmark Results (avg ms)
======================================================================

Name                                 Revs    Its    Avg(ms)     Status
bench_string_operations               100      5         12
bench_api_call                         10      3         45       ≤ 50
bench_slow_function                    50      2        150      > 100

Detailed Output

Shows timing for each iteration as it runs:

Command

./bashunit bench

Output

Running tests/benchmark/example_bench.sh
Bench string operations [1/5] 13 ms
Bench string operations [2/5] 11 ms
Bench string operations [3/5] 12 ms
Bench string operations [4/5] 12 ms
Bench string operations [5/5] 11 ms
Bench api call [1/3] 43 ms
Bench api call [2/3] 47 ms
Bench api call [3/3] 45 ms


Benchmark Results (avg ms)
=====================================================================

Name                                 Revs    Its    Avg(ms)    Status
bench_string_operations               100      5         12
bench_api_call                         10      3         45      ≤ 50

Machine-readable reports

The console table is for a human reading a terminal. For CI, write the run to a file that can be stored as an artifact, charted over time, or compared by a later run:

JSON

./bashunit bench --report-json bench.json

JUnit

./bashunit bench --report-junit bench.xml

bench.json

{
  "run": {
    "timestamp": "2026-08-12T21:52:36",
    "duration_ms": 209,
    "bashunit_version": "0.46.0",
    "bash_version": "5.2.21",
    "os": "Linux"
  },
  "benchmarks": [
    {
      "file": "tests/benchmark/example_bench.sh",
      "function": "bench_api_call",
      "name": "Bench api call",
      "revs": 10,
      "its": 3,
      "iterations_ms": [43, 47, 45],
      "average_ms": 45,
      "min_ms": 43.000,
      "max_ms": 47.000,
      "median_ms": 45.000,
      "threshold_ms": 50,
      "within_threshold": true
    }
  ]
}

threshold_ms and within_threshold are null for a benchmark with no @max_ms — an absent threshold is not a threshold of zero, and a chart has to be able to tell those apart.

The JUnit file reports one <testcase> per benchmark, with a <failure type="PerformanceRegression"> for each one over its @max_ms, so a CI test reporter shows benchmarks next to tests.

Failing on a regression

@max_ms is an absolute ceiling. To survive the slowest CI runner it has to be loose, so it only catches catastrophes — a 30% slowdown that stays under the ceiling passes. --baseline compares this run against a previous one instead:

Record

./bashunit bench --baseline-update bench-baseline.json

Compare

./bashunit bench --baseline bench-baseline.json

Stricter gate

./bashunit bench --baseline bench-baseline.json --baseline-tolerance 5

Output

Baseline comparison (median ms, tolerance 10%)
================================================================================
Name                                     Baseline      Current      Delta
bench_string_operations                    12.000       12.400      +3.3%
bench_api_call                             45.000       58.000     +28.9%
bench_added_later                               -        7.000        new
bench_deleted                               3.000            -    removed

Performance regression in 1 benchmark(s)

• The run exits non-zero when a benchmark is slower than baseline × (1 + tolerance/100). The default tolerance is 10 percent.
• An improvement never fails the run, and the delta is printed either way.
• A benchmark the baseline does not know is reported as new, not as a failure; one only in the baseline is reported as removed. Neither is fatal: the commit that deletes a benchmark must not fail on its absence.
• A missing or malformed baseline file exits non-zero. A gate that quietly compares against nothing would report every run green, which is worse than no gate at all.

The comparison uses the median, not the average: one descheduled iteration on a shared runner moves the mean far more than the middle value, and a gate that cries wolf gets disabled.

The baseline file is simply the --report-json document of an earlier run, so a CI job can publish today's run as an artifact and the next one can compare against it.

Status Column

The status column indicates threshold results:

Status	Meaning
(empty)	No threshold set
≤ N	Average time is at or below @max_ms threshold (pass)
> N	Average time exceeds @max_ms threshold (fail)

Setting Thresholds

Use @max_ms to fail benchmarks that exceed a time limit:

Example

# @revs=10 @its=3 @max_ms=100
function bench_critical_path() {
  process_request "$test_data"
}

WARNING

Thresholds are checked against the average time across all iterations. A single slow iteration won't cause failure if the average remains acceptable.

Best Practices

Isolate the Code Under Test

Minimize setup code inside the benchmark function:

Good

function set_up() {
  TEST_DATA=$(generate_large_dataset)
}

# @revs=100 @its=5
function bench_process_data() {
  process "$TEST_DATA"
}

Avoid

# @revs=100 @its=5
function bench_process_data() {
  local data=$(generate_large_dataset)  # Measured!
  process "$data"
}

Choose Appropriate Revolutions

• I/O operations (network, disk): Lower @revs (1-10)
• CPU operations (string processing, math): Higher @revs (100-1000)

Run Multiple Iterations

Use @its >= 3 for more reliable averages, especially for operations with variable timing.

Related

• Command-Line — full reference for CLI flags and options
• Configuration — environment variables such as BASHUNIT_DEFAULT_PATH
• Coverage — measure which code paths your tests exercise