Configuration

Inputs

All inputs are configured in the workflow YAML under with:.

Required Inputs

github-token

GitHub token for API access. Default: ${{ github.token }}. The token needs pull-requests: write and issues: write permissions.

Conditional Inputs

results-path

Path to the directory containing downloaded ASV results. Required unless comparison-text-file is provided.

SHA Resolution

The action needs to know which commits to compare. There are several ways to provide this depending on the comparison mode:

For compare Mode (Default)

Option 2: Explicit SHAs
base-sha

Base commit SHA (the commit to compare against)

pr-sha

PR head commit SHA

Explicit inputs override metadata file values if both are provided.

Option 3: Direct File Paths
base-file

Direct path to base result JSON file

pr-file

Direct path to PR result JSON file

Use this when files are not named by SHA prefix, or when you know the exact paths (e.g., from a matrix workflow).

For compare-many Mode

Option 2: SHA-Based Lookup
baseline-sha

The baseline commit SHA. Used to find the baseline result file by SHA prefix.

contender-shas

Comma-separated contender commit SHAs. Used to find contender result files by SHA prefix.

Use this when each contender ran a different commit.

Comparison Mode

comparison-mode

Either compare (default, two-way) or compare-many (multi-way baseline vs multiple contenders).

YAML-Based Pipeline Inputs

These inputs let the action run benchmarks before comparing. Each entry can include environment activation via two fields:

  • run-prefix – A wrapper command prepended with a space. Use this for tools like pixi, conda, or nix that wrap the command.

  • setup – A shell command prepended with &&. Use this for sourcing scripts, exporting variables, or activating virtualenvs.

Both can be combined. The {sha} placeholder is replaced in all three fields. The constructed shell command is:

bash -c '<setup with {sha}> && <run-prefix with {sha}> <benchmark-command with {sha}>'

Each part is optional – omit what you don’t need. If an entry has no sha, benchmarking is skipped for that entry (useful when results already exist).

Contender benchmarks run in parallel via Promise.all. If your builds modify shared state (same build directory), use setup to isolate them.

Init command

Use init-command to run a one-time setup before any benchmarks:

init-command: pixi run bash -c "pip install asv && asv machine --yes"

This runs once, before any baseline or contender benchmarks.

Build pattern (C++, Fortran, Rust)

Use setup with {sha} to checkout and build per-commit:

baseline: |
  label: main
  sha: abc123def
  setup: >-
    git checkout -f {sha} && git clean -fd &&
    meson setup bbdir --prefix=$CONDA_PREFIX --buildtype release --wipe 2>/dev/null ||
    meson setup bbdir --prefix=$CONDA_PREFIX --buildtype release &&
    meson install -C bbdir
  run-prefix: pixi run

The {sha} placeholder is replaced in setup, run-prefix, and benchmark-command. This enables one action step to handle checkout, build, and benchmark for each entry.

Wrapper pattern (pixi, conda, nix)

Use run-prefix. The tool wraps the entire benchmark command:

baseline: |
  label: main
  sha: abc123def
  run-prefix: pixi run -e bench

This runs: pixi run -e bench asv run --record-samples abc123def^!

Source pattern (virtualenv, shell scripts)

Use setup. The script runs first, then the benchmark command:

baseline: |
  label: main
  sha: abc123def
  setup: source ./envs/bench.sh

This runs: source ./envs/bench.sh && asv run --record-samples abc123def^!

Combined (e.g. env vars + wrapper)

contenders: |
  - label: pr-debug
    sha: def456abc
    setup: export CFLAGS="-O0 -g"
    run-prefix: pixi run -e bench-debug

This runs: export CFLAGS”-O0 -g” && pixi run -e bench-debug asv run –record-samples def456abc^!=

Field Reference

baseline

YAML string defining the baseline entry. Fields:

  • label (string) – Display label for the baseline column.

  • sha (string) – Commit SHA for benchmarking and result lookup.

  • file (string) – Direct path to a result JSON file (bypasses SHA lookup).

  • run-prefix (string) – Wrapper command prepended to benchmark command (e.g. pixi run -e bench, conda run -n bench, nix develop -c).

  • setup (string) – Shell command sourced before the benchmark command (e.g. source ./envs/bench.sh, export CC=gcc-12).

  • env (string) – Environment name shown in PR comment details.

  • description (string) – Human-readable description shown in PR comment.

When baseline is provided, it overrides baseline-sha, baseline-file, and baseline-label.

contenders

YAML list of contender objects. Each object accepts the same fields as baseline (label, sha, file, run-prefix, setup, env, description). Works in both compare mode (first contender used as the “after” side) and compare-many mode (all contenders compared against baseline). When provided, this overrides contender-shas, contender-files, and contender-labels.

init-command

Shell command run once before any benchmarks. Use for one-time setup like asv machine --yes or stashing benchmark suites.

preserve-paths

Comma-separated paths to preserve across git checkouts. When an entry has a sha field, the action runs git checkout -f {sha} && git clean -fd automatically. These paths are stashed to /tmp before checkout and restored after. Example:

preserve-paths: benchmarks/, asv.conf.json

This eliminates the need for manual cp / stash boilerplate in setup. Without preserve-paths, the action does not run git checkout at all – the setup field must handle it.

benchmark-command

Shell command template for running benchmarks. Default: asv run --record-samples {sha}^!. The placeholder {sha} is replaced in this field, setup, and run-prefix. Each entry’s run-prefix is prepended and setup is sourced before this command. Entries without a sha field skip benchmarking.

Example with a custom command:

benchmark-command: 'asv run --quick {sha}^!'

Full Pipeline with C++ Build (Single Job)

Use preserve-paths to automatically stash/restore files across git checkouts. The setup field handles only the build – no checkout or copy boilerplate needed.

- uses: prefix-dev/setup-pixi@v0.8.10
  with:
    activate-environment: true
- uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: .asv/results/
    init-command: pixi run bash -c "pip install asv && asv machine --yes"
    preserve-paths: benchmarks/, asv.conf.json
    benchmark-command: >-
      asv run -E "existing:$(which python)"
      --set-commit-hash {sha} --record-samples --quick
    baseline: |
      label: main
      sha: ${{ github.event.pull_request.base.sha }}
      setup: >-
        pixi run bash -c "meson setup bbdir
        --prefix=$CONDA_PREFIX --libdir=lib
        --buildtype release --wipe 2>/dev/null
        || meson setup bbdir --prefix=$CONDA_PREFIX
        --libdir=lib --buildtype release" &&
        pixi run meson install -C bbdir
      run-prefix: pixi run
    contenders: |
      - label: pr
        sha: ${{ github.event.pull_request.head.sha }}
        setup: >-
          pixi run bash -c "meson setup bbdir
          --prefix=$CONDA_PREFIX --libdir=lib
          --buildtype release --wipe 2>/dev/null
          || meson setup bbdir --prefix=$CONDA_PREFIX
          --libdir=lib --buildtype release" &&
          pixi run meson install -C bbdir
        run-prefix: pixi run
    label-before: main
    label-after: pr
    runner-info: ubuntu-22.04

Full Pipeline with C++ Build (Matrix – Parallel)

For parallel execution, use a matrix strategy with a combine job. Each SHA gets its own runner. The commenter downloads combined results.

jobs:
  benchmark:
    strategy:
      fail-fast: false
      matrix:
        include:
          - label: main
            sha: ${{ github.event.pull_request.base.sha }}
          - label: pr
            sha: ${{ github.event.pull_request.head.sha }}
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Checkout target
        run: |
          mkdir -p /tmp/_bench_preserve
          cp -r benchmarks/ /tmp/_bench_preserve/
          cp asv.conf.json /tmp/_bench_preserve/
          git checkout -f ${{ matrix.sha }}
          git clean -fd
          cp -r /tmp/_bench_preserve/benchmarks/ benchmarks/
          cp /tmp/_bench_preserve/asv.conf.json asv.conf.json
      - uses: prefix-dev/setup-pixi@v0.8.10
        with:
          activate-environment: true
      - name: Build and benchmark
        run: |
          pixi run bash -c "meson setup bbdir \
            --prefix=$CONDA_PREFIX --libdir=lib \
            --buildtype release --wipe 2>/dev/null \
            || meson setup bbdir --prefix=$CONDA_PREFIX \
            --libdir=lib --buildtype release"
          pixi run meson install -C bbdir
          pixi run bash -c "pip install asv && asv machine --yes"
          pixi run asv run -E "existing:$(which python)" \
            --set-commit-hash ${{ matrix.sha }} \
            --record-samples --quick
      - uses: actions/upload-artifact@v4
        with:
          name: bench-${{ matrix.label }}
          path: .asv/results/

  combine:
    needs: benchmark
    runs-on: ubuntu-latest
    steps:
      - uses: astral-sh/setup-uv@v5
      - uses: actions/download-artifact@v4
        with:
          name: bench-main
          path: results/
      - uses: actions/download-artifact@v4
        with:
          name: bench-pr
          path: results/
      - name: Compare and write metadata
        run: |
          BASE_FILE=$(find results -name "${{ github.event.pull_request.base.sha | truncate 8 }}*.json" | head -1)
          PR_FILE=$(find results -name "${{ github.event.pull_request.head.sha | truncate 8 }}*.json" | head -1)
          uvx --from "git+https://github.com/airspeed-velocity/asv_spyglass.git" \
            asv-spyglass compare "$BASE_FILE" "$PR_FILE" \
            --label-before main --label-after pr > results/comparison.txt
          echo "main_sha=${{ github.event.pull_request.base.sha }}" > results/metadata.txt
          echo "pr_sha=${{ github.event.pull_request.head.sha }}" >> results/metadata.txt
      - uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: results/

The commenter workflow then downloads benchmark-results and posts the comment. See the Getting Started tutorial.

Pre-computed Output

comparison-text-file

Path to a pre-computed comparison output file. When provided, the action skips running asv-spyglass entirely and parses this file directly. Both results-path and SHA inputs become optional.

You can also place a file named comparison.txt inside results-path for the same effect (checked before running asv-spyglass in compare mode).

Comparison Options

asv-spyglass-ref

Git ref (branch or tag) for asv-spyglass installation. Default: main. The action installs asv-spyglass via uvx --from "git+https://github.com/airspeed-velocity/asv_spyglass.git@<ref>".

label-before / label-after

Labels for the before/after columns in the comparison. Default: main / pr. Passed to asv-spyglass compare --label-before / --label-after. Only used in compare mode.

baseline-label

Label for the baseline column in compare-many mode. Passed to asv-spyglass compare-many --label (baseline first).

contender-labels

Comma-separated labels for each contender, matching the order of contender-shas. Each label is passed as an additional --label argument to asv-spyglass compare-many.

asv-spyglass-args

Extra CLI flags passed through to asv-spyglass (e.g., --split --only-changed --factor 1.05). Space-separated. See Custom ASV Flags for details.

Regression Detection

regression-threshold

Ratio above which a benchmark counts as a critical regression. Default: 10. For example, if a benchmark’s ratio is 15.0 (15x slower), and the threshold is 10, it is flagged as critical.

auto-draft-on-regression

When set to 'true' and a critical regression is detected, the action converts the PR to draft status via the GitHub GraphQL API. Default: 'false'. This prevents accidental merges of severely regressed code.

Comment Formatting

comment-marker

HTML comment used to identify the benchmark comment for idempotent updates. Default: <!-- asv-benchmark-result -->. Change this only if you have multiple benchmark actions posting to the same PR.

runner-info

Text shown in the “Details” section indicating which runner produced the results. Default: ubuntu-latest.

dashboard-url

URL to an ASV dashboard. If provided, a “View full results” link is added to the details section.

Outputs

Output

Type

Description

comparison

string

Raw asv-spyglass comparison output

regression-detected

string

'true' or 'false'

comment-id

string

GitHub comment ID (empty if no PR found)

pr-number

string

PR number (empty if no PR found)

Use these in subsequent steps:

- name: Post benchmark comment
  id: bench
  uses: HaoZeke/asv-perch@v1
  with:
    # ...

- name: Fail on regression
  if: steps.bench.outputs.regression-detected == 'true'
  run: exit 1

Full Examples

Two-Way Comparison

- name: Post benchmark comment
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: results/
    metadata-file: results/metadata.txt
    regression-threshold: '10'
    auto-draft-on-regression: 'true'
    label-before: main
    label-after: pr
    runner-info: ubuntu-22.04
    dashboard-url: https://my-project.github.io/benchmarks/
    asv-spyglass-ref: main

Multi-Way Comparison

- name: Post benchmark comment
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: results/
    comparison-mode: compare-many
    baseline-sha: ${{ env.BASELINE_SHA }}
    contender-shas: '${{ env.OPT_SHA }}, ${{ env.DEBUG_SHA }}'
    baseline-label: default
    contender-labels: 'optimized, debug'
    runner-info: ubuntu-latest

Full Pipeline with pixi (run-prefix)

- uses: prefix-dev/setup-pixi@v0.8.0
- name: Benchmark and compare
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: .asv/results/
    comparison-mode: compare-many
    baseline: |
      label: main
      sha: ${{ env.BASE_SHA }}
      run-prefix: pixi run -e bench
    contenders: |
      - label: pr
        sha: ${{ env.PR_SHA }}
        run-prefix: pixi run -e bench
      - label: pr-debug
        sha: ${{ env.PR_SHA }}
        run-prefix: pixi run -e bench-debug
        description: Debug build with sanitizers
    runner-info: ubuntu-latest

Full Pipeline with virtualenv (setup)

- name: Benchmark and compare
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: .asv/results/
    comparison-mode: compare-many
    baseline: |
      label: main
      sha: ${{ env.BASE_SHA }}
      setup: source ./envs/bench.sh
    contenders: |
      - label: pr
        sha: ${{ env.PR_SHA }}
        setup: source ./envs/bench.sh
    runner-info: ubuntu-latest

Pre-computed Output

- name: Post benchmark comment
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    comparison-text-file: comparison.txt
    base-sha: ${{ env.BASE_SHA }}
    pr-sha: ${{ env.PR_SHA }}