=============
Configuration
=============

Inputs
------

All inputs are configured in the workflow YAML under ``with:``.

Required Inputs
~~~~~~~~~~~~~~~

``github-token``
    GitHub token for API access. Default: ``${{ github.token }}``.
    The token needs ``pull-requests: write`` and ``issues: write`` permissions.

Conditional Inputs
~~~~~~~~~~~~~~~~~~

``results-path``
    Path to the directory containing downloaded ASV results.
    Required unless ``comparison-text-file`` is provided.

SHA Resolution
~~~~~~~~~~~~~~

The action needs to know which commits to compare. There are several ways to
provide this depending on the comparison mode:

For compare Mode (Default)
^^^^^^^^^^^^^^^^^^^^^^^^^^

Option 1: Metadata File (Recommended)
:::::::::::::::::::::::::::::::::::::

``metadata-file``
    Path to a text file containing SHA mappings. Supports these keys:

::

    main_sha=a1b2c3d4e5f67890abcdef1234567890abcdef12
    pr_sha=e5f6g7h890abcdef1234567890abcdef12345678

Both ``main_sha=`` and ``base_sha=`` are accepted for the base commit. This is the
approach used by eOn and metatensor -- the benchmark workflow writes this file
and includes it in the artifact.

Option 2: Explicit SHAs
:::::::::::::::::::::::

``base-sha``
    Base commit SHA (the commit to compare against)

``pr-sha``
    PR head commit SHA

Explicit inputs override metadata file values if both are provided.

Option 3: Direct File Paths
:::::::::::::::::::::::::::

``base-file``
    Direct path to base result JSON file

``pr-file``
    Direct path to PR result JSON file

Use this when files are not named by SHA prefix, or when you know the exact
paths (e.g., from a matrix workflow).

For compare-many Mode
^^^^^^^^^^^^^^^^^^^^^

Option 1: Direct File Paths (Recommended)
:::::::::::::::::::::::::::::::::::::::::

``baseline-file``
    Direct path to the baseline result JSON file

``contender-files``
    Comma-separated direct paths to contender result JSON
    files. Order must match ``contender-labels``.

Use this for cross-environment comparisons where the same commit is benchmarked
in different environments (both result files share the same SHA prefix).

Option 2: SHA-Based Lookup
::::::::::::::::::::::::::

``baseline-sha``
    The baseline commit SHA. Used to find the baseline result
    file by SHA prefix.

``contender-shas``
    Comma-separated contender commit SHAs. Used to find
    contender result files by SHA prefix.

Use this when each contender ran a different commit.

Comparison Mode
~~~~~~~~~~~~~~~

``comparison-mode``
    Either ``compare`` (default, two-way) or ``compare-many``
    (multi-way baseline vs multiple contenders).

YAML-Based Pipeline Inputs
~~~~~~~~~~~~~~~~~~~~~~~~~~

These inputs let the action run benchmarks before comparing. Each entry can
include environment activation via two fields:

- ``run-prefix`` -- A wrapper command prepended with a space. Use this for
  tools like pixi, conda, or nix that wrap the command.

- ``setup`` -- A shell command prepended with ``&&``. Use this for sourcing
  scripts, exporting variables, or activating virtualenvs.

Both can be combined. The ``{sha}`` placeholder is replaced in all three
fields. The constructed shell command is:

::

    bash -c '<setup with {sha}> && <run-prefix with {sha}> <benchmark-command with {sha}>'

Each part is optional -- omit what you don't need. If an entry has no ``sha``,
benchmarking is skipped for that entry (useful when results already exist).

Contender benchmarks run in parallel via ``Promise.all``. If your builds
modify shared state (same build directory), use ``setup`` to isolate them.

Init command
^^^^^^^^^^^^

Use ``init-command`` to run a one-time setup before any benchmarks:

.. code:: yaml

    init-command: pixi run bash -c "pip install asv && asv machine --yes"

This runs once, before any baseline or contender benchmarks.

Build pattern (C++, Fortran, Rust)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use ``setup`` with ``{sha}`` to checkout and build per-commit:

.. code:: yaml

    baseline: |
      label: main
      sha: abc123def
      setup: >-
        git checkout -f {sha} && git clean -fd &&
        meson setup bbdir --prefix=$CONDA_PREFIX --buildtype release --wipe 2>/dev/null ||
        meson setup bbdir --prefix=$CONDA_PREFIX --buildtype release &&
        meson install -C bbdir
      run-prefix: pixi run

The ``{sha}`` placeholder is replaced in ``setup``, ``run-prefix``, and
``benchmark-command``. This enables one action step to handle checkout,
build, and benchmark for each entry.

Wrapper pattern (pixi, conda, nix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use ``run-prefix``. The tool wraps the entire benchmark command:

.. code:: yaml

    baseline: |
      label: main
      sha: abc123def
      run-prefix: pixi run -e bench

This runs: ``pixi run -e bench asv run --record-samples abc123def^!``

Source pattern (virtualenv, shell scripts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use ``setup``. The script runs first, then the benchmark command:

.. code:: yaml

    baseline: |
      label: main
      sha: abc123def
      setup: source ./envs/bench.sh

This runs: ``source ./envs/bench.sh && asv run --record-samples abc123def^!``

Combined (e.g. env vars + wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: yaml

    contenders: |
      - label: pr-debug
        sha: def456abc
        setup: export CFLAGS="-O0 -g"
        run-prefix: pixi run -e bench-debug

This runs: ``export CFLAGS``"-O0 -g" && pixi run -e bench-debug asv run --record-samples def456abc^!=

Field Reference
^^^^^^^^^^^^^^^

``baseline``
    YAML string defining the baseline entry. Fields:

    - ``label`` (string) -- Display label for the baseline column.

    - ``sha`` (string) -- Commit SHA for benchmarking and result lookup.

    - ``file`` (string) -- Direct path to a result JSON file (bypasses SHA lookup).

    - ``run-prefix`` (string) -- Wrapper command prepended to benchmark command
      (e.g. ``pixi run -e bench``, ``conda run -n bench``, ``nix develop -c``).

    - ``setup`` (string) -- Shell command sourced before the benchmark command
      (e.g. ``source ./envs/bench.sh``, ``export CC=gcc-12``).

    - ``env`` (string) -- Environment name shown in PR comment details.

    - ``description`` (string) -- Human-readable description shown in PR comment.

    When ``baseline`` is provided, it overrides ``baseline-sha``, ``baseline-file``,
    and ``baseline-label``.

``contenders``
    YAML list of contender objects. Each object accepts the
    same fields as ``baseline`` (label, sha, file, run-prefix, setup, env,
    description). Works in both ``compare`` mode (first contender used as the
    "after" side) and ``compare-many`` mode (all contenders compared against
    baseline). When provided, this overrides ``contender-shas``,
    ``contender-files``, and ``contender-labels``.

``init-command``
    Shell command run once before any benchmarks. Use for
    one-time setup like ``asv machine --yes`` or stashing benchmark suites.

``preserve-paths``
    Comma-separated paths to preserve across git checkouts.
    When an entry has a ``sha`` field, the action runs
    ``git checkout -f {sha} && git clean -fd`` automatically. These paths are
    stashed to ``/tmp`` before checkout and restored after. Example:

.. code:: yaml

    preserve-paths: benchmarks/, asv.conf.json

This eliminates the need for manual ``cp`` / ``stash`` boilerplate in ``setup``.
Without ``preserve-paths``, the action does not run ``git checkout`` at all --
the ``setup`` field must handle it.

``benchmark-command``
    Shell command template for running benchmarks.
    Default: ``asv run --record-samples {sha}^!``. The placeholder ``{sha}`` is
    replaced in this field, ``setup``, and ``run-prefix``. Each entry's
    ``run-prefix`` is prepended and ``setup`` is sourced before this command.
    Entries without a ``sha`` field skip benchmarking.

    Example with a custom command:

.. code:: yaml

    benchmark-command: 'asv run --quick {sha}^!'

Full Pipeline with C++ Build (Single Job)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use ``preserve-paths`` to automatically stash/restore files across git
checkouts. The ``setup`` field handles only the build -- no checkout or copy
boilerplate needed.

.. code:: yaml

    - uses: prefix-dev/setup-pixi@v0.8.10
      with:
        activate-environment: true
    - uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        results-path: .asv/results/
        init-command: pixi run bash -c "pip install asv && asv machine --yes"
        preserve-paths: benchmarks/, asv.conf.json
        benchmark-command: >-
          asv run -E "existing:$(which python)"
          --set-commit-hash {sha} --record-samples --quick
        baseline: |
          label: main
          sha: ${{ github.event.pull_request.base.sha }}
          setup: >-
            pixi run bash -c "meson setup bbdir
            --prefix=$CONDA_PREFIX --libdir=lib
            --buildtype release --wipe 2>/dev/null
            || meson setup bbdir --prefix=$CONDA_PREFIX
            --libdir=lib --buildtype release" &&
            pixi run meson install -C bbdir
          run-prefix: pixi run
        contenders: |
          - label: pr
            sha: ${{ github.event.pull_request.head.sha }}
            setup: >-
              pixi run bash -c "meson setup bbdir
              --prefix=$CONDA_PREFIX --libdir=lib
              --buildtype release --wipe 2>/dev/null
              || meson setup bbdir --prefix=$CONDA_PREFIX
              --libdir=lib --buildtype release" &&
              pixi run meson install -C bbdir
            run-prefix: pixi run
        label-before: main
        label-after: pr
        runner-info: ubuntu-22.04

Full Pipeline with C++ Build (Matrix -- Parallel)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For parallel execution, use a matrix strategy with a combine job. Each SHA
gets its own runner. The commenter downloads combined results.

.. code:: yaml

    jobs:
      benchmark:
        strategy:
          fail-fast: false
          matrix:
            include:
              - label: main
                sha: ${{ github.event.pull_request.base.sha }}
              - label: pr
                sha: ${{ github.event.pull_request.head.sha }}
        runs-on: ubuntu-22.04
        steps:
          - uses: actions/checkout@v4
            with:
              fetch-depth: 0
          - name: Checkout target
            run: |
              mkdir -p /tmp/_bench_preserve
              cp -r benchmarks/ /tmp/_bench_preserve/
              cp asv.conf.json /tmp/_bench_preserve/
              git checkout -f ${{ matrix.sha }}
              git clean -fd
              cp -r /tmp/_bench_preserve/benchmarks/ benchmarks/
              cp /tmp/_bench_preserve/asv.conf.json asv.conf.json
          - uses: prefix-dev/setup-pixi@v0.8.10
            with:
              activate-environment: true
          - name: Build and benchmark
            run: |
              pixi run bash -c "meson setup bbdir \
                --prefix=$CONDA_PREFIX --libdir=lib \
                --buildtype release --wipe 2>/dev/null \
                || meson setup bbdir --prefix=$CONDA_PREFIX \
                --libdir=lib --buildtype release"
              pixi run meson install -C bbdir
              pixi run bash -c "pip install asv && asv machine --yes"
              pixi run asv run -E "existing:$(which python)" \
                --set-commit-hash ${{ matrix.sha }} \
                --record-samples --quick
          - uses: actions/upload-artifact@v4
            with:
              name: bench-${{ matrix.label }}
              path: .asv/results/

      combine:
        needs: benchmark
        runs-on: ubuntu-latest
        steps:
          - uses: astral-sh/setup-uv@v5
          - uses: actions/download-artifact@v4
            with:
              name: bench-main
              path: results/
          - uses: actions/download-artifact@v4
            with:
              name: bench-pr
              path: results/
          - name: Compare and write metadata
            run: |
              BASE_FILE=$(find results -name "${{ github.event.pull_request.base.sha | truncate 8 }}*.json" | head -1)
              PR_FILE=$(find results -name "${{ github.event.pull_request.head.sha | truncate 8 }}*.json" | head -1)
              uvx --from "git+https://github.com/airspeed-velocity/asv_spyglass.git" \
                asv-spyglass compare "$BASE_FILE" "$PR_FILE" \
                --label-before main --label-after pr > results/comparison.txt
              echo "main_sha=${{ github.event.pull_request.base.sha }}" > results/metadata.txt
              echo "pr_sha=${{ github.event.pull_request.head.sha }}" >> results/metadata.txt
          - uses: actions/upload-artifact@v4
            with:
              name: benchmark-results
              path: results/

The commenter workflow then downloads ``benchmark-results`` and posts the
comment. See the `Getting Started <../../tutorials/asv_getting_started.rst>`_ tutorial.

Pre-computed Output
~~~~~~~~~~~~~~~~~~~

``comparison-text-file``
    Path to a pre-computed comparison output file. When
    provided, the action skips running asv-spyglass entirely and parses this file
    directly. Both ``results-path`` and SHA inputs become optional.

You can also place a file named ``comparison.txt`` inside ``results-path`` for the
same effect (checked before running asv-spyglass in ``compare`` mode).

Comparison Options
~~~~~~~~~~~~~~~~~~

``asv-spyglass-ref``
    Git ref (branch or tag) for ``asv-spyglass`` installation.
    Default: ``main``. The action installs asv-spyglass via
    ``uvx --from "git+https://github.com/airspeed-velocity/asv_spyglass.git@<ref>"``.

``label-before`` / ``label-after``
    Labels for the before/after columns in the
    comparison. Default: ``main`` / ``pr``. Passed to ``asv-spyglass compare --label-before / --label-after``. Only used in ``compare`` mode.

``baseline-label``
    Label for the baseline column in ``compare-many`` mode.
    Passed to ``asv-spyglass compare-many --label`` (baseline first).

``contender-labels``
    Comma-separated labels for each contender, matching
    the order of ``contender-shas``. Each label is passed as an additional
    ``--label`` argument to ``asv-spyglass compare-many``.

``asv-spyglass-args``
    Extra CLI flags passed through to asv-spyglass
    (e.g., ``--split --only-changed --factor 1.05``). Space-separated.
    See `Custom ASV Flags <../../howto/custom_asv_flags.rst>`_ for details.

Regression Detection
~~~~~~~~~~~~~~~~~~~~

``regression-threshold``
    Ratio above which a benchmark counts as a critical
    regression. Default: ``10``. For example, if a benchmark's ratio is 15.0
    (15x slower), and the threshold is 10, it is flagged as critical.

``auto-draft-on-regression``
    When set to ``'true'`` and a critical regression
    is detected, the action converts the PR to draft status via the GitHub GraphQL
    API. Default: ``'false'``. This prevents accidental merges of severely regressed
    code.

Comment Formatting
~~~~~~~~~~~~~~~~~~

``comment-marker``
    HTML comment used to identify the benchmark comment for
    idempotent updates. Default: ``<!-- asv-benchmark-result -->``. Change this only
    if you have multiple benchmark actions posting to the same PR.

``runner-info``
    Text shown in the "Details" section indicating which runner
    produced the results. Default: ``ubuntu-latest``.

``dashboard-url``
    URL to an ASV dashboard. If provided, a "View full results"
    link is added to the details section.

Outputs
-------

.. table::

    +-------------------------+--------+------------------------------------------+
    | Output                  | Type   | Description                              |
    +=========================+========+==========================================+
    | ``comparison``          | string | Raw asv-spyglass comparison output       |
    +-------------------------+--------+------------------------------------------+
    | ``regression-detected`` | string | ``'true'`` or ``'false'``                |
    +-------------------------+--------+------------------------------------------+
    | ``comment-id``          | string | GitHub comment ID (empty if no PR found) |
    +-------------------------+--------+------------------------------------------+
    | ``pr-number``           | string | PR number (empty if no PR found)         |
    +-------------------------+--------+------------------------------------------+

Use these in subsequent steps:

.. code:: yaml

    - name: Post benchmark comment
      id: bench
      uses: HaoZeke/asv-perch@v1
      with:
        # ...

    - name: Fail on regression
      if: steps.bench.outputs.regression-detected == 'true'
      run: exit 1

Full Examples
-------------

Two-Way Comparison
~~~~~~~~~~~~~~~~~~

.. code:: yaml

    - name: Post benchmark comment
      uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        results-path: results/
        metadata-file: results/metadata.txt
        regression-threshold: '10'
        auto-draft-on-regression: 'true'
        label-before: main
        label-after: pr
        runner-info: ubuntu-22.04
        dashboard-url: https://my-project.github.io/benchmarks/
        asv-spyglass-ref: main

Multi-Way Comparison
~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    - name: Post benchmark comment
      uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        results-path: results/
        comparison-mode: compare-many
        baseline-sha: ${{ env.BASELINE_SHA }}
        contender-shas: '${{ env.OPT_SHA }}, ${{ env.DEBUG_SHA }}'
        baseline-label: default
        contender-labels: 'optimized, debug'
        runner-info: ubuntu-latest

Full Pipeline with pixi (run-prefix)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    - uses: prefix-dev/setup-pixi@v0.8.0
    - name: Benchmark and compare
      uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        results-path: .asv/results/
        comparison-mode: compare-many
        baseline: |
          label: main
          sha: ${{ env.BASE_SHA }}
          run-prefix: pixi run -e bench
        contenders: |
          - label: pr
            sha: ${{ env.PR_SHA }}
            run-prefix: pixi run -e bench
          - label: pr-debug
            sha: ${{ env.PR_SHA }}
            run-prefix: pixi run -e bench-debug
            description: Debug build with sanitizers
        runner-info: ubuntu-latest

Full Pipeline with virtualenv (setup)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    - name: Benchmark and compare
      uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        results-path: .asv/results/
        comparison-mode: compare-many
        baseline: |
          label: main
          sha: ${{ env.BASE_SHA }}
          setup: source ./envs/bench.sh
        contenders: |
          - label: pr
            sha: ${{ env.PR_SHA }}
            setup: source ./envs/bench.sh
        runner-info: ubuntu-latest

Pre-computed Output
~~~~~~~~~~~~~~~~~~~

.. code:: yaml

    - name: Post benchmark comment
      uses: HaoZeke/asv-perch@v1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        comparison-text-file: comparison.txt
        base-sha: ${{ env.BASE_SHA }}
        pr-sha: ${{ env.PR_SHA }}