Regression Gating

How to fail CI or convert PRs to draft when benchmark regressions are detected.

Using the regression-detected Output

The action sets regression-detected to true when any regressed benchmark’s ratio exceeds regression-threshold. Use this in a subsequent step:

- name: Post benchmark comment
  id: bench
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: results/
    metadata-file: results/metadata.txt
    regression-threshold: '5'

- name: Fail on regression
  if: steps.bench.outputs.regression-detected == 'true'
  run: |
    echo "::error::Critical benchmark regression detected"
    exit 1

Auto-Draft on Regression

Set auto-draft-on-regression: 'true' to automatically convert the PR to draft status when a critical regression is detected. This prevents accidental merges:

- name: Post benchmark comment
  uses: HaoZeke/asv-perch@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    results-path: results/
    metadata-file: results/metadata.txt
    regression-threshold: '10'
    auto-draft-on-regression: 'true'

The action uses the GitHub GraphQL API convertPullRequestToDraft mutation. The token needs pull-requests: write permission.

Choosing a Threshold

The regression-threshold is a ratio – if a benchmark’s “after” is 10x slower than “before”, a threshold of 10 catches it.

Threshold	Meaning	Use Case
2	2x slowdown	Strict, performance-critical code
5	5x slowdown	Moderate, catches major regressions
10	10x slowdown	Lenient, only catastrophic regressions

This threshold applies to the ratio column from asv-spyglass, which already accounts for statistical significance. A benchmark marked with ~ (statistically insignificant) can still trigger the threshold if the ratio is high enough, because statistical insignificance means the change is uncertain, not that it is small.

Gating in compare-many Mode

For compare-many mode, the action checks if any contender has any regressions. The regression-detected output is true if any contender column contains at least one regression.