======================
Table Format Reference
======================

This page documents the asv-spyglass output formats that the action parses.

compare Format (Two-Way)
------------------------

The ``asv-spyglass compare`` command produces a table with these columns:

.. table::

    +-----------------------+------------------------------------------------------------------------------------------+
    | Column                | Content                                                                                  |
    +=======================+==========================================================================================+
    | Change                | Mark: ``+`` regressed, ``-`` improved, space unchanged, ``x`` incomparable, ``!`` failed |
    +-----------------------+------------------------------------------------------------------------------------------+
    | Before                | Value with uncertainty (e.g., ``167+/-3ns``)                                             |
    +-----------------------+------------------------------------------------------------------------------------------+
    | After                 | Value with uncertainty                                                                   |
    +-----------------------+------------------------------------------------------------------------------------------+
    | Ratio                 | Decimal ratio. ``~`` prefix means statistically insignificant. ``n/a`` if incomparable   |
    +-----------------------+------------------------------------------------------------------------------------------+
    | Benchmark (Parameter) | Full benchmark name with optional ``[env -> env]`` suffix                                |
    +-----------------------+------------------------------------------------------------------------------------------+

Example:

::

    | Change   | Before      | After       |   Ratio | Benchmark (Parameter)                                    |
    |----------|-------------|-------------|---------|----------------------------------------------------------|
    | +        | 167+/-3ns   | 187+/-3ns   |    1.12 | benchmarks.TimeSuite.time_values(10) [env1 -> env1]      |
    | -        | 157+/-3ns   | 137+/-3ns   |    0.87 | benchmarks.TimeSuite.time_keys(10) [env1 -> env1]        |
    |          | 1.17+/-0us  | 1.07+/-0us  |   ~0.91 | benchmarks.TimeSuite.time_keys(200) [env1 -> env1]       |
    | x        | 50+/-2ns    | n/a         |     n/a | benchmarks.TimeSuite.time_broken [env1 -> env1]          |

compare-many Format (Multi-Way)
-------------------------------

The ``asv-spyglass compare-many`` command produces a multi-column table:

.. table::

    +-----------------------+---------------------------------------------------+
    | Column                | Content                                           |
    +=======================+===================================================+
    | Benchmark (Parameter) | Full benchmark name                               |
    +-----------------------+---------------------------------------------------+
    | Baseline (label)      | Baseline value with uncertainty                   |
    +-----------------------+---------------------------------------------------+
    | Contender N (Ratio)   | Value followed by ``(mark ratio)`` in parentheses |
    +-----------------------+---------------------------------------------------+

Each contender cell has the format: ``value (mark ratio)``

- ``187+/-3ns (+ 1.12)`` -- regressed, ratio 1.12

- ``137+/-3ns (- 0.87)`` -- improved, ratio 0.87

- ``1.07+/-0us (~0.91)`` -- unchanged (statistically insignificant)

- ``n/a (x n/a)`` -- incomparable

Example:

::

    | Benchmark (Parameter)      | baseline (py311)  | opt-build (Ratio)    | debug-build (Ratio)   |
    |----------------------------|-------------------|----------------------|-----------------------|
    | benchmarks.TimeSuite.foo   | 167+/-3ns         | 187+/-3ns (+ 1.12)   | 150+/-2ns (- 0.90)    |
    | benchmarks.TimeSuite.bar   | 200+/-5ns         | 195+/-4ns (~0.98)    | 350+/-5ns (+ 1.75)    |

Change Marks
------------

.. table::

    +---------+--------------+----------------------------------------------------+
    | Mark    | Meaning      | Description                                        |
    +=========+==============+====================================================+
    | ``+``   | Regressed    | Performance got worse (statistically significant)  |
    +---------+--------------+----------------------------------------------------+
    | ``-``   | Improved     | Performance got better (statistically significant) |
    +---------+--------------+----------------------------------------------------+
    | (space) | Unchanged    | No statistically significant change                |
    +---------+--------------+----------------------------------------------------+
    | ``x``   | Incomparable | Results cannot be compared (e.g., one failed)      |
    +---------+--------------+----------------------------------------------------+
    | ``!``   | Failed       | Benchmark execution failed                         |
    +---------+--------------+----------------------------------------------------+

The ``~`` Prefix
----------------

A ratio prefixed with ``~`` means the change exceeds the configured factor
threshold but fails the Mann-Whitney U statistical significance test. This
indicates the difference is uncertain -- it might be real or might be noise.

Uncertainty Values
------------------

Values like ``167+/-3ns`` show the measurement with its interquartile range
divided by 2 (IQR/2). This provides a robust measure of spread that is less
sensitive to outliers than standard deviation.

--record-samples
----------------

ASV's ``--record-samples`` flag is required for statistical significance testing.
Without it, asv-spyglass falls back to simple ratio comparison without
confidence intervals or Mann-Whitney U tests.

--split Output
--------------

When ``--split`` is used, asv-spyglass outputs multiple tables separated by
section headers (e.g., "Benchmarks that have improved:", "Benchmarks that have
got worse:"). Each table has the same column format. The action's parser
handles this by skipping non-table lines.