# `Excessibility.Benchmark`
[🔗](https://github.com/lessthanseventy/excessibility/blob/v0.20.0/lib/excessibility/benchmark.ex#L1)

Pure, robust statistics for benchmark mode (`mix excessibility.debug --benchmark=N`).

Benchmark mode loops a test N times and collects, per run, a map of
measurement key to `duration_ms`. `summarize/2` turns those per-run samples
into a value-free `benchmark.json` artifact.

## Timing contract

Timing is diagnostic, never comparable across environments and never part of
the digest. A key being free of outliers means "no *relative* outlier in these
samples," **not** "fast." Robust stats (median + MAD) are used deliberately
instead of mean/stddev so a single slow run cannot skew the picture.

Sample 1 is treated as **cold** (first-run costs: compilation, connection
warmup, cold caches). Samples 2..N are **warm**. Cold and warm are reported
separately per the timing contract; mixing them would poison the median.

## Input shape

`samples` is a list (one entry per run, in run order) of maps:

    [
      %{"MyLive/mount" => 12.0, "query:sha256:aaa" => 5.0},
      %{"MyLive/mount" => 6.0,  "query:sha256:aaa" => 3.0},
      ...
    ]

Keys are opaque strings (the mix task uses `"<view>/<callback>"` and
`"query:<fingerprint>"`); values are `duration_ms` numbers. Keys may vary
between runs — each key is summarized over the runs in which it appears.

## Output shape

    %{
      schema: "excessibility.benchmark/v1",
      runs: n,
      cold: %{key => %{median: number, mad: number, sample: number}},  # run 1 only
      warm: %{key => %{median: number, mad: number, samples: integer}}, # runs 2..n
      outliers: [
        %{key: String.t(), run: pos_integer(), value: number,
          median: number, mad: number, threshold: number}
      ],
      notes: [String.t()]
    }

`outliers` is **advisory only** — a warm sample must clear **three** gates to
be flagged, so scheduler/timer jitter in the sub-millisecond band is not
reported as actionable evidence:

  1. the robust statistical threshold `value > median + k * mad`
     (`k` defaults to 6, `:k`);
  2. a minimum **absolute** effect `value - median >= min_abs_ms`
     (defaults to 1.0 ms, `:min_abs_ms`); and
  3. a minimum **relative** effect `value >= median * min_rel_factor`
     (defaults to 1.5, `:min_rel_factor`).

Set `min_abs_ms: 0.0, min_rel_factor: 1.0` to restore pure-statistical
flagging. Each outlier carries `weak_evidence: true` when its key has fewer
than 5 warm samples, and a run-level `notes` entry labels the whole
artifact as weak evidence when there are too few warm runs for stable MAD
inference. It never encodes a pass/fail verdict; the raw `value`, `median`,
`mad`, `threshold`, and 1-based `run` index are attached so a reader can judge
for themselves.

# `mad`

Median Absolute Deviation: `median(for x <- xs, do: abs(x - median(xs)))`.

A robust measure of spread. Returns `nil` for an empty list and `0` when all
values are identical.

# `median`

Median of a list of numbers.

Returns the middle element (odd length) or the mean of the two middle
elements (even length). Returns `nil` for an empty list.

# `summarize`

Summarize per-run samples into the robust cold/warm benchmark artifact.

Options:

  * `:k` - outlier multiplier; a warm sample is flagged when it exceeds
    `median + k * mad`. Defaults to `6`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*