8447586fea
The most important change is that all bench scripts now must always output to `measurements.jsonl` instead of being allowed to output results on stdout/err.
24 lines
1.3 KiB
Markdown
24 lines
1.3 KiB
Markdown
# Lean 4 benchmark suite
|
|
|
|
This directory contains the new Lean 4 benchmark suite.
|
|
It is built around [radar](github.com/leanprover/radar)
|
|
and benchmark results can be viewed
|
|
on the [Lean FRO radar instance](https://radar.lean-lang.org/repos/lean4).
|
|
|
|
Benchmarks are organized into subdirectories.
|
|
Each benchmark directory must contain a script called `run` that executes the benchmark,
|
|
as well as any additional benchmark-specific required files.
|
|
Ideally, each benchmark directory also contains a `README.md` explaining the benchmark.
|
|
|
|
To execute the entire suite, run `tests/bench-radar/run` in the repo root.
|
|
To execute an individua benchmark, run `tests/bench-radar/<benchmark>/run` in the repo root.
|
|
All scripts output their measurements into the file `measurements.jsonl`.
|
|
|
|
Radar sums any duplicated measurements with matching metrics.
|
|
To post-process the `measurements.jsonl` file this way in-place,
|
|
run `tests/bench-radar/combine.py` in the repo root after executing the benchmark suite.
|
|
|
|
All scripts related to the new benchmark suite are contained in this directory.
|
|
The files at `tests/bench` belong to the old suite.
|
|
The `*.py` symlinks are only for convenience when editing the python scripts in VSCode,
|
|
so the python extensions (in particular pyrefly) treat it as a python file.
|