max/lean4-htt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
lean4-htt/tests/bench-radar/README.md
Garmelon debafca7e1
chore: add radar-based bench suite for stdlib (#11264) 
This PR adds a new [radar]-based [temci]-less bench suite that replaces
the `stdlib` benchmarks from the old suite and also measures per-module
instruction counts. All other benchmarks from the old suite are
unaffected.

The readme at `tests/bench-radar/README.md` explains in more detail how
the bench suite is structured and how it works. The readmes in the
benchmark subdirectories explain what each benchmark does and which
metrics it collects.

All metrics except `stdlib//max dynamic symbols` were ported to the new
suite, though most have been renamed.

[radar]: https://github.com/leanprover/radar
[temci]: https://github.com/parttimenerd/temci
2025-11-25 12:59:30 +00:00

1.4 KiB
Raw Blame History

Lean 4 benchmark suite

This directory contains the new Lean 4 benchmark suite. It is built around radar and benchmark results can be viewed on the Lean FRO radar instance. To run the entire benchmark suite, execute the script called run.

Benchmarks are organized into subdirectories. Each benchmark directory must contain a script called run that executes the benchmark, as well as any additional benchmark-specific required files. Ideally, each benchmark directory also contains a README.md explaining the benchmark.

A benchmark's run script emits measurements as JSON objects as defined by radar's bench repo specification. It can emit the measurement in one of two ways:

  1. Append the measurement to the file radar.jsonl in the repository root. This file follows the JSON Lines format.
  2. Print a line on stdout or stderr containing radar::measurement= followed by the measurement. After the measurement, only whitespace is allowed.

All scripts related to the new benchmark suite are contained in this directory. The files at tests/bench belong to the old suite. The *.py symlinks are only for convenience when editing the python scripts in VSCode, so the python extensions (in particular pyrefly) treat it as a python file.