lean4-htt/tests/bench
Sebastian Graf b626c6d326
test: apply simp theorems in SymM mvcgen' (#12872)
This PR adds support for simp/equational spec theorems in the SymM-based
`mvcgen'` tactic,
catching up with a feature that the original `mvcgen` has supported for
a long time.
Users can write `@[spec] theorem : get (m := StateT σ m) = fun s => pure
(s, s) := rfl`
instead of manually specifying equivalent Hoare triples. The equational
form is more
concise and natural for specs that simply unfold definitions.

The universe level normalization (`normalizeLevelsExpr`) applied in
`work` and the backward
rule constructors is a workaround; ideally this should be integrated
into
`preprocessMVar`/`preprocessExpr` in the SymM framework so all users
benefit.

Changes:
- Add `SpecTheoremKind` to distinguish triple vs simp specs in
`SpecTheoremNew`
- Add `mkSpecTheoremNewFromSimpDecl?` to create spec entries from
equational lemmas, filtering no-op equations
- Add `mkBackwardRuleFromSimpSpec` to build backward rules via
`Eq.mpr`/`congrArg`, with instance synthesis, projection reduction, and
`unfoldReducible` on the RHS
- Migrate simp theorems from `SimpTheorems` database during
`migrateSpecTheoremsDatabase`
- Normalize universe levels so structural matching in
`BackwardRule.apply` succeeds when `max u v` vs `max v u` arise from
different code paths
- Simplify `mkSpecContext` by removing the mock `simp` context
construction
- Use `mkBackwardRuleFromExpr` instead of `mkAuxLemma` for triple specs,
since the proof may contain free variables from the goal context
- Add `AddSubCancelSimp` benchmark case and test exercising the simp
spec code path
- Change `AddSubCancel` spec proofs from `mvcgen` to `mvcgen'`
(dogfooding)


🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 17:15:04 +00:00
..
build chore: switch to new test/bench suite (#12590) 2026-02-25 13:51:53 +00:00
mergeSort feat: Array.mergeSort (#12385) 2026-03-06 13:18:13 +00:00
mvcgen test: apply simp theorems in SymM mvcgen' (#12872) 2026-03-10 17:15:04 +00:00
qsort chore: relative lean-toolchains (#12652) 2026-02-25 10:23:35 +00:00
size chore: switch to new test/bench suite (#12590) 2026-02-25 13:51:53 +00:00
sym feat: improves simpArrowTelescope simproc (#12153) 2026-01-25 22:29:38 +00:00
.gitignore chore: switch to new test/bench suite (#12590) 2026-02-25 13:51:53 +00:00
accumulate_profile.py chore: add lakeprof benchmarks (#9709) 2025-08-06 11:25:45 +00:00
arith_eval.ml
binarytrees.ghc-6.hs
binarytrees.ocaml-2.ml
binarytrees.st.hs test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.mlton-2.sml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.sml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.swift test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.swift
binarytrees5.ml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees5_multicore.ml chore: more benchmarking setup 2023-01-17 13:28:05 +01:00
compile.sh feat: LLVM backend (#1837) 2022-12-30 12:45:30 +01:00
const_fold.hs
const_fold.ml
const_fold.sml
const_fold.swift
cross.yaml chore: fix more typos in comments 2023-10-08 14:37:34 -07:00
dag_hassorry_issue.lean chore: deprecate levelZero and levelOne (#12720) 2026-03-04 01:03:08 +00:00
dag_hassorry_issue.lean.args chore: reduce stack space usage at instantiate_mvars_fn (#4931) 2024-08-06 17:38:59 +00:00
dag_hassorry_issue.lean.expected.out chore: re-enable tests (#10923) 2025-10-23 08:38:57 +00:00
delayed_assign.lean test: delayed assignment performance issue (#12201) 2026-01-28 02:08:39 +00:00
deriv.hs
deriv.ml
deriv.sml
deriv.swift
flake.lock chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
flake.nix chore: robustify Nix shell (#8141) 2025-04-28 15:08:32 +00:00
full-stdlib.exec.yaml
ghc-gc.py
lean-gc.py
Makefile chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
mlkit-gc.py
ocaml-gc.py chore: more benchmarking setup 2023-01-17 13:28:05 +01:00
perf.py
qsort.hs
qsort.ml test: more fair qsort.ml benchmark 2022-10-12 20:22:55 +02:00
qsort.sml
qsort.swift test: more fair qsort.ml benchmark 2022-10-12 20:22:55 +02:00
rbmap.hs
rbmap.ml
rbmap.sml
rbmap.swift
rbmap2.lean
rbmap3.lean
rbmap500k.lean
rbmap_checkpoint.hs
rbmap_checkpoint.ml
rbmap_checkpoint.sml
rbmap_checkpoint.swift
rbmap_checkpoint2.lean
rbmap_checkpoint2.sml
rbmap_checkpoint_cpp_lean3.cpp
rbmap_checkpoint_cpp_std.cpp
rbmap_cpp_lean3.cpp
rbmap_cpp_std.cpp
README.md chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
report.py chore: safer bench script 2023-07-19 08:31:39 +02:00
run.sh
speedcenter.yaml chore: try refining some benchmark settings (#8377) 2025-05-16 11:24:11 +00:00
states35.lean
test_single.sh feat: LLVM backend (#1837) 2022-12-30 12:45:30 +01:00
unionfind_clean.lean

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight "Speedcenter" suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight "Cross" suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper "Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming" (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.