lean4-htt/tests/bench
David Thrane Christiansen eb58f46ce7
feat: language reference links and examples in docstrings (#7240)
This PR adds a canonical syntax for linking to sections in the language
reference along with formatting of examples in docstrings according to
the docstring style guide.


Docstrings are now pre-processed as follows:

* Output included as part of examples is shown with leading line comment
indicators in hovers

* URLs of the form `lean-manual://section/section-id` are rewritten to
links that point at the corresponding section in the Lean reference
manual. The reference manual's base URL is configured when Lean is built
and can be overridden with the `LEAN_MANUAL_ROOT` environment variable.
This way, releases can point documentation links to the correct
snapshot, and users can use their own, e.g. for offline reading.

Manual URLs in docstrings are validated when the docstring is added. The
presence of a URL starting with `lean-manual://` that is not a
syntactically valid section link causes the docstring to be rejected.
This allows for future extensibility to the set of allowed links. There
is no validation that the linked-to section actually exists. To provide
the best possible error messages in case of validation failures,
`Lean.addDocString` now takes a `TSyntax ``docComment` instead of a
string; clients should adapt by removing the step that extracts the
string, or by calling the lower-level `addDocStringCore` in cases where
the docstring in question is obtained from the environment and has thus
already had its links validated.

A stage0 update is required to make the documentation site configurable
at build time and for releases. A local commit on top of a stage0 update
that will be sent in a followup PR includes the configurable reference
manual root and updates to the release checklist.

---------

Co-authored-by: Marc Huisinga <mhuisi@protonmail.com>
2025-03-12 09:17:27 +00:00
..
inundation chore: remove the coercion from String to Name (#3589) 2024-03-21 23:46:03 +00:00
mergeSort chore: deprecate List.iota (#6708) 2025-01-21 02:32:35 +00:00
qsort feat: remove runtime bounds checks and partial from qsort (#6241) 2024-12-01 06:26:00 +00:00
.gitignore test: lake: add build Init/Lean/Lake benchmark 2023-09-22 20:05:20 +02:00
accumulate_profile.py chore: include full build in stdlib benchmark (#3104) 2023-12-23 16:27:07 +00:00
arith_eval.ml
big_do.lean test: add a benchmark that is slow to elaborate (#5656) 2024-10-23 08:20:15 +00:00
big_omega.lean test: big_omega benchmark (#5817) 2024-10-24 07:26:29 +00:00
binarytrees.ghc-6.hs
binarytrees.lean test: clean up binarytrees.lean 2023-01-19 14:44:20 +01:00
binarytrees.lean.args
binarytrees.lean.expected.out
binarytrees.ocaml-2.ml
binarytrees.st.hs test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.lean test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.mlton-2.sml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.sml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.st.swift test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees.swift
binarytrees5.ml test: add binarytrees.st benchmark 2023-01-19 14:44:20 +01:00
binarytrees5_multicore.ml chore: more benchmarking setup 2023-01-17 13:28:05 +01:00
bv_decide_inequality.lean fix: bv_decide benchmarks (#6017) 2024-11-09 11:18:33 +00:00
bv_decide_mod.lean fix: bv_decide benchmarks (#6017) 2024-11-09 11:18:33 +00:00
bv_decide_mul.lean feat: add bv_decide benchmarks (#5203) 2024-08-29 12:45:58 +00:00
bv_decide_realworld.lean chore: notation ^^ for Bool.xor (#5332) 2024-09-18 08:59:11 +00:00
compile.sh feat: LLVM backend (#1837) 2022-12-30 12:45:30 +01:00
const_fold.hs
const_fold.lean
const_fold.lean.args
const_fold.lean.expected.out
const_fold.ml
const_fold.sml
const_fold.swift
cross.yaml chore: fix more typos in comments 2023-10-08 14:37:34 -07:00
dag_hassorry_issue.lean chore: split up & simplify importModules 2023-08-31 15:37:33 -04:00
dag_hassorry_issue.lean.args chore: reduce stack space usage at instantiate_mvars_fn (#4931) 2024-08-06 17:38:59 +00:00
dag_hassorry_issue.lean.expected.out chore: improve test 2023-07-11 19:19:42 -07:00
deriv.hs
deriv.lean
deriv.lean.args
deriv.lean.expected.out
deriv.ml
deriv.sml
deriv.swift
ex-50-50-1.leq
flake.lock chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
flake.nix chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
full-stdlib.exec.yaml
ghc-gc.py
identifier_completion.lean feat: language reference links and examples in docstrings (#7240) 2025-03-12 09:17:27 +00:00
identifier_completion_didOpen.log feat: language reference links and examples in docstrings (#7240) 2025-03-12 09:17:27 +00:00
identifier_completion_initialization.log test: identifier completion benchmark (#6796) 2025-01-27 19:31:32 +00:00
identifier_completion_runner.lean test: identifier completion benchmark (#6796) 2025-01-27 19:31:32 +00:00
ilean_roundtrip.lean feat: language reference links and examples in docstrings (#7240) 2025-03-12 09:17:27 +00:00
lean-gc.py
liasolver.lean chore: remove deprecated aliases for Int.tdiv and Int.tmod (#6322) 2024-12-08 05:19:42 +00:00
liasolver.lean.args
liasolver.lean.expected.out
Makefile chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
mlkit-gc.py
nat_repr.lean chore: Nat.repr microbenchmark (#3888) 2024-04-17 18:10:32 +00:00
nat_repr.lean.args chore: Nat.repr microbenchmark (#3888) 2024-04-17 18:10:32 +00:00
nat_repr.lean.expected.out chore: Nat.repr microbenchmark (#3888) 2024-04-17 18:10:32 +00:00
ocaml-gc.py chore: more benchmarking setup 2023-01-17 13:28:05 +01:00
omega_stress.lean perf: optimize sorry detection in unused variables linter (#7129) 2025-02-22 16:43:39 +00:00
parser.lean test: update parser benchmark, add to speedcenter suite 2023-08-08 18:40:19 +02:00
perf.py chore: update benchmark suite 2022-05-25 18:26:36 +02:00
qsort.hs chore: update benchmark suite 2022-05-25 18:26:36 +02:00
qsort.lean feat: Array.swap takes Nat arguments, with tactic provided proofs (#6194) 2024-11-24 07:59:57 +00:00
qsort.lean.args
qsort.lean.expected.out
qsort.ml test: more fair qsort.ml benchmark 2022-10-12 20:22:55 +02:00
qsort.sml
qsort.swift test: more fair qsort.ml benchmark 2022-10-12 20:22:55 +02:00
rbmap.hs chore: make rbmap.hs more similar to other implementations 2022-09-24 14:16:48 +02:00
rbmap.lean chore: modernize rbmap benchmarks a bit 2022-09-24 14:16:48 +02:00
rbmap.lean.args
rbmap.lean.expected.out
rbmap.ml
rbmap.sml
rbmap.swift
rbmap2.lean
rbmap3.lean
rbmap500k.lean
rbmap_checkpoint.hs chore: make rbmap.hs more similar to other implementations 2022-09-24 14:16:48 +02:00
rbmap_checkpoint.lean chore: modernize rbmap benchmarks a bit 2022-09-24 14:16:48 +02:00
rbmap_checkpoint.lean.args
rbmap_checkpoint.lean.expected.out
rbmap_checkpoint.ml
rbmap_checkpoint.sml
rbmap_checkpoint.swift
rbmap_checkpoint2.lean
rbmap_checkpoint2.sml
rbmap_checkpoint_cpp_lean3.cpp
rbmap_checkpoint_cpp_std.cpp
rbmap_cpp_lean3.cpp
rbmap_cpp_std.cpp
rbmap_fbip.lean feat: add rbmap_fbip benchmark 2022-10-06 17:26:43 -07:00
rbmap_library.lean chore: more RBMap cleanup 2022-10-06 17:26:43 -07:00
README.md chore: update cross-bench setup 2024-04-15 10:59:07 +02:00
reduceMatch.lean chore: upstream omega (#3367) 2024-02-19 00:19:55 +00:00
report.py chore: safer bench script 2023-07-19 08:31:39 +02:00
run.sh
server_startup.lean fix: make watchdog more resilient against badly behaving clients (#4443) 2024-06-13 13:48:36 +00:00
server_startup.log test: add language server startup benchmark (#3558) 2024-03-04 09:01:51 +00:00
simp_arith1.lean test: fix simp_arith1 benchmark (#7049) 2025-02-12 10:22:32 +00:00
speedcenter.exec.velcom.yaml perf: optimize sorry detection in unused variables linter (#7129) 2025-02-22 16:43:39 +00:00
speedcenter.yaml chore: speedcenter: reduce number of runs for "fast" benchmarks from 10 to 3 (#5009) 2024-08-13 09:06:06 +00:00
states35.lean
test_single.sh feat: LLVM backend (#1837) 2022-12-30 12:45:30 +01:00
unionfind.lean feat: change Array.get to take a Nat and a proof (#6032) 2024-11-12 03:30:46 +00:00
unionfind.lean.args
unionfind.lean.expected.out
unionfind_clean.lean
workspaceSymbols.lean chore: fix benchmark 2024-06-20 18:18:41 +02:00

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight "Speedcenter" suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight "Cross" suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper "Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming" (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.