This PR completes the new pattern matching and unification procedures
for the symbolic simulation framework using a two-phase approach.
**Phase 1 (Syntactic Matching):**
- Patterns use de Bruijn indices for expression variables and renamed
level params for universe variables
- Purely structural matching after reducible definitions are unfolded
- Universe levels treat `max`/`imax` as uninterpreted functions
- Proof arguments skipped via proof irrelevance
- Instance and binder constraints deferred to Phase 2
**Phase 2 (Pending Constraints):**
- Level constraints: structural equality with mvar assignment
- Instance constraints: `isDefEqI` (full `isDefEq` for TC synthesis)
- Expression constraints: `isDefEqS` with Miller pattern support
- Unassigned instance pattern variables synthesized via
`trySynthInstance`
**`isDefEqS` (Structural DefEq):**
- Miller pattern detection and assignment (`?m x y z := rhs` → `?m :=
fun x y z => rhs`)
- Scope checking via `maxFVar` to prevent out-of-scope assignments
- Optional zeta-delta reduction for let-declarations
- Proof irrelevance and instance delegation to `isDefEqI`
**Key optimizations:**
- `abstractFVars` skips metavariables and uses `maxFVar` for early
cutoff
- Per-pattern `ProofInstInfo` cache for fast argument classification
- Maximal sharing.
This PR implements `isDefEqS`, a lightweight structural definitional
equality for the symbolic simulation framework. Unlike the full
`isDefEq`, it avoids expensive operations while still supporting Miller
pattern unification.
**Key features:**
- Structural matching with optional zeta-delta reduction for
let-declarations
- Miller pattern detection and assignment (`?m x y z := rhs` → `?m :=
fun x y z => rhs`)
- Scope checking via `maxFVar` to prevent out-of-scope assignments
- Proof arguments skipped via proof irrelevance
- Instance arguments delegated to full `isDefEq` (need TC machinery)
- Universe levels treated structurally (`max`/`imax` as uninterpreted)
This PR adds optimized `abstractFVars` and `abstractFVarsRange` for
converting free variables to de Bruijn indices during pattern
matching/unification.
**Optimizations:**
- Metavariables are skipped (their contexts must not include abstracted
fvars)
- Subterms whose `maxFVar` is below the minimal abstracted fvar are
skipped via early cutoff
- Results are maximally shared via `AlphaShareBuilderM`
These optimizations are sound for Miller pattern matching where
metavariables are created before entering binders.
This PR implements `instantiateRevBetaS`, which is similar to
`instantiateRevS` but beta-reduces nested applications whose function
becomes a lambda after substitution.
For example, if `e` contains a subterm `#0 a` and we apply the
substitution `#0 := fun x => x + 1`, then `instantiateRevBetaS` produces
`a + 1` instead of `(fun x => x + 1) a`.
This is useful when applying theorems. For example, when applying
`Exists.intro`:
```lean
Exists.intro.{u} {α : Sort u} {p : α → Prop} (w : α) (h : p w) : Exists p
```
to a goal of the form `∃ x : Nat, p x ∧ q x`, we create metavariables
`?w` and `?h`. With `instantiateRevBetaS`, the type of `?h` becomes `p
?w ∧ q ?w` instead of `(fun x => p x ∧ q x) ?w`.
This PR introduces a fast pattern matching and unification module for
the symbolic simulation framework (`Sym`). The design prioritizes
performance by using a two-phase approach:
**Phase 1 (Syntactic Matching)**
- Patterns use de Bruijn indices for expression variables and renamed
level params (`_uvar.0`, `_uvar.1`, ...) for universe variables
- Matching is purely structural after reducible definitions are unfolded
during preprocessing
- Universe levels treat `max` and `imax` as uninterpreted functions (no
AC reasoning)
- Binders and term metavariables are deferred to Phase 2
**Phase 2 (Pending Constraints)** [WIP]
- Handles binders (Miller patterns) and metavariable unification
- Converts remaining de Bruijn variables to metavariables
- Falls back to `isDefEq` when necessary
**Key design decisions:**
- Preprocessing unfolds reducible definitions and performs beta/zeta
reduction
- Kernel projections are expected to be folded as projection
applications before matching
- Assignment conflicts are deferred to pending rather than invoking
`isDefEq` inline
- `instantiateRevS` ensures maximal sharing of result expressions
**TODO:**
- Skip instance arguments during matching, synthesize later
- Skip proof arguments (proof irrelevance)
- Implement `processPending` for Phase 2 constraints
This PR refactors the `Goal` type used in `grind`. The new
representation allows multiple goals with different metavariables to
share the same `GoalState`. This is useful for automation such as
symbolic simulator, where applying theorems create multiple goals that
inherit the same E-graph, congruence closure and solvers state, and
other accumulated facts.
This PR implements `intro` (and its variants) for `SymM`. These versions
do not use reduction or infer types, and ensure expressions are
maximally shared.
This PR adds the function `Sym.instantiateS` and its variants, which are
similar to `Expr.instantiate` but assumes the input is maximally shared
and ensures the output is also maximally shared.
This PR adds the function `Sym.replaceS`, which is similar to
`replace_fn` available in the kernel but assumes the input is maximally
shared and ensures the output is also maximally shared. The PR also
generalizes the `AlphaShareBuilder` API.
This PR simplifies `AlphaShareCommon.State` by separating the persistent
and transient parts of the state.
The `map` field caches visited sub-expressions during a single
`shareCommonAlpha` call to handle DAGs efficiently, the input expression
may contain shared sub-expressions that are not yet maximally shared.
However, this cache does not need to persist between different
`shareCommonAlpha` calls.
**Changes:**
- Moved `map` from the persistent `AlphaShareCommon.State` to a private
`State` used only within individual `shareCommonAlpha` calls.
- Replaced `PHashMap ExprPtr Expr` with (the more efficient)
`Std.HashMap ExprPtr Expr` for `map`, since it is now local to each call
and does not need persistence.
- The public `AlphaShareCommon.State` now only contains the `set` of
alpha-equivalent expressions that should persist
This PR adds functions for creating maximally shared terms from
maximally shared terms. It is more efficient than creating an expression
and then invoking `shareCommon`. We are going to use these functions for
implementing the symbolic simulation primitives.
This PR introduces `SymM`, a new monad for implementing symbolic
simulators (e.g., verification condition generators) in Lean. The monad
addresses performance issues found in symbolic simulators built on top
of user-facing tactics like `apply` and `intros`.
**Key features:**
- Goals are represented by `Grind.Goal` objects, enabling incremental
hypothesis processing
- No `revert` or `clear` operations, allowing O(1) local context checks
instead of O(n log n)
- Carries `GrindM` state across goals to avoid reprocessing shared
hypotheses
- Provides `mkGoal` for creating new goals within the monad
This is the foundational infrastructure for `SymM`. Future PRs will add
operations like `intro`, `apply`, and the optimized definitional
equality test.
This PR adds support for incrementally processing local declarations in
`grind`. Instead of processing all hypotheses at once during goal
initialization, `grind` now tracks which local declarations have been
processed via `Goal.nextDeclIdx` and provides APIs to process new
hypotheses incrementally.
This feature will be used by the new `SymM` monad for efficient symbolic
simulation.
This PR disables closed term extraction in the reflection terms used by
`bv_decide`. These terms do
not profit at all from closed term extraction but can in practice cause
thousands of new closed term
declarations which in turn slows down the compiler.
This PR improves the performance of and flattening in `bv_decide`.
The two main insights of this PR are:
1. When embedded constraint substitution is disabled it makes no sense
to have and flattening on in
the first place, given that we do not profit from it in any way.
2. The new fvars produced by and flattening can also be inserted into
the rewriting caches of the
preprocessing pipeline if the fvar they were derived from is already in
the cache. This
drastically decreases the amount of work we have to do in the second
rewriting pass after running
and flattening.
This PR adds the attributes `[grind norm]` and `[grind unfold]` for
controlling the `grind` normalizer/preprocessor.
The `norm` modifier instructs `grind` to use a theorem as a
normalization rule. That is, the theorem is applied during the
preprocessing step. This feature is meant for advanced users who
understand how the preprocessor and `grind`'s search procedure interact
with each other.
New users can still benefit from this feature by restricting its use to
theorems that completely eliminate a symbol from the goal. Example:
```lean
theorem max_def : max n m = if n ≤ m then m else n
```
For a negative example, consider:
```lean
opaque f : Int → Int → Int → Int
theorem fax1 : f x 0 1 = 1 := sorry
theorem fax2 : f 1 x 1 = 1 := sorry
attribute [grind norm] fax1
attribute [grind =] fax2
example (h : c = 1) : f c 0 c = 1 := by
grind -- fails
```
In this example, `fax1` is a normalization rule, but it is not
applicable to the input goal since `f c 0 c` is not an instance of `f x
0 1`. However, `f c 0 c` matches the pattern `f 1 x 1` modulo the
equality `c = 1`. Thus, `grind` instantiates `fax2` with `x := 0`,
producing the equality `f 1 0 1 = 1`, which the normalizer simplifies to
`True`. As a result, nothing useful is learned. In the future, we plan
to include linters to automatically detect issues like these. Example:
```lean
opaque f : Nat → Nat
opaque g : Nat → Nat
@[grind norm] axiom fax : f x = x + 2
@[grind norm ←] axiom fg : f x = g x
example : f x ≥ 2 := by grind
example : f x ≥ g x := by grind
example : f x + g x ≥ 4 := by grind
```
The `unfold` modifier instructs `grind` to unfold the given definition
during the preprocessing step. Example:
```lean
@[grind unfold] def h (x : Nat) := 2 * x
example : 6 ∣ 3*h x := by grind
```
This PR fixes a mismatch between the behavior of `foldlM` and
`foldlMUnsafe` in the three array
types. This mismatch is only exposed when manually specifying a `stop`
value greater than the size
of the array and only exploitable through `native_decide`.
The mismatch was introduced as part of
4ba21ea10c which introduced
`foldlMUnsafe` and thus likely a mistake when building the `unsafe`
implementation instead of a
specification mistake.
Closes#11773
This PR implements support for user-defined attributes at
`grind_pattern`. Suppose we have declared the `grind` attribute
```lean
register_grind_attr my_grind
```
Then, we can now write
```lean
opaque f : Nat → Nat
opaque g : Nat → Nat
axiom fg : g (f x) = x
grind_pattern [my_grind] fg => g (f x)
```
This PR uses the new support for user-defined `grind` attributes to
implement the default `[grind]` attribute.
A manual update-stage0 is required because it affects the .olean files.
This PR implements user-defined `grind` attributes. They are useful for
users that want to implement tactics using the `grind` infrastructure
(e.g., `progress*` in Aeneas). New `grind` attributes are declared using
the command
```lean
register_grind_attr my_grind
```
The command is similar to `register_simp_attr`. After the new attribute
is declared. Recall that similar to `register_simp_attr`, the new
attribute cannot be used in the same file it is declared.
```lean
opaque f : Nat → Nat
opaque g : Nat → Nat
@[my_grind] theorem fax : f (f x) = f x := sorry
example theorem fax2 : f (f (f x)) = f x := by
fail_if_success grind
grind [my_grind]
```
TODO: remove leftovers after update stage0
This PR allows `grind` to use `List.eq_nil_of_length_eq_zero` (and
`Array.eq_empty_of_size_eq_zero`), but only when it has already proved
the length is zero.
This PR moves the grind pattern from `Sublist.eq_of_length` to the
slightly more general `Sublist.eq_of_length_le`, and adds a grind
pattern guard so it only activates if we have a proof of the hypothesis.
This PR adds additional test coverage for #11758 (fix for #11745:
nonstandard instances in grind and simp +arith).
The existing test `grind_11745.lean` only covers Int LE with `grind
-order` and `lia -order`. This adds tests for:
- LT instances (Int and Nat)
- Nat LE instances
- Mixed canonical and non-canonical instances in the same goal
- Equality derived from two LE constraints
- `simp +arith` with non-canonical instances
🤖 Prepared with Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
This PR adds a Python script that helps find which commit introduced a
behavior change in Lean. It supports multiple bisection modes and
automatically downloads CI artifacts when available.
- [x] depends on: #11735
## Usage
```
usage: lean-bisect [-h] [--timeout SEC] [--ignore-messages] [--verbose]
[--selftest] [--clear-cache] [--nightly-only]
[file] [RANGE]
Bisect Lean toolchain versions to find where behavior changes.
positional arguments:
file Lean file to test (must only import Lean.* or Std.*)
RANGE Range to bisect: FROM..TO, FROM, or ..TO
options:
-h, --help show this help message and exit
--timeout SEC Timeout in seconds for each test run
--ignore-messages Compare only exit codes, ignore stdout/stderr differences
--verbose, -v Show stdout/stderr from each test
--selftest Run built-in selftest to verify lean-bisect works
--clear-cache Clear CI artifact cache (~600MB per commit) and exit
--nightly-only Stop after finding nightly range (don't bisect individual
commits)
Range Syntax:
FROM..TO Bisect between FROM and TO
FROM Start from FROM, bisect to latest nightly
..TO Bisect to TO, search backwards for regression start
If no range given, searches backwards from latest nightly to find regression.
Identifier Formats:
nightly-YYYY-MM-DD Nightly build date (e.g., nightly-2024-06-15)
Uses pre-built toolchains from leanprover/lean4-nightly.
Fast: downloads via elan (~30s each).
v4.X.Y or v4.X.Y-rcN Version tag (e.g., v4.8.0, v4.9.0-rc1)
Converts to equivalent nightly range.
Commit SHA Git commit hash (short or full, e.g., abc123def)
Bisects individual commits between two points.
Tries CI artifacts first (~30s), falls back to building (~2-5min).
Commits with failed CI builds are automatically skipped.
Artifacts cached in ~/.cache/lean-bisect/artifacts/
Bisection Modes:
Nightly mode: Both endpoints are nightly dates.
Binary search through nightlies to find the day behavior changed.
Then automatically continues to bisect individual commits.
Use --nightly-only to stop after finding the nightly range.
Version mode: Either endpoint is a version tag.
Converts to equivalent nightly range and bisects.
Commit mode: Both endpoints are commit SHAs.
Binary search through individual commits on master.
Output: "Behavior change introduced in commit abc123"
Examples:
# Simplest: just provide the file, finds the regression automatically
lean-bisect test.lean
# Specify an endpoint if you know roughly when it broke
lean-bisect test.lean ..nightly-2024-06-01
# Full manual control over the range
lean-bisect test.lean nightly-2024-01-01..nightly-2024-06-01
# Only find the nightly range, don't continue to commit bisection
lean-bisect test.lean nightly-2024-01-01..nightly-2024-06-01 --nightly-only
# Add a timeout (kills slow/hanging tests)
lean-bisect test.lean --timeout 30
# Bisect commits directly (if you already know the commit range)
lean-bisect test.lean abc1234..def5678
# Only compare exit codes, ignore output differences
lean-bisect test.lean --ignore-messages
# Clear downloaded CI artifacts to free disk space
lean-bisect --clear-cache
```
🤖 Prepared with Claude Code
---------
Co-authored-by: Claude <noreply@anthropic.com>
This PR fixes an issue where `grind` fails when trying to unfold a
definition by pattern matching imported by `import all` (or from a
non-`module`).
Fixes#11715
---------
Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
This PR turns even more commonly used bv_decide theorems that require
unification into fast simprocs
using syntactic equality. This pushes the overall performance across
sage/app7 to <= 1min10s for
every problem.
This PR replaces `ffi.md` with links to the corresponding sections of
the manual, so we don't have to keep two documents up to date.
A corresponding reference manual PR re-synchronizes them:
https://github.com/leanprover/reference-manual/pull/714
This PR upstreams dependency-management commands from Mathlib:
- `#import_path Foo` prints the transitive import chain that brings
`Foo` into scope
- `assert_not_exists Foo` errors if declaration `Foo` exists (for
dependency management)
- `assert_not_imported Module` warns if `Module` is transitively
imported
- `#check_assertions` verifies all pending assertions are eventually
satisfied
These commands help maintain the independence of different parts of a
library by catching unintended transitive dependencies early.
### Example usage
```lean
-- Find out how Nat got into scope
#import_path Nat
-- Declaration Nat is imported via
-- Init.Prelude,
-- which is imported by Init.Coe,
-- which is imported by Init.Notation,
-- ...
-- which is imported by this file.
-- Assert that a declaration should not be in scope yet
assert_not_exists SomeAdvancedType
-- Assert that a module should not be imported
assert_not_imported Some.Heavy.Module
-- Verify all assertions are eventually satisfied
#check_assertions
```
Addresses
https://lean-fro.zulipchat.com/#narrow/channel/398861-general/topic/path.20of.20an.20import🤖 Prepared with Claude Code
---------
Co-authored-by: Claude <noreply@anthropic.com>
This PR adds a standalone script to download pre-built CI artifacts from
GitHub Actions. This allows us to quickly switch commits without
rebuilding.
**Features:**
- Downloads artifacts for current HEAD or specified commit (`--sha`)
- Caches in `~/.cache/lean_build_artifact/` for reuse
- Platform detection (Linux/macOS, x86_64/aarch64)
**Usage:**
```
build_artifact.py # Download for current HEAD
build_artifact.py --sha abc1234 # Download for specific commit
build_artifact.py --clear-cache # Clear cache
```
This is extracted to be shared with `lean-bisect`.
🤖 Prepared with Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
