This PR introduces ranges that are polymorphic, in contrast to the
existing `Std.Range` which only supports natural numbers.
Breakdown of core changes:
* `Lean.Parser.Basic`: Modified the number parser (`Lean.Parser.Basic`)
so that it will only consider a *single* dot to be part of a decimal
number. `1..` will no longer be parsed as `1.` followed by `.`, but as
`1` followed by `..`.
* The test `ellipsisProjIssue` ensures that `#check Nat.add ...succ`
produces a syntax error. After introducing the new range notation (see
below), it returns a different (less nice) error message. I updated the
test to reflect the new error message. (The error message will become
nicer as soon as a delaborator for the ranges is implemented. This is
out of scope for this PR.)
Breakdown of standard library changes:
Modified modules: `Init.Data.Range.Polymorphic` (added),
`Init.Data.Iterators`, `Std.Data.Iterators`
* Introduced the type `Std.PRange` that is parameterized over the type
in which the range operates and the shapes of the lower and upper bound.
* Introduced a new notation for ranges. Examples for this notation are:
`1...*`, `1...=3`, `1...<3`, `1<...=2`, `*...=3`.
* Defined lots of typeclasses for different capabilities of ranges,
depending on their shape and underlying type.
* Introduced `Iter(M).size`.
* Introduced the `Iter(M).stepSize n` combinator, which iterates over an
iterator with the given step size `n`. It will drop `n - 1` values
between every value it emits.
* Replaced `LawfulPureIterator` with a new and better typeclass
`LawfulDeterministicIterator`.
* Simplified some lemma statements in the iterator library such as
`IterM.toList_eq_match`, which unnecessarily matched over a `Subtype`,
hindering rewrites due to type dependencies.
Reasons for the concrete choice of notation:
* `lean4-cli` uses `...`-based notation for the `Cmd` notation and it
clashes with `...a` range notation.
* test `2461` fails when using two-dot-based notation because of the
existing `{ a.. }` notation.
This PR adds a generic `MonadLiftT Id m` instance. We do not implement a
`MonadLift Id m` instance because it would slow down instance resolution
and because it would create more non-canonical instances. This change
makes it possible to iterate over a pure iterator, such as `[1, 2,
3].iter`, in arbitrary monads.
This PR adds a new monadic interface for `Async` operations.
This is the design for the `Async` monad that I liked the most. The idea
was refined with the help of @tydeu. Before that, I had some
prerequisites in mind:
1. Good performance
2. Explicit `yield` points, so we could avoid using `bindTask` for every
lifted IO operation
3. A way to avoid creating an infinite chain of `Task`s during recursion
The 2 and 3 points are not covered in this PR, I wish I had a good
solution but right now only a few sketches of this.
### Explicit `yield` points
I thought this would be easy at first, but it actually turned out kinda
tricky. I ended up creating the `suspend` syntax, which is just a small
modification of the lift method (`<- ...`) syntax. It desugars to
`Suspend.suspend task fun _ => ...`. So something like:
```lean
do
IO.println "a"
IO.println "b"
let result := suspend (client.recv? 1024)
IO.println "c"
IO.println "d"
```
Would become:
```lean
Bind.bind (IO.println "a") fun _ =>
Bind.bind (IO.println "b") fun _ =>
Suspend.suspend (client.recv? 1024) fun message =>
Bind.bind (IO.println "c") fun _ =>
IO.println "d"
```
This makes things a bit more efficient. When using `bind`, we would try
to avoid creating a `Task` chain, and the `suspend` would be the only
place we use `Task.bind`. But there's a problem if we use `bind` with
something that needs `suspend`, it’ll block the whole task. Blocking is
the only way to prevent task accumulation when using plain `bind` inside
a structure like that:
```
inductive AsyncResult (ε σ α : Type u) where
| ok : α → σ → AsyncResult ε σ α
| error : ε → σ → AsyncResult ε σ α
| ofTask : Task (EStateM.Result ε σ α) → σ →AsyncResult ε σ α
```
Because we simply need to remove the `ofTask` and transform it into an
`ok`.
### Infinite chain of Tasks
If you create an infinite recursive function using `Task` (which is
super common in servers like HTTP ones), it can lead to a lot of memory
usage. Because those tasks get chained forever and won't be freed until
the function returns.
To get around that, I used CPS and instead of just calling `Task.bind`,
I’d spawn a new task and return an "empty" one like:
```lean
fun k => Task.bind (...) fun value => do k value; pure emptyTask
```
This works great with a CPS-style monad, but it generates a huge IR by
itself.
Just doing CPS alone was too much, though, because every lifted
operation created a new continuation and a `Task.bind`. So, I used it
with `suspend` and got a better performance, but the usage is not good
with `suspend`.
### The current monad
Right now, the monad I’m using is super simple. It doesn't solve the
earlier problems, but the API is clean, and the generated IR is small
enough. An example of how we should use it is:
```lean
-- A loop that repeatedly sends a message and waits for a reply.
partial def writeLoop (client : Socket.Client) (message : String) : Async (AsyncTask Unit) := async do
IO.println s!"sending: {message}"
await (← client.send (String.toUTF8 message))
if let some mes ← await (← client.recv? 1024) then
IO.println s!"received: {String.fromUTF8! mes}"
-- use parallel to avoid building up an infinite task chain
parallel (writeLoop client message)
else
IO.println "client disconnected from receiving"
-- Server’s main accept loop, keeps accepting and echoing for new clients.
partial def acceptLoop (server : Socket.Server) (promise : IO.Promise Unit) : Async (AsyncTask Unit) := async do
let client ← await (← server.accept)
await (← client.send (String.toUTF8 "tutturu "))
-- allow multiple clients to connect at the same time
parallel (writeLoop client "hi!!")
-- and keep accepting more clients, parallel again to avoid building up an infinite task chain
parallel (acceptLoop server promise)
-- A simple client that connects and sends a message.
def echoClient (addr : SocketAddress) (message : String) : Async (AsyncTask Unit) := async do
let socket ← Client.mk
await (← socket.connect addr)
parallel (writeLoop socket message)
-- TCP setup: bind, listen, serve, and run a sample client.
partial def mainTCP : Async Unit := do
let addr := SocketAddressV4.mk (.ofParts 127 0 0 1) 8080
let server ← Server.mk
server.bind addr
server.listen 128
-- promise exists since the server is (probably) never going to stop
let promise ← IO.Promise.new
let acceptAction ← acceptLoop server promise
await (← echoClient addr "hi!")
await acceptAction
await promise
-- Entry point
def main : IO Unit := mainTCP.wait
```
---------
Co-authored-by: Henrik Böving <hargonix@gmail.com>
Co-authored-by: Mac Malone <tydeu@hatpress.net>
This PR adds a logic of stateful predicates `SPred` to `Std.Do` in order
to support reasoning about monadic programs. It comes with a dedicated
proof mode the tactics of which are accessible by importing
`Std.Tactic.Do`.
Co-authored-by: Sebastian Graf <sg@lean-fro.org>
This PR changes the definition of `DHashMap` to a structure. This makes
it more consistent with the other map types, which are generally defined
as structures. It also ensures that the type `DHashMap α β` is already
in weak head normal form, making it easier for `grind` to successfully
generate patterns for `DHashMap` lemmas.
Although `HEq` was abbreviated as `≍` in #8503, many instances of the
form `HEq x y` still remain.
Therefore, I searched for occurrences of `HEq x y` using the regular
expression `(?<![A-Za-z/@]|``)HEq(?![A-Za-z.])` and replaced as many as
possible with the form `x ≍ y`.
This PR introduces a `ForIn'` instance and a `size` function for
iterators in a minimal fashion. The `ForIn'` instance is not marked as
an instance because it is unclear which `Membership` relation is
sufficiently useful. The `ForIn'` instance existing as a `def` and
inducing the `ForIn` instance, it becomes possible to provide more
specialized `ForIn'` instances, with nice `Membership` relations, for
various types of iterators. The `size` function has no lemmas yet.
This PR adds a new `BitVec.clz` operation and a corresponding `clz`
circuit to `bv_decide`, allowing to bitblast the count leading zeroes
operation. The AIG circuit is linear in the number of bits of the
original expression, making the bitblasting convenient wrt. rewriting.
`clz` is common in numerous compiler intrinsics (see
[here](https://clang.llvm.org/docs/LanguageExtensions.html#intrinsics-support-within-constant-expressions))
and architectures (see
[here](https://en.wikipedia.org/wiki/Find_first_set)).
Co-authored by @bollu.
---------
Co-authored-by: Tobias Grosser <github@grosser.es>
Co-authored-by: Siddharth <siddu.druid@gmail.com>
This PR moves parts of the iterator library from `Std` to `Init`. The
reason is that the polymorphic range API must be in `Init` and it
depends on the iterators.
This PR avoids importing all of `BitVec.Lemmas` and `BitVec.BitBlast`
into `UInt.Lemmas`. (They are still imported into `SInt.Lemmas`; this
seems much harder to avoid.)
This PR adds an equivalence relation to tree maps akin to the existing
one for hash maps. In order to get many congruence lemmas to eventually
use for defining functions on extensional tree maps, almost all of the
remaining tree map functions have also been given lemmas to relate them
to list functions, although these aren't currently used to prove lemmas
other than congruence lemmas.
This PR provides a special empty iterator type. Although its behavior
can be emulated with a list iterator (for example), having a special
type has the advantage of being easier to optimize for the compiler.
This PR replaces special, more optimized `IteratorLoop` instances, for
which no lawfulness proof has been made, with the verified default
implementation. The specialization of the loop/collect implementations
is low priority, but having lawfulness instances for all iterators is
important for verification.
This PR provides the means to reason about "equivalent" iterators.
Simply speaking, two iterators are equivalent if they behave the same as
long as consumers do not introspect their states.
This PR generalizes `Std.Sat.AIG. relabel(Nat)_unsat_iff` to allow the
AIG type to be empty. We generalize the proof, by showing that in the
case when `α` is empty, the environment doesn't matter, since all
environments `α → Bool` are isomorphic.
This showed up when reusing the AIG primitives for building a
k-induction based model checker to prove arbitrary width bitvector
identities.
This PR uses `grind` to shorten some proofs in the LRAT checker. The
intention is not particularly to improve the quality or maintainability
of these proofs (although hopefully this is a side effect), but just to
give `grind` a work out.
There are a number of remaining notes, either about places where `grind`
fails with an internal error (for which #8608 is hopefully
representative, and we can fix after that), or `omega` works but `grind`
doesn't (to be investigated later).
Only in some of the files have I thoroughly used grind. In many files
I've just replaced leaves or branches of proofs with `grind` where it
worked easily, without setting up the internal annotations in the LRAT
library required to optimize the use of `grind`. It's diminishing
returns to do this in a proof library that is not high priority, so I've
simply drawn a line.
This PR provides the iterator combinator `drop` that transforms any
iterator into one that drops the first `n` elements.
Additionally, the PR removes the specialized `IteratorLoop` instance on
`Take`. It currently does not have a `LawfulIteratorLoop` instance,
which needs to exist for the loop consumer lemmas to work. Having the
specialized instance is low priority.
This PR adjusts the grind annotation on
`Std.HashMap.map_fst_toList_eq_keys` and variants, so `grind` can reason
bidirectionally between `m.keys` and `m.toList`.
This PR provides array iterators (`Array.iter(M)`,
`Array.iterFromIdx(M)`), infinite iterators produced by a step function
(`Iter.repeat`), and a `ForM` instance for finite iterators that is
implemented in terms of `ForIn`.
This PR provides the iterator combinators `takeWhile` (forwarding all
emitted values of another iterator until a predicate becomes false)
`dropWhile` (dropping values until some predicate on these values
becomes false, then forwarding all the others).
This PR provides the iterator combinator `filterMap` in a pure and
monadic version and specializations `map` and `filter`. This new
combinator allows to apply a function to the emitted values of a stream
while filtering out certain elements.
`map` should have an optimized `IteratorCollect` implementation but it
turns out that this is not possible without a major refactor of
`IteratorCollect`: `toArrayMapped` requires a proof that the iterator is
finite. If `it.mapM f` is `Finite` but `it` is not, then such a proof
does not exist. `IteratorCollect` needs to take a proof that the loop
will terminate for the given monadic function `f` instead. This will not
be done in this PR.
This PR provides the `take` iterator combinator that transforms any
iterator into an iterator that stops after a given number of steps. The
change contains the implementation and lemmas.
`take` has a special implementation of `IteratorLoop` that relies on a
potentially more efficient `forIn` implementation of the inner iterator.
The mysterious `@[specialize]` on a test has been removed because it is
not necessary anymore according to a manual inspection of the IR. Either
I erroneously concluded from experiments that it was necessary of
something has changed in the meantime that makes it unnecessary.
This PR reworks the `simp` set around the `Id` monad, to not elide or
unfold `pure` and `Id.run`
In particular, it stops encoding the "defeq abuse" of `Id X = X` in the
statements of theorems, instead using `Id.run` and `pure` to pass back
and forth between these two spellings. Often when writing these with
`pure`, they generalize to other lawful monads; though such changes were
split off to other PRs.
This fixes the problem with the current simp set where `Id.run (pure x)`
is simplified to `Id.run x`, instead of the desirable `x`.
This is particularly bad because the` x` is sometimes inferred with type
`Id X` instead of `X`, which prevents other `simp` lemmas about `X` from
firing.
Making `Id` reducible instead is not an option, as then the `Monad`
instances would have nothing to key on.
---------
Co-authored-by: Sebastian Graf <sg@lean-fro.org>
Co-authored-by: Kim Morrison <kim@tqft.net>
Co-authored-by: Paul Reichert <6992158+datokrat@users.noreply.github.com>
This PR provides simple lemmas about `toArray`, `toList` and `toListRev`
for the iterator library.
It also changes the definition of `Iter` and `IterM` so that they aren't
equal anymore and in particular not definitionally equal. While it was
very convenient to have them be definitionally equal when working with
dependent code, it was also confusing and annoying that one would
sometimes end up with something like `it.toList = IterM.toList it`,
where `it : Iter β`.
This PR introduces a very minimal version of the new iterator library.
It comes with list iterators and various consumers, namely `toArray`,
`toList`, `toListRev`, `ForIn`, `fold`, `foldM` and `drain`. All
consumers also come in a partial variant that can be used without any
proofs. This limited version of the iterator library generates decent
code, even with the old code generator.
