This PR enables the elaboration of theorem bodies, i.e. proofs, to
happen in parallel to each other as well as to other elaboration tasks.
Specifically, to be eligible for parallel proof elaboration,
* the theorem must not be in a `mutual` block
* `deprecated.oldSectionVars` must not be set
* `Elab.async` must be set (currently defaults to `true` in the language
server, `false` on the cmdline)
To be activated for downstream projects (i.e. in stage 1) pending
further Mathlib validation.
This PR adds server-side support for dedicated 'unsolved goals' and
'goals accomplished' diagnostics that will have special support in the
Lean 4 VS Code extension. The special 'unsolved goals' diagnostic is
adapted from the 'unsolved goals' error diagnostic, while the 'goals
accomplished' diagnostic is issued when a `theorem` or `Prop`-typed
`example` has no errors or `sorry`s. The Lean 4 VS Code extension
companion PR is at leanprover/vscode-lean4#585.
Specifically, this PR extends the diagnostics served by the language
server with the following fields:
- `leanTags`: Custom tags that denote the kind of diagnostic that is
being served. As opposed to the `code`, `leanTags` should never be
displayed in the UI. Examples introduced by this PR are a tag to
distinguish 'unsolved goals' errors from other diagnostics, as well as a
tag to distinguish the new 'goals accomplished' diagnostic from other
diagnostics.
- `isSilent`: Whether a diagnostic should not be displayed as a regular
diagnostic in the editor. In VS Code, this means that the diagnostic is
displayed in the InfoView under 'Messages', but that it will not be
displayed under 'All Messages' and that it will also not be displayed
with a squiggly line.
The `isSilent` field is also implemented for `Message` so that silent
diagnostics can be logged in the elaborator. All code paths except for
the language server that display diagnostics to users are adjusted to
filter `Message`s with `isSilent := true`.
This PR fixes several inlay hint race conditions that could result in a
violation of the monotonic progress assumption, introduced in #7149.
Specifically:
- In rare circumstances, it could happen that stateful LSP requests were
executed out-of-order with their `didChange` handlers, as both requests
and the `didChange` handlers waited on `lake setup-file` to complete,
with the latter running those handlers in a dedicated task afterwards.
This meant that a request could be added to the stateful LSP handler
request queue before the corresponding `didChange` call that actually
came before it. This PR resolves this issue by folding the task that
waits for `lake setup-file` into the `RequestContext`, which ensures
that we only need to wait for it when actually executing the request
handler.
- While #7164 fixed the monotonic progress assertion violation that was
caused by `$/cancelRequest`, it did not account for our internal notion
of silent request cancellation in stateful LSP requests, which we use to
cancel the inlay hint edit delay when VS Code fails to emit a
`$/cancelRequest` notification. This issue is resolved by always
producing the full finished prefix of the command snapshot queue, even
on cancellation. Additionally, this also fixes an issue where in the
same circumstances, the language server could produce an empty inlay
hint response when a request was cancelled by our internal notion of
silent request cancellation.
- For clients that use `fullChange` `didChange` notifications (e.g. not
VS Code), we would get several aspects of stateful LSP request
`didChange` state handling wrong, which is also addressed by this PR.
This PR ensures that all tasks in the language server either use
dedicated tasks or reuse an existing thread from the thread pool. This
ensures that elaboration tasks cannot prevent language server tasks from
being scheduled. This is especially important with parallelism right
around the corner and elaboration becoming more likely to starve the
language server of computation, which could drive up language server
latencies significantly on machines with few cores.
Specifically, all language server tasks are refactored to use a new thin
`ServerTask` API wrapper with a single "costly" vs "cheap" dimension,
where costly tasks are always scheduled as dedicated tasks, and cheap
tasks are always made to either run on the calling thread or to reuse
the thread of the task being mapped on by using the `sync` flag.
ProofWidgets4 adaption PR:
https://github.com/leanprover-community/ProofWidgets4/pull/106
### Other changes
- This PR makes several tasks dedicated that weren't dedicated before,
and uses `sync := true` for some others. The rules for this are
described in the module docstring of `ServerTask.lean`.
- Most notably, the reporting task in the file worker was *not* a
dedicated task before this PR, which could easily lead to thread pool
starvation on successive changes. It also did not support cancellation.
This PR ensures that it does.
### Breaking changes
- `RequestTask` and the request-oriented snapshot API are refactored to
use `ServerTask` instead of `Task`. All functions in `Task` have close
analogues in `ServerTask`, and functions on `RequestTask` now need to
distinguish between whether a `map` or a `bind` is cheap or costly. This
affects all downstream users of `RequestM`, e.g. tools that extend the
language server with their own requests, or some users of the RPC
mechanism.
- The following unused functions of the `AsyncList` API have been
deleted: `append`, `unfoldAsync`, `getAll`, `waitHead?`, `cancel`
This PR adds language server support for request cancellation to the
following expensive requests: Code actions, auto-completion, document
symbols, folding ranges and semantic highlighting. This means that when
the client informs the language server that a request is stale (e.g.
because it belongs to a previous state of the document), the language
server will now prematurely cancel the computation of the response in
order to reduce the CPU load for requests that will be discarded by the
client anyways.
This PR fixes a bug where the goal state selection would sometimes
select incomplete incremental snapshots on whitespace, leading to an
incorrect "no goals" response. Fixes#6594, a regression that was
originally introduced in 4.11.0 by #4727.
The fundamental cause of #6594 was that the snapshot selection would
always select the first snapshot with a range that contains the cursor
position. For tactics, whitespace had to be included in this range.
However, in the test case of #6594, this meant that the snapshot
selection would also sometimes pick a snapshot before the cursor that
still contains the cursor in its whitespace, but which also does not
necessarily contain all the information needed to produce a correct goal
state. Specifically, at the `InfoTree`-level, when the cursor is in
whitespace, we distinguish competing goal states by their level of
indentation. The snapshot selection did not have access to this
information, so it necessarily had to do the wrong thing in some cases.
This PR fixes the issue by adjusting the snapshot selection for goals to
explicitly account for whitespace and indentation, and refactoring the
language processor architecture to thread enough information through to
the snapshot selection so that it can decide which snapshots to use
without having to force too many tasks, which would destroy
incrementality in goal state requests.
Specifically, this PR makes the following adjustments:
- Refactor `SnapshotTask` to contain both a `Syntax` and a `Range`.
Before, `SnapshotTask`s had a single range that was used both for
displaying file progress information and for selecting snapshots in
server requests. For most snapshots, this range did not include
whitespace, though for tactics it did. Now, the `reportingRange` field
of `SnapshotTask` is intended exclusively for reporting file progress
information, and the `Syntax` is used for selecting snapshots in server
requests. Importantly, the `Syntax` contains the full range information
of the snapshot, i.e. its regular range and its range including
whitespace.
- Adjust all call-sites of `SnapshotTask` to produce a reasonable
`Syntax`.
- Adjust the goal snapshot selection to account for whitespace and
indentation, as the `InfoTree` goal selection does.
- Fix a bug in the snapshot tree tracing that would cause it to render
the `Info` of a snapshot at the wrong location when `trace.Elab.info`
was also set.
This PR is based on #6329.
This PR enables the language server to present multiple disjoint line
ranges as being worked on. Even before parallelism lands, we make use of
this feature to show post-elaboration tasks such as kernel checking on
the first line of a declaration to distinguish them from the final
tactic step.
This PR implements a number of refinements for the auto-implicit inlay
hints implemented in #6768.
Specifically:
- In #6768, there was a bug where the inlay hint edit delay could
accumulate on successive edits, which meant that it could sometimes take
much longer for inlay hints to show up. This PR implements the basic
infrastructure for request cancellation and implements request
cancellation for semantic tokens and inlay hints to resolve the issue.
With this edit delay bug fixed, it made more sense to increase the edit
delay slightly from 2000ms to 3000ms.
- In #6768, we applied the edit delay to every single inlay hint request
in order to reduce the amount of inlay hint flickering. This meant that
the edit delay also had a significant effect on how far inlay hints
would lag behind the file progress bar. This PR adjusts the edit delay
logic so that it only affects requests sent directly after a
corresponding `didChange` notification. Once the edit delay is used up,
all further semantic token requests are responded to without delay, so
that the only latency that affects how far the inlay hints lag behind
the progress bar is how often we emit refresh requests and how long VS
Code takes to respond to them.
- For inlay hints, refresh requests are now emitted 500ms after a
response to an inlay hint request, not 2000ms, which means that after
the edit delay, inlay hints should only lag behind the progress bar by
about up to 500ms. This is justifiable for inlay hints because the
response should be much smaller than e.g. is the case for semantic
tokens.
- In #6768, 'Restart File' did not prompt a refresh, but it does now.
- VS Code does not immediately remove old inlay hints from the document
when they are applied. In #6768, this meant that inlay hints would
linger around for a bit once applied. To mitigate this issue, this PR
adjusts the inlay hint edit delay logic to identify edits sent from the
client as being inlay hint applications, and sets the edit delay to 0ms
for the inlay hint requests following it. This means that inlay hints
are now applied immediately.
- In #6768, hovering over single-letter auto-implicit inlay hints was a
bit finicky because VS Code uses the regular cursor icon on inlay hints,
not the thin text cursor icon, which means that it is easy to put the
cursor in the wrong spot. We now add the separation character (` ` or
`{`) preceding an auto-implicit to the hover range as well, which makes
hovering over inlay hints much smoother.
This PR adds support for plugins to the frontend and server.
Implementation-wise, this adds a `plugins` argument to `runFrontend`,
`processHeader`, amd `importModules`, a `plugins` field to
`SetupImportsResult` and `FileSetupResult`. and a `pluginsPath` field to
`LeanPaths`, and then threads the value through these.
This PR adds preliminary support for inlay hints, as well as support for
inlay hints that denote the auto-implicits of a function. Hovering over
an auto-implicit displays its type and double-clicking the auto-implicit
inserts it into the text document.
This PR is an extension of #3910.
### Known issues
- In VS Code, when inserting an inlay hint, the inlay hint may linger
for a couple of seconds before it disappears. This is a defect of the VS
Code implementation of inlay hints and cannot adequately be resolved by
us.
- When making a change to the document, it may take a couple of seconds
until the inlay hints respond to the change. This is deliberate and
intended to reduce the amount of inlay hint flickering while typing. VS
Code has a mechanism of its own for this, but in my experience it is
still far too sensitive without additional latency.
- Inserting an auto-implicit inlay hint that depends on an auto-implicit
meta-variable causes a "failed to infer binder type" error. We can't
display these meta-variables in the inlay hint because they don't have a
user-displayable name, so it is not clear how to resolve this problem.
- Inlay hints are currently always resolved eagerly, i.e. we do not
support the `textDocument/inlayHint/resolve` request yet. Implementing
support for this request is future work.
### Other changes
- Axioms did not support auto-implicits due to an oversight in the
implementation. This PR ensures they do.
- In order to reduce the amount of inlay hint flickering when making a
change to the document, the language server serves old inlay hints for
parts of the file that have not been processed yet. This requires LSP
request handler state (that sometimes must be invalidated on
`textDocument/didChange`), so this PR introduces the notion of a
stateful LSP request handler.
- The partial response mechanism that we use for semantic tokens, where
we simulate incremental LSP responses by periodically emitting refresh
requests to the client, is generalized to accommodate both inlay hints
and semantic tokens. Additionally, it is made more robust to ensure that
we never emit refresh requests while a corresponding request is in
flight, which causes VS Code to discard the respond of the request, as
well as to ensure that we keep prompting VS Code to send another request
if it spuriously decides not to respond to one of our refresh requests.
- The synthetic identifier of an `example` had the full declaration as
its (non-canonical synthetic) range. Since we need a reasonable position
for the identifier to insert an inlay hint for the auto-implicits of an
`example`, we change the (canonical synthetic) range of the synthetic
identifier to that of the `example` keyword.
- The semantic highlighting request handling is moved to a separate
file.
### Breaking changes
- The semantic highlighting request handler is not a pure request
handler anymore, but a stateful one. Notably, this means that clients
that extend the semantic highlighting of the Lean language server with
the `chainLspRequestHandler` function must now use the
`chainStatefulLspRequestHandler` function instead.
Avoids build time overhead until the option is proven to speed up
average projects. Adds Init.Prelude (many tiny declarations, "worst
case") and Init.List.Sublist (many nontrivial theorems, "best case")
under -DElab.async=true as new benchmarks for tracking.
This PR moves `IO.Channel` and `IO.Mutex` from `Init` to `Std.Sync` and
renames them to `Std.Channel` and `Std.Mutex`.
Note that the original files are retained and the deprecation is written
manually as we cannot import `Std` from `Init` so this is the only way
to deprecate without a hard breaking change. In particular we do not yet
move `Std.Queue` from `Init` to `Std` both because it needs to be
retained for this deprecation to work but also because it is already
within the `Std` namespace and as such we cannot maintain two copies of
the file at once. After the deprecation period is finished `Std.Queue`
will find a new home in `Std.Data.Queue`.
This PR fixes three problems:
- When the language server is being stopped in a non-normal way without
going through the regular LSP shutdown protocol (e.g. by closing VS
Code), it could sometimes happen that both the watchdog and the file
worker were not properly terminated and lingered around forever,
resulting in zombie processes (#5296)
- When the file worker crashes and the user restarts it by making a
change to the document, the file worker would produce incorrect
diagnostics for the document until the file is restarted.
- (Minor) When the file worker would crash during initialization, the
error diagnostic would be reported on stderr instead of stdout
The deadlock-induced termination issue from #5296 should be resolved by
the following measures:
- The watchdog main task is always terminated with `IO.Process.exit` to
ensure that it terminates even if some other tasks in the process are
still running.
- The file worker communication task in the watchdog no longer waits for
the file worker process to terminate when writing to the client fails,
only when reading from the file worker fails.
- When the watchdog shuts down (either as a result of an orderly or a
non-normal shutdown), instead of waiting for the file worker
communication tasks to complete, it kills the file worker process. The
rationale behind this is that the file worker currently should have no
essential work to complete if the server is being stopped anyways, and
so waiting for the communication task is not necessary.
The file worker diagnostic desync after a crash was caused by us
tracking changes to the document of a crashed file worker twice: Once as
part of the document, and once as part of the queued messages to the
file worker. This meant that when the file worker was restarted, it
would receive the changes made to the document while the file worker was
crashed twice, leading to a desynced document state.
(Probably) fixes#5296.
On a document edit, it may be the case that the first nontrivial
snapshot is e.g. for a macro-generated tactic call that does not have
range information. In that case, instead of just displaying nothing, we
should fall back to a previous range, in this case of the original
tactic macro.
This PR fixes a small bug where over time, "import out of data" messages
would accumulate in files when their size changed before restarting its
file worker.
Extends Lean's incremental reporting and reuse between commands into
various steps inside declarations:
* headers and bodies of each (mutual) definition/theorem
* `theorem ... := by` for each contained tactic step, including
recursively inside supported combinators currently consisting of
* `·` (cdot), `case`, `next`
* `induction`, `cases`
* macros such as `next` unfolding to the above
*Incremental reuse* means not recomputing any such steps if they are not
affected by a document change. *Incremental reporting* includes the
parts seen in the recording above: the progress bar and messages. Other
language server features such as hover etc. are *not yet* supported
incrementally, i.e. they are shown only when the declaration has been
fully processed as before.
---------
Co-authored-by: Scott Morrison <scott.morrison@gmail.com>
To eliminate parsing differences between Windows and other platforms,
the frontend now normalizes all CRLF line endings to LF, like [in
Rust](https://github.com/rust-lang/rust/issues/62865).
Effects:
- This makes Lake hashes be faithful to what Lean sees (Lake already
normalizes line endings before computing hashes).
- Docstrings now have normalized line endings. In particular, this fixes
`#guard_msgs` failing multiline tests for Windows users using CRLF.
- Now strings don't have different lengths depending on the platform.
Before this PR, the following theorem is true for LF and false for CRLF
files.
```lean
example : "
".length = 1 := rfl
```
Note: the normalization will take `\r\r\n` and turn it into `\r\n`. In
the elaborator, we reject loose `\r`'s that appear in whitespace. Rust
instead takes the approach of making the normalization routine fail.
They do this so that there's no downstream confusion about any `\r\n`
that appears.
Implementation note: the LSP maintains its own copy of a source file
that it updates when edit operations are applied. We are assuming that
edit operations never split or join CRLFs. If this assumption is not
correct, then the LSP copy of a source file can become slightly out of
sync. If this is an issue, there is some discussion
[here](https://github.com/leanprover/lean4/pull/3903#discussion_r1592930085).
As [reported on
Zulip](https://leanprover.zulipchat.com/#narrow/stream/341532-lean4-dev/topic/Find.20references.20broken.20in.20lean.20core/near/437051935).
The `mainModuleName` was being set incorrectly when browsing lean core
sources, resulting in failure of cross-file server requests like "Find
References". Because the `srcSearchPath` is generated asynchronously, we
store it as a `Task Name` which is resolved some time before the header
is finished parsing. (I don't think the `.get` here will ever block,
because the srcSearchPath will be ready by the time the initial command
snap is requested.)
---------
Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
While implementing #3925, I noticed that the performance of the
`textDocument/semanticTokens/full` request is *extremely* bad due to a
quadratic implementation. Specifically, on my machine, computing the
full semantic tokens for `Lean/Elab/Do.lean` took a full 5s. In
practice, this means that while elaborating the file, one core is
entirely busy with computing the semantic tokens for the file.
This PR fixes this performance bug by re-implementing the semantic token
handling, reducing the latency for `Lean/Elab/Do.lean` from 5s to 60ms.
As a result, the overly cautious refresh latency of 5s in #3925 can
easily be reduced to 2s again.
Since the previous semantic tokens implementation used a very brittle
hack to identify projections, this PR also changes the projection
notation elaboration to augment the `InfoTree` syntax for the field of a
projection with a special syntax node of kind
`Lean.Parser.Term.identProjKind`. With this syntax kind, projection
fields can now easily be identified in the `InfoTree`.
The concrete dependency that is stale isn't really actionable
information for users (ideally we'd like something like "amount of
dependencies that will be rebuilt when you restart file"). This also
makes the diagnostic an "information" diagnostic so that non-infoview
users can still see it.
Since we are moving away from using notifications for stale dependency
information, we don't need to provide an ID anymore, either.
I forgot to use the sticky diagnostics in `getInteractiveDiagnostics` in
#3247, leading to them not consistently showing up in the "Messages"
panel of the InfoView.
This lead to incorrect diagnostic spans in the editor and resulted in
header errors that did not show up under "Messages" everywhere in the
file because the `fullRange?` property was missing.
Also changes the "Import out of date" warning diagnostic severity to
"Hint" so that it doesn't show up in the "Problems" view.
Sends a diagnostic informing the user to run Restart File when a file
dependency is saved.
Based on #3014 because this feature was easier to implement with the new
architecture.
ToDo:
- [x] Adjust vscode-lean4 to display a notification when this diagnostic
appears in a non-annoying way
(https://github.com/leanprover/vscode-lean4/pull/393)
- [x] Use a file watcher to identify changes to files not tracked by VS
Code
- [x] Rebase onto master when #3014 is merged
This reverts commit 4e3a8468c3 for PR
#3619. It looks like the CI in that commit didn't inform me that a test
was broken by the PR, so I managed to commit it despite the broken test.
This PR fixes an issue where the file worker would not provide the
client with semantic tokens until the file had been elaborated
completely. The file worker now also tells the client to refresh its
semantic tokens after running "Restart File". This PR is based on #3271.
This PR adds support for requests from the server to the client in the
language server. It is based on #3014 and was developed during an
experiment for #3247 that unfortunately did not go anywhere.
This PR addresses several performance issues in the auto-completion
implementation. It also fixes a number of smaller bugs related to
auto-completion.
In a file with `import Mathlib`, the performance of various kinds of
completions has improved as follows:
- Completing `C`: 49000ms -> 1400ms
- Completing `Cat`: 14300ms -> 1000ms
- Completing `x.` for `x : Nat`: 3700ms -> 220ms
- Completing `.` for an expected type of `Nat`: 11000ms -> 180ms
The following bugs have been fixed as well:
- VS Code never used our custom completion order. Now, the server fuzzy
completion score decides the order that completions appear in.
- Dot auto-completion for private types did not work at all. It does
now.
- Completing `.<identifier>` (where the expected type is used to infer
the namespace) did not filter by the expected type and instead displayed
all matching constants in the respective namespace. Now, it uses the
expected type for filtering. Note that this is not perfect because
sub-namespaces are technically correct completions as well (e.g.
`.Foo.foobar`). Implementing this is future work.
- Completing `.` was often not possible at all. Now, as long as the `.`
is not used in a bracket (where it may be used for the anonymous lambda
feature, e.g. `(. + 1)`), it triggers the correct completion.
- Fixes#3228.
- The auto-completion in `#check` commands would always try to complete
identifiers using the full declaration name (including namespaces) if it
could be resolved. Now it simply uses the identifier itself in case
users want to complete this identifier to another identifier.
## Details
Regarding completion performance, I have more ideas on how to improve it
further in the future.
Other changes:
- The feature that completions with a matching expected type are sorted
to the top of the server-side ordering was removed. This was never
enabled in VS Code because it would use its own completion item order
and when testing it I found it to be more confusing than useful.
- In the server-side ordering, we would always display keywords at the
top of the list. They are now displayed according to their fuzzy match
score as well.
The following approaches have been used to improve performance:
- Pretty-printing the type for every single completion made up a
significant amount of the time needed to compute the completions. We now
do not pretty-print the type for every single completion that is offered
to the user anymore. Instead, the language server now supports
`completionItem/resolve` requests to compute the type lazily when the
user selects a completion item.
- Note that we need to keep the amount of properties that we compute in
a resolve request to a minimum. When the server receives the resolve
request, the document state may have changed from the state it was in
when the initial auto-completion request was received. LSP doesn't tell
us when it will stop sending resolve requests, so we cannot keep this
state around, as we would have to keep it around forever.
LSP's solution for this dilemma is to have servers send all the state
they need to compute a response to a resolve request to the client as
part of the initial auto completion response (which then sends it back
as part of the resolve request), but this is clearly infeasible for all
real language servers where the amount of state needed to resolve a
request is massive.
This means that the only practical solution is to use the current state
to compute a response to the resolve request, which may yield an
incorrect result. This scenario can especially occur when using
LiveShare where the document is edited by another person while cycling
through available completions.
- Request handlers can now specify a "header caching handler" that is
called after elaborating the header of a file. Request handlers can use
this caching handler to compute caches for information stored in the
header. The auto-completion uses this to pre-compute non-blacklisted
imported declarations, which in turn allow us to iterate only over
non-blacklisted imported declarations where we would before iterate over
all declarations in the environment. This is significant because
blacklisted declarations make up about 4/5 of all declarations.
- Dot completion now looks up names modulo private prefixes to figure
out whether a declaration is in the namespace of the type to the left of
the dot instead of first stripping the private prefix from the name and
then comparing it. This has the benefit that we do not need to scan the
full name in most cases.
This PR also adds a couple of regression tests for fixed bugs, but *no
benchmarks*. We will add these in the future when we add proper support
for benchmarking server interaction sessions to our benchmarking
architecture.
All tests that were broken by producing different completion output
(empty `detail` field, added `sortText?` and `data?` fields) have been
manually checked by me to be still correct before replacing their
expected output.
This PR adds support for the "call hierarchy" feature of LSP that allows
quickly navigating both inbound and outbound call sites of functions. In
this PR, "call" is taken to mean "usage", so inbound and outbound
references of all kinds of identifiers (e.g. functions or types) can be
navigated. To implement the call hierarchy feature, this PR implements
the LSP requests `textDocument/prepareCallHierarchy`,
`callHierarchy/incomingCalls` and `callHierarchy/outgoingCalls`.
<details>
<summary>Showing the call hierarchy (click to show image)</summary>
</details>
<details>
<summary>Incoming calls (click to show image)</summary>
</details>
<details>
<summary>Outgoing calls (click to show image)</summary>
</details>
It is based on #3159, which should be merged before this PR.
To route the parent declaration name through to the language server, the
`.ilean` format is adjusted, breaking backwards compatibility with
version 1 of the ILean format and yielding version 2.
This PR also makes the following more minor adjustments:
- `Lean.Server.findModuleRefs` now also combines the identifiers of
constants and FVars and prefers constant over FVars for the combined
identifier. This is necessary because e.g. declarations declared using
`where` yield both a constant (for usage outside of the function) and an
FVar (for usage inside of the function) with the same range, whereas we
would typically like all references to refer to the former. This also
fixes a bug introduced in #2462 where renaming a declaration declared
using `where` would not rename usages outside of the function, as well
as a bug in the unused variable linter where `where` declarations would
be reported as unused even if they were being used outside of the
function.
- The function converting `Lean.Server.RefInfo` to `Lean.Lsp.RefInfo`
now also computes the `Lean.DeclarationRanges` for parent declaration
names via `MetaM` and must hence be in `IO` now.
- Add a utility function `Array.groupByKey` to `HashMap.lean`.
- Stylistic refactoring of `Watchdog.lean` and `LanguageFeatures.lean`.
This PR facilitates augmenting the context of an `InfoTree` with
*partial* contexts while elaborating a command. Using partial contexts,
this PR also adds support for tracking the parent declaration name of a
term in the `InfoTree`. The parent declaration name is needed to compute
the call hierarchy in #3082.
Specifically, the `Lean.Elab.InfoTree.context` constructor is refactored
to take a value of the new type `Lean.Elab.PartialContextInfo` instead
of a `Lean.Elab.ContextInfo`, which now refers to a full `InfoTree`
context. The `PartialContextInfo` is then merged into a `ContextInfo`
while traversing the tree using
`Lean.Elab.PartialContextInfo.mergeIntoOuter?`. The partial context
after executing `liftTermElabM` is stored in values of a new type
`Lean.Elab.CommandContextInfo`.
As a result of this, `Lean.Elab.ContextInfo.save` moves to
`Lean.Elab.CommandContextInfo.save`.
For obtaining the parent declaration for a term, a new typeclass
`MonadParentDecl` is introduced to save the parent declaration in
`Lean.Elab.withSaveParentDeclInfoContext`. `Lean.Elab.Term.withDeclName
x` now calls `withSaveParentDeclInfoContext x` to save the declaration
name.
### Migration
**The changes to the `InfoTree.context` constructor break backwards
compatibility with all downstream users that traverse the `InfoTree`
manually instead of going through the functions in `InfoUtils.lean`.**
To fix this, you can merge the outer `ContextInfo` in a traversal with
the `PartialContextInfo` of an `InfoTree.context` node using
`PartialContextInfo.mergeIntoOuter?`. See e.g.
`Lean.Elab.InfoTree.foldInfo` for an example:
```lean
partial def InfoTree.foldInfo (f : ContextInfo → Info → α → α) (init : α) : InfoTree → α :=
go none init
where go ctx? a
| context ctx t => go (ctx.mergeIntoOuter? ctx?) a t
| node i ts =>
let a := match ctx? with
| none => a
| some ctx => f ctx i a
ts.foldl (init := a) (go <| i.updateContext? ctx?)
| _ => a
```
Downstream users that manually save `InfoTree`s may need to adjust calls
to `ContextInfo.save` to use `CommandContextInfo.save` instead and
potentially wrap their `CommandContextInfo` in a
`PartialContextInfo.commandCtx` constructor when storing it in an
`InfoTree` or `ContextInfo.mk` when creating a full context.
### Motivation
As of now, `ContextInfo`s are always *full* contexts, constructed as if
they were always created in `liftTermElabM` after running the
`TermElabM` action. This is not strictly true; we already create
`ContextInfo`s in several places other than `liftTermElabM` and work
around the limitation that `ContextInfo`s are always full contexts in
certain places (e.g. `Info.updateContext?` is a crux that we need
because we can't always create partial contexts at the term-level), but
it has mostly worked out so far. Note that one must be very careful when
saving a `ContextInfo` in places other than `liftTermElabM` because the
context may not be as complete as we would like (e.g. it may lack
meta-variable assignments, potentially leading to a language server
panic).
Unfortunately, the parent declaration of a term is another example of a
context that cannot be provided in `liftTermElabM`: The parent
declaration is usually set via `withDeclName`, which itself lives in
`TermElabM`. So by the time we are trying to save the full
`ContextInfo`, the declaration name is already gone. There is no easy
fix for this like in the other cases where we would really just like to
augment the context with an extra field.
The refactor that we decided on to resolve the issue is to refactor the
`InfoTree` to take a `PartialContextInfo` instead of a `ContextInfo` and
have code that traverses the `InfoTree` merge inner contexts with outer
contexts to produce a full `ContextInfo` value.
### Bumps for downstream projects
- `lean-pr-testing-3159` branch at Std, not yet opened as a PR
- `lean-pr-testing-3159` branch at Mathlib, not yet opened as a PR
- https://github.com/leanprover/LeanInk/pull/57
- https://github.com/hargoniX/LeanInk/pull/1
- https://github.com/tydeu/lean4-alloy/pull/7
- https://github.com/leanprover-community/repl/pull/29
---------
Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
This PR adds per-package server options to resolve#2455. It is based on
the previous work in #2456, but takes a different approach: options are
loaded for the specific file in the file worker when `print-paths` is
called, instead of loading them in the watchdog with a separate Lake
command. This change addresses review comments made in #2456.
In doing so, it introduces two new Lake config fields:
- `leanOptions`: `-D` flag options that are passed to both the language
server and `lean` when building.
- `moreServerOptions`: `-D` flag options that are passed to the language
server.
Since `print-paths` must also accept a file path to compute the options
for that file, this PR is changing the API for `print-paths`. As there
have been numerous complaints about the name `print-paths`, I also
decided to change it to `setup-file` in this PR, since it would break
compatibility with the old Lake API anyways.
This PR deprecates the Lakefile field `moreServerArgs` in favor of
`moreGlobalServerArgs`, as suggested in the review for #2456.
Fixes#2455
---------
Co-authored-by: digama0 <mcarneir@andrew.cmu.edu>
This PR adds basic auto-completion support for imports. Since it still
lacks Lake support for accurate completion suggestions (cc @tydeu - we
already know what needs to be done), it falls back to traversing the
`LEAN_SRC_PATH` for available imports.
Three kinds of import completion requests are supported:
- Completion of the full `import` command. Triggered when requesting
completions in an empty space within the header.
- Known issue: It is possible to trigger this completion within a
comment in the header. Fixing this would require architecture for
parsing some kind of sub-syntax between individual commands.
- Completion of the full module name after an incomplete `import`
command.
- Completion of a partial module name with a trailing dot.
Since the set of imports is potentially expensive to compute, they are
cached for 10 seconds after the last import auto-completion request.
Closes#2655.
### Changes
This PR also makes the following changes:
- To support completions on the trailing dot, the `import` syntax was
adjusted to provide partial syntax when a trailing dot is used.
- `FileWorker.lean` was refactored lightly with some larger definitions
being broken apart.
- The `WorkerState` gained two new fields:
- `currHeaderStx` tracks the current header syntax, as opposed to
tracking only the initial header syntax in `initHeaderStx`. When the
header syntax changes, a task is launched that restarts the file worker
after a certain delay to avoid constant restarts while editing the
header. During this time period, we may still want to serve import
auto-completion requests, so we need to know the up-to-date header
syntax.
- `importCachingTask?` contains a task that computes the set of
available imports.
- `determineLakePath` has moved to a new file `Lean/Util/LakePath.lean`
as it is now needed both in `ImportCompletion.lean` and
`FileWorker.lean`.
- `forEachModuleIn` from `Lake/Config/Blob.lean` has moved to
`Lean/Util/Path.lean` as it is a generally useful utility function that
was useful for traversing the `LEAN_SRC_PATH` as well.
### Tests
Unfortunately, this PR lacks tests since the set of imports available in
`tests/lean/interactive` will not be stable. In the future, I will add
support for testing LSP requests in full project setups, which is when
tests for import auto-completion will be added as well.
