This PR significantly improves the test coverage of the language server,
providing at least a single basic test for every request that is used by
the client. It also implements infrastructure for testing all of these
requests, e.g. the ability to run interactive tests in a project context
and refactors the interactive test runner to be more maintainable.
Finally, it also fixes a small bug with the recently implemented unknown
identifier code actions for auto-implicits (#10442) that was discovered
in testing, where the "import all unambiguous unknown identifiers" code
action didn't work correctly on auto-implicit identifiers.
This PR adds support for remote artifact caches (e.g., Reservoir) to
Lake. As part of this support, a new suite of `lake cache` CLI commands
has been introduced to help manage Lake's cache. Also, the existing
local cache support has been overhauled for better interplay with the
new remote support.
**Cache CLI**
Artifacts are uploaded to a remote cache via `lake cache put`. This
command takes a JSON Lines input-to-outputs file which describes the
output artifacts for a build (indexed by its input hash). This file can
be produced by a run of `lake build` with the new `-o` option. Lake will
write the input-to-outputs mappings of thee root package artifacts
traversed by the build to the file specified via `-o`. This file can
then be passed to `lake cache put` to upload both it and the built
artifacts from the local cache to the remote cache.
The remote cache service can be customized using the following
environment variables:
* `LAKE_CACHE_KEY`: This is the authorization key for the remote cache.
Lake uploads artifacts via `curl` using the AWS Signature Version 4
protocol, so this should be the S3 `<key>:<secret>` pair expected by
`curl`.
* `LAKE_CACHE_ARTIFACT_ENDPOINT`: This is the base URL to upload (or
download) artifacts to a given remote cache. Artifacts will be stored at
`<endpoint>/<scope/<content-hash>.art`.
* `LAKE_CACHE_REVISION_ENDPOINT`: This is the base URL to upload (or
download) input-to-output mappings to a given remote cache. Mappings are
indexed by the Git revision of the package, and are stored at
`<endpoint>/<scope/<rev>.jsonl`.
The `<scope>` is provided through the `--scope` option to `lake cache
put`. This option is used to prevent one package from overwriting the
artifacts/mappings of another. Lake artifact hashes and Git revisions
hashes are not cryptographically secure, so it is not safe for a service
to store untrusted files across packages in a single flat store.
Once artifacts are available in a remote cache, the `lake cache get`
command can be used to retrieve them. By default, it will fetch
artifacts for the root package's dependencies from Reservoir using its
API. But, like `cache put`, it can be configured to use a custom
endpoint with the above environment variables and an explicit `--scope`.
When so configured, `cache get` will instead download artifacts for the
root package. Lake only downloads artifacts for a single package in this
case, because it cannot deduce the necessary package scopes without
Reservoir.
**Significant local cache changes**
* Lake now always has a cache directory. If Lake cannot find a good
candidate directory on the system for the cache, it will instead store
the cache at `.lake/cache` within the workspace.
* If the local cache is disabled, Lake will not save built artifacts to
the cache. However, Lake will, nonetheless, always attempt to lookup
build artifacts in the cache. If found, the cached artifact will be
copied to the the build location ("restored").
* Input-to-outputs mappings in the local cache are no longer stored in a
single file for a package, but rather in individual files per input (in
the `outputs` subdirectory of the cache).
* Outputs in a trace file, outputs file, or mappings file are now an
`ArtifactDescr`, which is currently composed of both the content hash
and the file extension.
* Trace files now contain a date-based `schemaVersion` to help make
version to version migration easier. Hashes in JSON and in artifacts
names now use a 16-digit hexadecimal encoding (instead of a variable
decimal encoding).
* `buildArtifactUnlessUpToDate` now returns an `Artifact` instead of a
`FilePath`.
**NOTE:** The Lake local cache is still disabled by default. This means
that built artifacts, by default, will not be placed in the cache
directory, and thus will not be available for `lake cache put` to
upload. Users must first explicitly enable the cache by either setting
the `LAKE_ARTIFACT_CACHE` environment variable to a truthy value or by
setting the `enableArtifactCache` package configuration option to
`true`.
This PR refactors Lake's package naming procedure to allow packages to
be renamed by the consumer. With this, users can now require a package
using a different name than the one it was defined with.
This is support will be used in the future to enable seamlessly
including the same package at multiple different versions within the
same workspace.
In a Lake package configuration file written in Lean, the current
package's assigned name is now accessed through `__name__` instead of
the previous `_package.name`. A deprecation warning has been added to
`_package.name` to assist in migration.
This PR upstreams the Verso parser and adds preliminary support for
Verso in docstrings. This will allow the compiler to check examples and
cross-references in documentation.
After a `stage0` update, a follow-up PR will add the appropriate
attributes that allow the feature to be used. The parser tests from
Verso also remain to be upstreamed, and user-facing documentation will
be added once the feature has been used on more internals.
This PR fixes bugs in #7809 and #7909 that were not caught partially
because the `badImport` test had been disabled.
**Bugs Fixed:**
* Building by path no longer drops top-level logs.
* "bad import" errors are once again printed.
* Transitively imported precompiled modules are once again loaded during
elaboration.
These tests are currently flaky in `merge-ci` and nightly releases, so
they are being temporarily disabled. Whatever the issue is will be
debugged in a separate PR.
This PR fixes the order of libraries when loading them via
`--load-dynlib` or `--plugin` in `lean` and when linking them into a
shared library or executable. A `Dynlib` now tracks its dependencies and
they are topologically sorted before being passed to either linking or
loading.
Closes#7790.
Adds a new type of `require` which fetches package metadata from a
registry API endpoint (i.e., Reservoir) and then clones a Git package
using the information provided. To require such a dependency, the new
syntax is:
```lean
require <scope> / <pkg-name> [@ "git#<rev>"] -- e.g., require "leanprover" / "doc-gen4"
```
Or in TOML:
```toml
[[require]]
name = "<pkg-name>"
scope = "<scope>"
rev = "<rev>"
```
Unlike with Git dependencies, Lake can make use of the richer
information provided by the registry to determine the default branch of
the package. This means for repositories of packages like `doc-gen4`
which have a default branch that is not `master`, Lake will now use said
default branch (e.g., in `doc-gen4`'s case, `main`).
Lake also supports configuring the registry endpoint via an environment
variable: `RESERVIOR_API_URL`. Thus, any server providing a similar
interface to Reservoir can be used as the registry. Further
configuration options paralleling those of Cargo's [Alternative
Registries](https://doc.rust-lang.org/cargo/reference/registries.html)
and [Source
Replacement](https://doc.rust-lang.org/cargo/reference/source-replacement.html)
will come in the future.
Updated and split from #3174.
On Windows, we now compile all core `.o`s twice, once with and without
`dllexport`, for use in the shipped dynamic and static libraries,
respectively. On other platforms, we export always as before to avoid
the duplicate work.
---------
Co-authored-by: tydeu <tydeu@hatpress.net>
This is a temporary workaround for a limitation on Windows shared
libraries. We are getting errors of the form:
```
ld.lld: error: too many exported symbols (got 65572, max 65535)
```
The test is flaky due to the presence of a fixed 'sleep()'.
The LLVM backend has introduced a performance
regression in Lake which causes this test to fail, as the
current sleep duration of 3s is insufficient.
Further investigation into the performance regression is pending.
We decided to disable the `leanlaketest_116` entirely on account
of the test being flaky by construction
(https://github.com/leanprover/lean4/pull/2358#issuecomment-1655371232).
This adds `lean.h.bc`, a LLVM bitcode file of the Lean
runtime that is to be inlined. This is programatically generated.
1. This differs from the previous `libleanrt.ll`, since it produces an
LLVM bitcode file, versus a textual IR file. The bitcode file
is faster to parse and build an in-memory LLVM module from.
2. We build `lean.h.bc` by adding it as a target to `shell`,
which ensures that it is always built.
3. We eschew the need for:
```cpp
```
which causes breakage in the build, since changing the meaning of
`static` messes with definitions in the C++ headers.
Instead, we build `lean.h.bc` by copying everything in
`src/include/lean/lean.h`, renaming `inline` to
`__attribute__(alwaysinline)` [which forces LLVM to generate
`alwaysinline` annotations], then running the `-O3` pass pipeline
to get reasonably optimised IR, and will be perfectly inlined
when linked into the generated LLVM code by
`src/Lean/Compiler/IR/EmitLLVM.lean`.
Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
I guess CMake might not actually be that painful if one was motivated to
learn it
```
sed -Ei 's/set\(([A-Z_]+) "\$\{\1\}/string(APPEND \1 "/' src/**/CMakeLists.txt
```
* fix msys2 windows build so the windows apps support utf-8 file paths.
* use windres to compile default-manifest.o
* windres is in binutils.
* stop modifying default-manifest.o
* copy to stage0
* fix semicolon joining of lists in add_custom_target
* undo changes to stage0 as per CR feedback.
* fix makefile
* fix: revert cmakelists.txt COMMAND_EXPAND_LISTS change
* fix: msys2 dependencies
* add unit test for decoding UTF-8 chars to prove "lean.exe" can read utf-8 encoded files where utf-8 is also used in the file name.
* fix: utf-8 test by using windows-2022
* fix: do we really need cmake 3.11 or will 3.10 do?
* nope, really does require cmake 11.
