diff --git a/.gitignore b/.gitignore index bbf8906..7b364ef 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ build/ .clangd +.nvim.lua .clang-format pyrightconfig.json diff --git a/README.md b/README.md index 978f560..aaefa21 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,35 @@ Each supplied entry replaces that file's entire configuration. Omitted entries u The authored `cmake/` directory belongs in Git. `CMakeFiles/`, compiler caches, and generated sources belong in build directories. Modulus rejects project/source-root output directories before clearing configure or LSP caches. Conan may still use its own isolated build folders. Existing build caches are retained; they are not source files. +### Container Clangd + +All managed repos use clangd 19 in their build image. `configure` and `lsp` rebuild images whose Dockerfile fingerprint has changed and prepare container headers on the host. `lsp` also refreshes extra databases, such as Pico firmware. Run these commands on the host before starting your editor. Direct/container-only builds do not prepare a host header cache. + +Neovim 0.11+ with `exrc` enabled loads the generated, ignored `.nvim.lua` after you review and trust it. The shared Lua helper finishes LazyVim's LSP setup before changing clangd's command and project routing; completion capabilities, keybindings, and attach handlers are preserved. Non-Modulus projects keep their existing server. Content-stable generation avoids repeated trust prompts. An existing hand-authored `.nvim.lua` is never overwritten; it can call `dofile(root .. "/build/build_tools/nvim/modulus.lua").setup(root)` (use `/tools/nvim/` in Modulus itself). Set `tooling.editor.nvim.enabled` to `false` to opt out. + +Other editors can launch `python3 modulus.py clangd` in a consumer repo, or `python3 tools/clangd.py` in Modulus. This is a stdio-only transport: no TTY, downloads, builds, credential environment forwarding, Docker socket, or host-server fallback. Startup fails on stderr if the image, database, or header cache needs preparation. Workspace paths stay identical in the container; sources are read-only and `build/` is writable. The container exits on EOF/shutdown and is removed on launcher termination. + +Server settings are separate from `.clangd`'s parsing/diagnostic settings: + +```json +{ + "tooling": { + "lsp": { + "clangd": { + "executable": "/usr/bin/clangd-19", + "arguments": ["--background-index", "--completion-style=detailed", "--header-insertion=never"], + "query_drivers": ["/usr/bin/gcc-12", "/usr/bin/g++-12", "/usr/bin/aarch64-linux-gnu-gcc", "/usr/bin/aarch64-linux-gnu-g++"], + "header_roots": [] + } + } + } +} +``` + +Omitted settings use these defaults; supplied lists replace defaults. Compiler paths must be explicit, trusted absolute paths, not wildcards. Add embedded compiler paths and SDK header roots in product config. `--query-driver` and `--path-mappings` are managed by the launcher, not the argument list. The default image uses Debian's clangd layout; custom images must provide `/usr/lib/llvm-/lib/clang//include` for built-in headers. + +Header-only snapshots are shared by immutable image ID and header settings under `$XDG_CACHE_HOME/modulus/clangd` (default `~/.cache/modulus/clangd`). Preparation is locked and atomic. Symlinks are dereferenced in the container, and extracted files are read-only. clangd maps definition locations to these local files; Neovim additionally opens them unmodifiable. Old image caches can be removed when no editor uses them; run `lsp` to recreate a removed cache. Toolchains themselves are not copied to the host. + ## Package Create Conan packages with: diff --git a/conanfile.py b/conanfile.py index 21ac319..995418b 100644 --- a/conanfile.py +++ b/conanfile.py @@ -78,6 +78,7 @@ class modulus_recipe(conan.ConanFile): "tests/*.hpp", "schemas/*.json", "tools/*.py", + "tools/nvim/*.lua", "tools/conan/profiles/*", "tools/docker/*", "README.md", diff --git a/modulus/clangd.py b/modulus/clangd.py new file mode 100644 index 0000000..b30aabe --- /dev/null +++ b/modulus/clangd.py @@ -0,0 +1,207 @@ +"""Prepare container headers separately from the stdout-clean LSP transport.""" +import fcntl +import hashlib +import json +import os +import pathlib +import re +import shutil +import signal +import subprocess +import sys +import tarfile +import tempfile +import uuid + +import modulus.config + +image_label = "org.khodo.modulus.dockerfile" +cache_version = 1 + +def settings(context): + return {**modulus.config.clangd_defaults, **context.lsp_settings.get("clangd", {})} + +def recipe_digest(context): + return hashlib.sha256(context.dockerfile.read_bytes()).hexdigest() + +def current_image(context, architecture=None): + try: + result = subprocess.run( + ["docker", "image", "inspect", context.docker_image], + stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, check=False, + ) + if result.returncode != 0: + return None + image = json.loads(result.stdout)[0] + labels = image.get("Config", {}).get("Labels") or {} + if labels.get(image_label) != recipe_digest(context): + return None + if architecture and image["Architecture"] != architecture: + return None + return image["Id"] if re.fullmatch(r"sha256:[0-9a-f]{64}", image["Id"]) else None + except (OSError, ValueError, KeyError, IndexError, TypeError): + return None + +def cache_directory(context, image): + cache_home = pathlib.Path(os.environ.get("XDG_CACHE_HOME", pathlib.Path.home() / ".cache")).resolve() + config = settings(context) + key = {name: config[name] for name in ("executable", "query_drivers", "header_roots")} + key["version"] = cache_version + digest = hashlib.sha256(json.dumps(key, sort_keys=True).encode()).hexdigest()[:20] + return cache_home / "modulus" / "clangd" / image.removeprefix("sha256:") / digest + +def isolated_run(image): + return [ + "docker", "run", "--rm", "--init", "--network", "none", "--read-only", + "--cap-drop=ALL", "--security-opt=no-new-privileges", + "--tmpfs", "/tmp:rw,nosuid,nodev", "--user", f"{os.getuid()}:{os.getgid()}", + "--env", "HOME=/tmp", + ] + +def read_manifest(cache): + data = json.loads((cache / "manifest.json").read_text(encoding="utf-8")) + roots = data["roots"] + if data["version"] != cache_version or not isinstance(roots, list) or not roots: + raise ValueError("invalid header cache manifest") + for root in roots: + path = pathlib.PurePosixPath(root) + if not path.is_absolute() or root == "/" or ".." in path.parts: + raise ValueError("invalid header root") + if not (cache / "rootfs" / root.lstrip("/")).is_dir(): + raise ValueError("incomplete header cache") + return roots + +def extract_headers(stream, destination, roots): + # Only dereferenced regular files/directories below the discovered header roots. + allowed = [pathlib.PurePosixPath(root.lstrip("/")) for root in roots] + with tarfile.open(fileobj=stream, mode="r|") as archive: + for member in archive: + path = pathlib.PurePosixPath(member.name) + if path.is_absolute() or ".." in path.parts or not any(path.is_relative_to(root) for root in allowed): + raise ValueError(f"unsafe header archive path: {member.name}") + target = destination / path + if member.isdir(): + target.mkdir(parents=True, exist_ok=True) + elif member.isfile(): + target.parent.mkdir(parents=True, exist_ok=True) + with archive.extractfile(member) as source, target.open("wb") as output: + shutil.copyfileobj(source, output) + target.chmod(0o444) + else: + raise ValueError(f"non-regular header archive entry: {member.name}") + +def prepare(context): + from modulus.cli import docker_platform_arch + try: + image = current_image(context, docker_platform_arch()) + if image is None: + raise RuntimeError("build image is missing or stale; run configure first") + cache = cache_directory(context, image) + cache.parent.mkdir(parents=True, exist_ok=True) + with (cache.parent / ".lock").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + if cache.exists(): + read_manifest(cache) + return 0 + probe = context.tools_dir / "clangd_headers.py" + result = subprocess.run( + [*isolated_run(image), "--volume", f"{probe}:/modulus-headers.py:ro", image, + "python3", "/modulus-headers.py", json.dumps(settings(context))], + stdout=subprocess.PIPE, text=True, check=True, + ) + roots = json.loads(result.stdout) + with tempfile.TemporaryDirectory(prefix=".headers-", dir=cache.parent) as temporary: + stage = pathlib.Path(temporary) / "cache" + stage.mkdir() + command = [*isolated_run(image), image, "tar", "--dereference", "--hard-dereference", + "--exclude=*/.git", "--exclude=*/__pycache__", + "-C", "/", "-cf", "-", *[root.lstrip("/") for root in roots]] + with subprocess.Popen(command, stdout=subprocess.PIPE) as process: + try: + extract_headers(process.stdout, stage / "rootfs", roots) + finally: + process.stdout.close() + if process.wait() != 0: + raise RuntimeError("could not copy container headers") + (stage / "manifest.json").write_text(json.dumps({"version": cache_version, "roots": roots}) + "\n") + read_manifest(stage) + stage.rename(cache) + print(f"Prepared clangd headers: {cache}", file=sys.stderr) + return 0 + except (OSError, ValueError, KeyError, RuntimeError, subprocess.SubprocessError, tarfile.TarError) as error: + print(f"Modulus clangd preparation: {error}", file=sys.stderr) + return 1 + +def path_mappings(cache, roots): + mappings = [] + for root in sorted(roots, key=len, reverse=True): + client = str(cache / "rootfs" / root.lstrip("/")) + if any(c in client + root for c in ",=\n\r"): + raise ValueError("header paths cannot contain clangd mapping delimiters") + mappings.append(f"{client}={root}") + return "--path-mappings=" + ",".join(mappings) + +def server_arguments(context, argv): + config = settings(context) + return [config["executable"], *config["arguments"], *argv, + "--query-driver=" + ",".join(config["query_drivers"])] + +def launch_command(context, argv, image, cache, roots, name): + root = context.repo_root + return [ + *isolated_run(image), "--interactive", "--name", name, + "--volume", f"{root}:{root}:ro", + "--volume", f"{context.build_root}:{context.build_root}:rw", + "--env", f"XDG_CACHE_HOME={context.common_dir / 'clangd'}", + "--workdir", str(root), image, *server_arguments(context, argv), path_mappings(cache, roots), + ] + +def launch(context, argv): + from modulus.cli import docker_platform_arch + try: + if context.docker_marker.exists(): + os.execv(settings(context)["executable"], server_arguments(context, argv)) + image = current_image(context, docker_platform_arch()) + if image is None: + raise RuntimeError("missing or stale build image; run ./modulus.py configure (Modulus: ./tools/configure.py)") + databases = [context.default_target.build_dir, *[ + context.repo_root / database["build_dir"] for database in context.lsp_settings.get("compile_databases", []) + ]] + for database in databases: + if not (database / "compile_commands.json").is_file(): + raise RuntimeError(f"missing {database / 'compile_commands.json'}; run the lsp command") + cache = cache_directory(context, image) + try: + roots = read_manifest(cache) + except (OSError, ValueError, KeyError) as error: + raise RuntimeError("header cache is missing or invalid; run the lsp command") from error + name = "modulus-clangd-" + uuid.uuid4().hex + command = launch_command(context, argv, image, cache, roots, name) + return run_server(command, name) + except (OSError, ValueError, RuntimeError) as error: + print(f"Modulus clangd: {error}", file=sys.stderr) + return 1 + +def run_server(command, name): + # Inherit stdio verbatim; never send build logs or a terminal over the LSP pipe. + def interrupted(signum, frame): + raise SystemExit(128 + signum) + previous = {sig: signal.signal(sig, interrupted) for sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP)} + process = None + try: + process = subprocess.Popen(command) + return process.wait() + finally: + for sig, handler in previous.items(): + signal.signal(sig, handler) + if process is not None: + try: + subprocess.run(["docker", "rm", "--force", name], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, check=False, timeout=15) + except (OSError, subprocess.TimeoutExpired) as error: + print(f"Modulus clangd cleanup: {error}", file=sys.stderr) + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired: + process.kill() + process.wait() diff --git a/modulus/cli.py b/modulus/cli.py index df1199b..abafda1 100644 --- a/modulus/cli.py +++ b/modulus/cli.py @@ -11,6 +11,7 @@ import urllib.parse import modulus.config import modulus.export +import modulus.clangd build_root = pathlib.Path("build") common_dir = build_root / "build_common" @@ -29,6 +30,8 @@ def main(command): repo_root = find_repo_root(pathlib.Path.cwd()) config = modulus.config.load_repo_config(repo_root) context = build_context(repo_root, config) + if command == "clangd": + return modulus.clangd.launch(context, sys.argv[1:]) if command in {"configure", "lsp"}: modulus.export.sync_editor_files(repo_root, config) if command == "configure": @@ -269,27 +272,7 @@ def docker_platform_arch(): }.get(parts[1]) def docker_image_available(context): - command = [ - "docker", - "image", - "inspect", - context.docker_image, - ] - expected_arch = docker_platform_arch() - if expected_arch is not None: - command.extend(["--format", "{{.Architecture}}"]) - completed = subprocess.run( - command, - check=False, - stdout=subprocess.PIPE if expected_arch is not None else subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - text=True, - ) - if completed.returncode != 0: - return False - if expected_arch is None: - return True - return completed.stdout.strip() == expected_arch + return modulus.clangd.current_image(context, docker_platform_arch()) is not None def build_image(context): return run( @@ -302,6 +285,8 @@ def build_image(context): context.dockerfile, "--tag", context.docker_image, + "--label", + f"{modulus.clangd.image_label}={modulus.clangd.recipe_digest(context)}", ".", ], ) @@ -469,13 +454,14 @@ def configure_one(context, target, update, no_remote=False): def configure(context, argv): if not in_docker(context): - return docker_run( + status = docker_run( context, [ f"./{context.tools_rel}/configure.py", *argv, ], ) + return status or modulus.clangd.prepare(context) parser = argparse.ArgumentParser(description=f"Configure {context.package['name']} for Conan/CMake targets.") parser.add_argument( "--target", @@ -644,13 +630,14 @@ def configure_lsp_database(context, database): def lsp(context, argv): if not in_docker(context): - return docker_run( + status = docker_run( context, [ f"./{context.tools_rel}/lsp.py", *argv, ], ) + return status or modulus.clangd.prepare(context) database_map = lsp_database_map(context) parser = argparse.ArgumentParser(description=f"Refresh {context.package['name']} compile databases for clangd.") parser.add_argument( diff --git a/modulus/config.py b/modulus/config.py index ad21def..0d0a3a3 100644 --- a/modulus/config.py +++ b/modulus/config.py @@ -97,6 +97,16 @@ run_keys = { lsp_keys = { "enabled", "compile_databases", + "clangd", +} +clangd_defaults = { + "executable": "/usr/bin/clangd-19", + "arguments": ["--background-index", "--completion-style=detailed", "--header-insertion=never"], + "query_drivers": [ + "/usr/bin/gcc-12", "/usr/bin/g++-12", + "/usr/bin/aarch64-linux-gnu-gcc", "/usr/bin/aarch64-linux-gnu-g++", + ], + "header_roots": [], } lsp_compile_database_keys = { "name", @@ -247,6 +257,9 @@ def normalize(data): lsp = tooling["lsp"] lsp.setdefault("enabled", False) lsp.setdefault("compile_databases", []) + clangd = lsp.setdefault("clangd", {}) + for name, value in clangd_defaults.items(): + clangd.setdefault(name, copy.deepcopy(value)) for database in lsp["compile_databases"]: database.setdefault("source_dir", ".") database.setdefault("generator", "Ninja") @@ -264,6 +277,7 @@ def normalize(data): def default_editor_config(tooling): target = tooling["targets"][tooling["default_target"]] return { + "nvim": {"enabled": True}, "clangd": [{ "CompileFlags": { "CompilationDatabase": target["build_dir"], @@ -284,7 +298,7 @@ def default_editor_config(tooling): def validate_editor(editor): if not isinstance(editor, dict): raise config_error("modulus.json tooling.editor must be an object") - validate_keys(editor, {"clangd", "clang_format", "pyright"}, "tooling.editor") + validate_keys(editor, {"clangd", "clang_format", "pyright", "nvim"}, "tooling.editor") for name, value in editor.items(): field = f"tooling.editor.{name}" if name == "clangd": @@ -292,6 +306,9 @@ def validate_editor(editor): raise config_error(f"modulus.json {field} must be a non-empty list of configuration objects") elif not isinstance(value, dict): raise config_error(f"modulus.json {field} must be an object") + if name == "nvim": + validate_keys(value, {"enabled"}, field) + validate_bool(value.get("enabled"), f"{field}.enabled", optional=True) try: json.dumps(value, allow_nan=False) except (ValueError, TypeError) as error: @@ -462,6 +479,21 @@ def validate_run(run): def validate_lsp(lsp): validate_bool(lsp.get("enabled"), "tooling.lsp.enabled", optional=True) + if "clangd" in lsp: + clangd = require_mapping(lsp, "clangd") + field = "tooling.lsp.clangd" + validate_keys(clangd, set(clangd_defaults), field) + for name, value in clangd.items(): + values = [value] if name == "executable" else value + validate_string_list(values, f"{field}.{name}") + for item in values: + if name == "arguments": + if item.lstrip("-").split("=", 1)[0] in {"query-driver", "path-mappings"}: + raise config_error(f"modulus.json {field}.arguments must not override query-driver or path-mappings") + else: + validate_absolute_path(item, f"{field}.{name}") + if item == "/" or ".." in pathlib.PurePosixPath(item).parts or any(c in item for c in "*,=[]?\\"): + raise config_error(f"modulus.json {field}.{name} requires literal absolute paths, not patterns") databases = lsp.get("compile_databases", []) if not isinstance(databases, list): raise config_error("modulus.json tooling.lsp.compile_databases must be a list") diff --git a/modulus/export.py b/modulus/export.py index 5520d50..0657640 100644 --- a/modulus/export.py +++ b/modulus/export.py @@ -1,6 +1,7 @@ import json import pathlib import shutil +import sys import modulus.config @@ -80,6 +81,30 @@ def sync_editor_files(repo_root, config): path = pathlib.Path(repo_root) / destination if not path.exists() or path.read_text(encoding="utf-8") != content: path.write_text(content, encoding="utf-8") + sync_nvim_file(pathlib.Path(repo_root), editor.get("nvim", {})) + +def sync_nvim_file(repo_root, settings): + marker = "-- Generated by Modulus from modulus.json tooling.editor.nvim.\n" + path = repo_root / ".nvim.lua" + existing = path.read_text(encoding="utf-8") if path.exists() else None + if existing is not None and not existing.startswith(marker): + print(f"Preserving hand-authored {path}; load tools/nvim/modulus.lua to opt in.", file=sys.stderr) + return + if not settings.get("enabled", True): + if existing is not None: + path.unlink() + return + content = marker + '''local root = vim.fs.dirname(debug.getinfo(1, "S").source:sub(2)) +local tools = vim.uv.fs_stat(root .. "/modulus/cli.py") and "/tools" or "/build/build_tools" +local helper = root .. tools .. "/nvim/modulus.lua" +if vim.uv.fs_stat(helper) then + dofile(helper).setup(root) +else + vim.notify("Modulus editor tooling is missing; run configure first", vim.log.levels.ERROR) +end +''' + if content != existing: + path.write_text(content, encoding="utf-8") def replace_directory(path): if path.exists(): diff --git a/schemas/modulus.schema.json b/schemas/modulus.schema.json index bf3b7cc..65c6005 100644 --- a/schemas/modulus.schema.json +++ b/schemas/modulus.schema.json @@ -147,7 +147,12 @@ "items": { "type": "object" } }, "clang_format": { "type": "object" }, - "pyright": { "type": "object" } + "pyright": { "type": "object" }, + "nvim": { + "type": "object", + "additionalProperties": false, + "properties": { "enabled": { "type": "boolean" } } + } } }, "upload_env": { @@ -200,6 +205,16 @@ "type": "object", "additionalProperties": false, "properties": { + "clangd": { + "type": "object", + "additionalProperties": false, + "properties": { + "executable": { "$ref": "#/$defs/clangd_path" }, + "arguments": { "type": "array", "items": { "type": "string" } }, + "query_drivers": { "type": "array", "items": { "$ref": "#/$defs/clangd_path" } }, + "header_roots": { "type": "array", "items": { "$ref": "#/$defs/clangd_path" } } + } + }, "enabled": { "type": "boolean" }, @@ -211,6 +226,10 @@ } } }, + "clangd_path": { + "type": "string", + "pattern": "^/(?!$)(?!.*(?:^|/)\\.\\.(?:/|$))[^*,=\\[\\]?\\\\\\n\\r]+$" + }, "lsp_compile_database": { "type": "object", "required": [ diff --git a/tests/nvim_clangd.lua b/tests/nvim_clangd.lua new file mode 100644 index 0000000..ee0ccd3 --- /dev/null +++ b/tests/nvim_clangd.lua @@ -0,0 +1,71 @@ +-- Explicit test loading leaves Neovim's persistent exrc trust database unchanged. +if vim.env.MODULUS_NVIM_USER_CONFIG == "1" then + dofile(vim.fn.stdpath("config") .. "/init.lua") + require("lazy").load({ plugins = { "nvim-lspconfig" } }) +else + vim.lsp.config("clangd", { cmd = { "clangd" }, filetypes = { "c", "cpp" }, capabilities = { offsetEncoding = { "utf-16" } } }) +end +vim.o.exrc = false +local root = assert(vim.env.MODULUS_LSP_REPO) +local attached = false +local previous = vim.lsp.config.clangd.on_attach +local on_attach = function(client, buffer) + attached = true + if previous then previous(client, buffer) end +end +vim.lsp.config("clangd", { on_attach = on_attach, capabilities = { modulus_test = true } }) +local original_cmd = vim.lsp.config.clangd.cmd +dofile(root .. "/.nvim.lua") +assert(vim.lsp.config.clangd.on_attach == on_attach) +assert(vim.lsp.config.clangd.capabilities.modulus_test) +local rpc_start = vim.lsp.rpc.start +local fallback +vim.lsp.rpc.start = function(command) fallback = command end +vim.lsp.config.clangd.cmd({}, { root_dir = "/tmp/not-a-modulus-project" }) +vim.lsp.rpc.start = rpc_start +assert(vim.deep_equal(fallback, original_cmd), "non-Modulus server command changed") + +vim.schedule(function() + local ok, error = pcall(function() + vim.cmd.edit(root .. "/" .. assert(vim.env.MODULUS_LSP_SOURCE)) + assert(vim.wait(60000, function() + local clients = vim.lsp.get_clients({ bufnr = 0, name = "clangd" }) + return #clients == 1 and clients[1].initialized and attached + end, 50), "container clangd did not attach") + local clients = vim.lsp.get_clients({ bufnr = 0, name = "clangd" }) + assert(#clients == 1, "duplicate clangd clients") + local client = clients[1] + assert(client.config._modulus_container, "not the container server") + assert(client.config.root_dir == root, "wrong root") + local header = vim.env.MODULUS_LSP_HEADER or "cstdint" + local changed = false + local notification = vim.api.nvim_create_autocmd("LspNotify", { + callback = function(event) + if event.data.method == "textDocument/didChange" then changed = true end + end, + }) + vim.api.nvim_buf_set_lines(0, 0, 0, false, { "#include <" .. header .. ">" }) + assert(vim.wait(5000, function() return changed end, 10), "buffer change was not sent") + vim.api.nvim_del_autocmd(notification) + local result = assert(client:request_sync("textDocument/definition", { + textDocument = { uri = vim.uri_from_bufnr(0) }, position = { line = 0, character = 12 }, + }, 60000, 0)) + assert(result.result and #result.result > 0, vim.inspect(result)) + local errors = vim.diagnostic.get(0, { severity = vim.diagnostic.severity.ERROR }) + assert(#errors == 0, vim.inspect(errors)) + local location = result.result[1] + local target = vim.uri_to_fname(location.uri or location.targetUri) + assert(target:find("/modulus/clangd/", 1, true), target) + vim.lsp.util.show_document(location, client.offset_encoding, { focus = true }) + assert(vim.bo.readonly and not vim.bo.modifiable, "cached header is writable") + assert(#vim.lsp.get_clients({ name = "clangd" }) == 1, "navigation spawned another clangd") + print("MODULUS_NVIM_OK " .. root .. " " .. target) + client:stop() + vim.wait(20000, function() return client:is_stopped() end, 50) + end) + if not ok then + print("MODULUS_NVIM_FAILURE " .. tostring(error)) + vim.cmd("cquit 1") + end + vim.cmd("qa!") +end) diff --git a/tests/test_clangd.py b/tests/test_clangd.py new file mode 100644 index 0000000..a65ca7a --- /dev/null +++ b/tests/test_clangd.py @@ -0,0 +1,161 @@ +import contextlib +import io +import json +import os +import pathlib +import subprocess +import tarfile +import tempfile +import unittest +from unittest.mock import MagicMock, patch + +import modulus.clangd as clangd +import modulus.cli as cli +import modulus.config as config +import modulus.export as export + +class clangd_test(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = pathlib.Path(temporary.name) + self.context = cli.build_context(self.root, config.normalize({"package": {"name": "sample"}})) + self.context.dockerfile.parent.mkdir(parents=True) + self.context.dockerfile.write_text("FROM sample\n") + self.image = "sha256:" + "1" * 64 + + def test_defaults_and_custom_settings(self): + self.assertEqual(clangd.settings(self.context)["executable"], "/usr/bin/clangd-19") + self.assertIn("/usr/bin/g++-12", clangd.settings(self.context)["query_drivers"]) + custom = {"executable": "/opt/bin/clangd", "arguments": [], "query_drivers": ["/opt/bin/g++"], "header_roots": ["/opt/sdk"]} + result = config.normalize({"package": {"name": "sample"}, "tooling": {"lsp": {"clangd": custom}}}) + self.assertEqual(result["tooling"]["lsp"]["clangd"], custom) + + def test_config_rejects_unsafe_or_unknown_settings(self): + for invalid in ([], {"unknown": True}, {"executable": "clangd"}, {"arguments": "--background-index"}, + {"query_drivers": ["/usr/bin/*"]}, {"header_roots": ["/"]}, + {"header_roots": ["/usr/../etc"]}, {"query_drivers": ["/bin/g++,/bin/gcc"]}, + {"arguments": ["--query-driver=*"]}, {"arguments": ["--path-mappings=/=/"]}): + with self.subTest(invalid=invalid), self.assertRaises(config.config_error): + config.normalize({"package": {"name": "sample"}, "tooling": {"lsp": {"clangd": invalid}}}) + + def test_image_fingerprint_and_architecture(self): + image = {"Id": self.image, "Architecture": "amd64", "Config": {"Labels": {clangd.image_label: clangd.recipe_digest(self.context)}}} + with patch.object(subprocess, "run", return_value=subprocess.CompletedProcess([], 0, json.dumps([image]))): + self.assertEqual(clangd.current_image(self.context), self.image) + self.assertIsNone(clangd.current_image(self.context, "arm64")) + self.context.dockerfile.write_text("FROM changed\n") + self.assertIsNone(clangd.current_image(self.context)) + + def test_cache_is_shared_by_image_and_header_settings_not_repo(self): + cache = clangd.cache_directory(self.context, self.image) + other = cli.build_context(self.root / "other", self.context.config) + self.assertEqual(cache, clangd.cache_directory(other, self.image)) + self.context.lsp_settings["clangd"]["arguments"].append("--log=error") + self.assertEqual(cache, clangd.cache_directory(self.context, self.image)) + self.context.lsp_settings["clangd"]["header_roots"].append("/opt/sdk") + self.assertNotEqual(cache, clangd.cache_directory(self.context, self.image)) + self.assertNotEqual(cache, clangd.cache_directory(other, "sha256:" + "2" * 64)) + + def test_stdio_launch_has_no_tty_socket_credentials_or_host_home(self): + command = clangd.launch_command(self.context, [], self.image, self.root / "cache", ["/usr/include"], "test-server") + self.assertIn("--interactive", command) + self.assertIn("--read-only", command) + self.assertIn(f"{self.root}:{self.root}:ro", command) + self.assertIn(f"{self.context.build_root}:{self.context.build_root}:rw", command) + self.assertIn("--network", command) + self.assertNotIn("-t", command) + for unwanted in ("--tty", "--volumes-from", "CONAN_TOKEN", "/var/run/docker.sock", str(pathlib.Path.home()) + ":"): + self.assertNotIn(unwanted, " ".join(command)) + self.assertIn("--query-driver=/usr/bin/gcc-12,/usr/bin/g++-12,/usr/bin/aarch64-linux-gnu-gcc,/usr/bin/aarch64-linux-gnu-g++", command) + + def test_startup_failure_does_not_prepare_and_keeps_stdout_clean(self): + output, errors = io.StringIO(), io.StringIO() + with patch.object(clangd, "current_image", return_value=None), patch.object(clangd, "prepare") as prepare, \ + patch.object(cli, "ensure_build_image") as build, contextlib.redirect_stdout(output), contextlib.redirect_stderr(errors): + self.assertEqual(clangd.launch(self.context, []), 1) + prepare.assert_not_called() + build.assert_not_called() + self.assertEqual(output.getvalue(), "") + self.assertIn("configure", errors.getvalue()) + + def test_path_mapping_uses_client_to_server_direction_and_specificity(self): + self.assertEqual(clangd.path_mappings(pathlib.Path("/cache"), ["/usr/include", "/opt/sdk"]), + "--path-mappings=/cache/rootfs/usr/include=/usr/include,/cache/rootfs/opt/sdk=/opt/sdk") + + def archive(self, name, kind=tarfile.REGTYPE): + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w") as archive: + member = tarfile.TarInfo(name) + member.type = kind + member.linkname = "/etc/passwd" if kind == tarfile.SYMTYPE else "" + archive.addfile(member) + stream.seek(0) + return stream + + def test_header_extraction_rejects_escape_and_links(self): + for name, kind in (("../bad", tarfile.REGTYPE), ("/usr/include/bad", tarfile.REGTYPE), + ("usr/include/../bad", tarfile.REGTYPE), ("etc/passwd", tarfile.REGTYPE), + ("usr/include/link", tarfile.SYMTYPE)): + with self.subTest(name=name), self.assertRaises(ValueError): + clangd.extract_headers(self.archive(name, kind), self.root, ["/usr/include"]) + clangd.extract_headers(self.archive("usr/include/header.h"), self.root, ["/usr/include"]) + self.assertEqual((self.root / "usr/include/header.h").stat().st_mode & 0o777, 0o444) + + def test_completed_cache_never_invokes_container_again(self): + cache = self.root / "cache" + (cache / "rootfs/usr/include").mkdir(parents=True) + (cache / "manifest.json").write_text(json.dumps({"version": clangd.cache_version, "roots": ["/usr/include"]})) + with patch.object(clangd, "cache_directory", return_value=cache), patch.object(clangd, "current_image", return_value=self.image), \ + patch.object(subprocess, "run") as run: + self.assertEqual(clangd.prepare(self.context), 0) + run.assert_not_called() + + def test_failed_extraction_never_publishes_partial_cache(self): + cache = self.root / "image/cache" + process = MagicMock() + process.__enter__.return_value = process + process.stdout = self.archive("../escape") + with patch.object(clangd, "cache_directory", return_value=cache), patch.object(clangd, "current_image", return_value=self.image), \ + patch.object(subprocess, "run", return_value=subprocess.CompletedProcess([], 0, '["/usr/include"]')), \ + patch.object(subprocess, "Popen", return_value=process), contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(clangd.prepare(self.context), 1) + self.assertFalse(cache.exists()) + self.assertEqual(list(cache.parent.glob(".headers-*")), []) + + def test_cleanup_always_waits_for_child_even_if_docker_removal_times_out(self): + process = MagicMock() + process.wait.side_effect = [0, subprocess.TimeoutExpired("docker", 5), 0] + with patch.object(subprocess, "Popen", return_value=process), \ + patch.object(subprocess, "run", side_effect=subprocess.TimeoutExpired("docker rm", 15)), \ + contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(clangd.run_server(["docker"], "test-server"), 0) + process.kill.assert_called_once() + self.assertEqual(process.wait.call_count, 3) + + def test_nvim_generation_is_stable_opt_out_and_respects_ownership(self): + export.sync_nvim_file(self.root, {}) + path = self.root / ".nvim.lua" + before = path.stat().st_mtime_ns + export.sync_nvim_file(self.root, {}) + self.assertEqual(before, path.stat().st_mtime_ns) + export.sync_nvim_file(self.root, {"enabled": False}) + self.assertFalse(path.exists()) + path.write_text("-- user settings\n") + with contextlib.redirect_stderr(io.StringIO()): + export.sync_nvim_file(self.root, {}) + export.sync_nvim_file(self.root, {"enabled": False}) + self.assertEqual(path.read_text(), "-- user settings\n") + + def test_lsp_prepares_after_successful_database_refresh_only(self): + with patch.object(cli, "in_docker", return_value=False), patch.object(cli, "docker_run", return_value=0) as run, \ + patch.object(clangd, "prepare", return_value=0) as prepare: + self.assertEqual(cli.lsp(self.context, []), 0) + prepare.assert_called_once_with(self.context) + prepare.reset_mock() + run.return_value = 1 + self.assertEqual(cli.lsp(self.context, []), 1) + prepare.assert_not_called() + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_clangd_integration.py b/tests/test_clangd_integration.py new file mode 100644 index 0000000..ba7c1db --- /dev/null +++ b/tests/test_clangd_integration.py @@ -0,0 +1,164 @@ +"""Opt-in real Docker/LSP checks: MODULUS_LSP_TEST_ROOTS=/repo:/other/repo.""" +import json +import os +import pathlib +import queue +import shutil +import subprocess +import tempfile +import threading +import time +import unittest +import urllib.parse + +class lsp_session: + def __init__(self, root): + command = ["python3", "tools/clangd.py"] if (root / "modulus/cli.py").is_file() else ["python3", "modulus.py", "clangd"] + self.log = tempfile.TemporaryFile(mode="w+b") + self.process = subprocess.Popen(command, cwd=root, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=self.log) + self.messages = queue.Queue() + self.reader = threading.Thread(target=self.read_messages, daemon=True) + self.reader.start() + self.next_id = 0 + self.pending = [] + + def read_messages(self): + try: + while True: + line = self.process.stdout.readline() + if not line: + raise EOFError("LSP stream closed") + headers = {} + while line != b"\r\n": + name, value = line.decode().strip().split(":", 1) + headers[name.lower()] = value.strip() + line = self.process.stdout.readline() + if not line: + raise EOFError("incomplete LSP header") + self.messages.put(json.loads(self.process.stdout.read(int(headers["content-length"])))) + except Exception as error: + self.messages.put(error) + + def send(self, message): + body = json.dumps({"jsonrpc": "2.0", **message}).encode() + self.process.stdin.write(f"Content-Length: {len(body)}\r\n\r\n".encode() + body) + self.process.stdin.flush() + + def wait(self, predicate, timeout=90): + deadline = time.monotonic() + timeout + while True: + for index, message in enumerate(self.pending): + if predicate(message): + return self.pending.pop(index) + try: + message = self.messages.get(timeout=max(0, deadline - time.monotonic())) + except queue.Empty: + raise AssertionError("LSP response timed out") from None + if isinstance(message, Exception): + self.log.seek(0) + raise AssertionError(f"{message}\n{self.log.read().decode(errors='replace')}") + self.pending.append(message) + + def request(self, method, params): + self.next_id += 1 + self.send({"id": self.next_id, "method": method, "params": params}) + response = self.wait(lambda message: message.get("id") == self.next_id) + if "error" in response: + raise AssertionError(response["error"]) + return response.get("result") + + def initialize(self, root): + result = self.request("initialize", {"processId": os.getpid(), "rootUri": root.as_uri(), + "capabilities": {"textDocument": {"publishDiagnostics": {"versionSupport": True}}}}) + self.send({"method": "initialized", "params": {}}) + return result + + def open(self, path, text): + uri = path.as_uri() + self.send({"method": "textDocument/didOpen", "params": {"textDocument": { + "uri": uri, "languageId": "cpp", "version": 1, "text": text}}}) + return self.wait(lambda message: message.get("method") == "textDocument/publishDiagnostics" + and message["params"]["uri"] == uri)["params"]["diagnostics"] + + def close(self): + if self.process.poll() is None: + self.process.terminate() + self.process.wait(timeout=25) + self.process.stdin.close() + self.reader.join(timeout=5) + self.process.stdout.close() + self.log.close() + +@unittest.skipUnless(os.environ.get("MODULUS_LSP_TEST_ROOTS"), "requires prepared Docker images and databases") +class container_clangd_test(unittest.TestCase): + def roots(self): + return [pathlib.Path(root).resolve() for root in os.environ["MODULUS_LSP_TEST_ROOTS"].split(os.pathsep)] + + @unittest.skipUnless(shutil.which("nvim"), "Neovim is not installed") + def test_neovim_attachment_and_readonly_navigation(self): + script = pathlib.Path(__file__).with_name("nvim_clangd.lua") + for root in self.roots(): + database = json.loads((root / "build/build_x86_debug/compile_commands.json").read_text()) + source = next(pathlib.Path(entry["file"]) for entry in database + if pathlib.Path(entry["file"]).is_relative_to(root / "src")) + sources = [(source, "cstdint")] + if (root / "src/vamp_pico/main.cpp").exists(): + sources.append((root / "src/vamp_pico/main.cpp", "pico/stdlib.h")) + for source, header in sources: + with self.subTest(repo=root.name, source=str(source)): + env = {**os.environ, "MODULUS_LSP_REPO": str(root), "MODULUS_LSP_SOURCE": str(source.relative_to(root)), + "MODULUS_LSP_HEADER": header} + result = subprocess.run(["nvim", "--headless", "-i", "NONE", "-u", str(script)], cwd=root, + env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, timeout=180) + self.assertEqual(result.returncode, 0, result.stdout) + self.assertIn("MODULUS_NVIM_OK", result.stdout) + + def test_native_and_pico_diagnostics_and_local_header_definitions(self): + for root in self.roots(): + with self.subTest(repo=root.name): + session = lsp_session(root) + try: + initialized = session.initialize(root) + self.assertIn("19.", initialized["serverInfo"]["version"]) + database = json.loads((root / "build/build_x86_debug/compile_commands.json").read_text()) + source = next(pathlib.Path(entry["file"]) for entry in database + if pathlib.Path(entry["file"]).is_relative_to(root / "src")) + sources = [(source, "cstdint")] + if (root / "src/vamp_pico").is_dir(): + sources.extend((path, "pico/stdlib.h") for path in sorted((root / "src/vamp_pico").glob("*.cpp"))) + for source, header in sources: + with self.subTest(source=str(source)): + text = f"#include <{header}>\n" + source.read_text() + diagnostics = session.open(source, text) + self.assertEqual([item for item in diagnostics if item.get("severity") == 1], []) + definitions = session.request("textDocument/definition", { + "textDocument": {"uri": source.as_uri()}, "position": {"line": 0, "character": 12}}) + self.assertTrue(definitions) + uri = definitions[0].get("uri", definitions[0].get("targetUri")) + path = pathlib.Path(urllib.parse.unquote(urllib.parse.urlparse(uri).path)) + self.assertIn("/modulus/clangd/", str(path)) + self.assertTrue(path.is_file(), str(path)) + self.assertEqual(path.stat().st_mode & 0o222, 0) + session.request("shutdown", None) + session.send({"method": "exit"}) + session.process.stdin.close() + self.assertEqual(session.process.wait(timeout=25), 0) + finally: + session.close() + + def test_eof_and_termination_remove_container(self): + def containers(): + return set(subprocess.check_output(["docker", "ps", "-q", "--filter", "name=modulus-clangd-"], text=True).split()) + before = containers() + for terminate in (False, True): + session = lsp_session(self.roots()[0]) + try: + session.initialize(self.roots()[0]) + if terminate: + session.process.terminate() + else: + session.process.stdin.close() + session.process.wait(timeout=25) + self.assertEqual(containers() - before, set()) + finally: + session.close() diff --git a/tests/test_cli.py b/tests/test_cli.py index 2167c10..b194ac4 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -189,7 +189,7 @@ class upload_packages_test(unittest.TestCase): previous_cwd = pathlib.Path.cwd() os.chdir(repo) try: - for command in ("image", "docs", "shell", "clangd", "docker_support"): + for command in ("image", "docs", "shell", "docker_support"): with self.assertRaisesRegex(SystemExit, "unknown modulus command"): modulus.cli.main(command) finally: @@ -332,6 +332,8 @@ class upload_packages_test(unittest.TestCase): context.dockerfile, "--tag", "sample-build:dev", + "--label", + f"{modulus.cli.modulus.clangd.image_label}={modulus.cli.modulus.clangd.recipe_digest(context)}", ".", ], ) diff --git a/tests/test_export.py b/tests/test_export.py index 73c4eb4..959caa0 100644 --- a/tests/test_export.py +++ b/tests/test_export.py @@ -263,7 +263,7 @@ class modulus_export_test(unittest.TestCase): } config["tooling"] = {"editor": editor} normalized = modulus.config.normalize(config) - self.assertEqual(normalized["tooling"]["editor"], editor) + self.assertEqual(normalized["tooling"]["editor"], {**editor, "nvim": {"enabled": True}}) with tempfile.TemporaryDirectory() as directory: root = pathlib.Path(directory) modulus.export.sync_editor_files(root, normalized) diff --git a/tools/clangd.py b/tools/clangd.py new file mode 100755 index 0000000..9d8ec80 --- /dev/null +++ b/tools/clangd.py @@ -0,0 +1,5 @@ +#!/usr/bin/env python3 +import modulus_tool + +if __name__ == "__main__": + modulus_tool.run("clangd") diff --git a/tools/clangd_headers.py b/tools/clangd_headers.py new file mode 100755 index 0000000..f0aff9c --- /dev/null +++ b/tools/clangd_headers.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +"""Runs only in the build image, during configure/lsp, never server startup.""" +import json +import pathlib +import re +import subprocess +import sys + +def header_roots(config): + roots = set() + for driver in config["query_drivers"]: + for language in ("c", "c++"): + result = subprocess.run([driver, "-E", "-v", "-x", language, "-"], input="", + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=True) + listing = False + for line in result.stderr.splitlines(): + if line == "#include <...> search starts here:": + listing = True + elif line == "End of search list.": + listing = False + elif listing: + roots.add(pathlib.Path(line.strip()).resolve()) + version = subprocess.check_output([config["executable"], "--version"], text=True) + major = re.search(r"version (\d+)", version).group(1) + builtin = pathlib.Path(f"/usr/lib/llvm-{major}/lib/clang/{major}/include") + roots.add(builtin) + roots.update(pathlib.Path(root) for root in config["header_roots"]) + for root in roots: + if not root.is_dir() or root == pathlib.Path("/"): + raise ValueError(f"missing or invalid header directory: {root}") + return sorted(str(root) for root in roots if not any(parent in roots for parent in root.parents)) + +if __name__ == "__main__": + print(json.dumps(header_roots(json.loads(sys.argv[1])))) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index f0fc403..7dae6ee 100755 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -9,7 +9,7 @@ RUN apt-get update \ bash \ build-essential \ ca-certificates \ - clangd \ + clangd-19 \ cmake \ curl \ g++-12 \ @@ -31,6 +31,7 @@ RUN apt-get update \ pkg-config \ python3 \ python3-venv \ + && ln -s /usr/bin/clangd-19 /usr/local/bin/clangd \ && rm -rf /var/lib/apt/lists/* ENV CC=/usr/bin/gcc-12 diff --git a/tools/nvim/modulus.lua b/tools/nvim/modulus.lua new file mode 100644 index 0000000..9eb54ad --- /dev/null +++ b/tools/nvim/modulus.lua @@ -0,0 +1,96 @@ +local M = {} + +local function within(path, root) + return path == root or path:sub(1, #root + 1) == root .. "/" +end + +function M.setup(root) + if vim.fn.has("nvim-0.11") ~= 1 then + vim.notify("Modulus container clangd requires Neovim 0.11+", vim.log.levels.ERROR) + return + end + root = vim.fs.normalize(root) + -- Finish LazyVim's server setup before changing only the command and root routing. + local has_lazy, lazy = pcall(require, "lazy") + if has_lazy then + lazy.load({ plugins = { "nvim-lspconfig" } }) + end + local state = rawget(_G, "modulus_clangd") + if not state then + state = { roots = {} } + _G.modulus_clangd = state + local original = vim.lsp.config.clangd or { cmd = { "clangd" }, filetypes = { "c", "cpp", "objc", "objcpp", "cuda" } } + if not vim.lsp.config.clangd then + vim.lsp.config("clangd", original) + end + local original_cmd, original_root = original.cmd, original.root_dir + local cache = vim.fs.normalize((vim.env.XDG_CACHE_HOME or (vim.env.HOME .. "/.cache")) .. "/modulus/clangd") + local function project(path) + local best + for candidate in pairs(state.roots) do + if within(path, candidate) and (not best or #candidate > #best) then + best = candidate + end + end + return best + end + state.project = project + vim.lsp.config("clangd", { + cmd = function(dispatchers, config) + local selected = config.root_dir and state.roots[config.root_dir] + if selected then + config._modulus_container = true + return vim.lsp.rpc.start(selected, dispatchers, { cwd = config.root_dir }) + end + if type(original_cmd) == "function" then + return original_cmd(dispatchers, config) + end + return vim.lsp.rpc.start(original_cmd or { "clangd" }, dispatchers, { cwd = config.root_dir }) + end, + root_dir = function(buf, on_dir) + local name = vim.api.nvim_buf_get_name(buf) + if within(name, cache) then + local clients = vim.tbl_filter(function(client) return client.config._modulus_container end, vim.lsp.get_clients({ name = "clangd" })) + if #clients == 1 then + vim.lsp.buf_attach_client(buf, clients[1].id) + end + return + end + local selected = project(name) + if selected then + on_dir(selected) + elseif type(original_root) == "function" then + original_root(buf, on_dir) + elseif type(original_root) == "string" then + on_dir(original_root) + else + on_dir(vim.fs.root(name, { ".clangd", "compile_commands.json", ".git" }) or vim.fs.dirname(name)) + end + end, + }) + vim.api.nvim_create_autocmd("BufReadPost", { + group = vim.api.nvim_create_augroup("ModulusClangdHeaders", { clear = true }), + callback = function(event) + if within(vim.api.nvim_buf_get_name(event.buf), cache) then + vim.bo[event.buf].readonly = true + vim.bo[event.buf].modifiable = false + end + end, + }) + end + local command + if vim.uv.fs_stat(root .. "/modulus/cli.py") then + command = { "python3", root .. "/tools/clangd.py" } + else + command = { "python3", root .. "/modulus.py", "clangd" } + end + state.roots[root] = command + for _, client in ipairs(vim.lsp.get_clients({ name = "clangd" })) do + if not client.config._modulus_container and client.config.root_dir and within(client.config.root_dir, root) then + client:stop() + end + end + vim.lsp.enable("clangd") +end + +return M