Skip to content

Just: introduce common "verbs"#19978

Open
redsun82 wants to merge 53 commits intomainfrom
redsun82/just2
Open

Just: introduce common "verbs"#19978
redsun82 wants to merge 53 commits intomainfrom
redsun82/just2

Conversation

@redsun82
Copy link
Copy Markdown
Contributor

@redsun82 redsun82 commented Jul 4, 2025

This introduces common "verbs" (build, test, format, lint, generate) that individual parts of the project can implement, and shared infrastructure in misc/just/ to support them.

Forwarding

The core of the functionality is given by forwarding. The idea is that:

  • if you are in the directory where a verb is implemented, you will get that as per standard just behaviour (possibly using fallback).
  • if on the other hand you are above it, and you run something like just test ql/rust/ql/test/{a,b}, then a forwarder script finds a common justfile implementing the verb for all the positional arguments passed there, and then retries calling just test from there. So if test is implemented beneath that (in that case, it is in rust/ql/test), it uses that recipe.
  • even if there isn't a recipe that is common to all the positional arguments, the forwarder will still group the arguments in batches using the same recipe. So just build ql/rust ql/java, or just test ql/rust/ql/test/some/language/test ql/rust/ql/integration-test/some/integration/test will also work, with corresponding recipes run sequentially.

Running QL tests

How launching QL tests can be tweaked:

  • by default, the corresponding CLI is built from the internal repo (nothing is done if working in codeql standalone), and no additional database or consistency checks are made
  • --codeql=built can be passed to skip the build step (if no changes were made to the CLI/extractors). This is consistent with the same pytest option
  • you can add the additional checks that CI does with --all-checks or the + abbreviation. These additional checks are configured in justfiles per language, and correspond to all the additional checks that CI adds (but that a dev might not want to run by default).

What this PR does

Adds shared just infrastructure (misc/just/)

  • defs.just: core verb definitions and helpers
  • forward.just / forward_command.py: forwarding mechanism to dispatch verbs from any directory
  • language_tests.py / codeql_test_run.py: Python helpers for running codeql test run with the right options
  • build.just / format.just / lib.just: shared recipes for building, formatting, and common settings
  • semmle-code-stub.just: stubs for internal-repo-specific settings when working standalone
  • README.md: documentation for the whole system

Adds per-language justfiles

Each language (actions, cpp, csharp, go, java, javascript, python, ruby, rust, swift) gets:

  • a top-level <lang>/justfile importing the shared infrastructure
  • <lang>/ql/justfile and <lang>/ql/test/justfile for QL tests
  • <lang>/ql/integration-tests/justfile for integration tests

Moves C++ consistency queries

semmlecode-cpp-consistency-queries/cpp/ql/consistency-queries/ (these were in the internal repo only, but belong here).

Other changes

  • Adds codegen support (notably for rust)
  • Adds root justfile and lib.just for top-level verb dispatch
  • Upgrades the just binary and modernizes justfile syntax

Caveats

  • Passing arguments with spaces generally doesn't work, although setting arguments with spaces in justfiles (for the base arguments) is supported using escaping as in \\ . This is a known limitation of just (see Question: How to forward *args? casey/just#1988)
  • When running different recipes for the same verb, non-positional arguments need to be supported by all recipes involved. For example, this will work ok for --learn or --codeql options in language and integration tests

Companion PR

The internal repo companion PR ports all languages to use this new infrastructure and updates CI accordingly.

@github-actions github-actions bot added Rust Pull requests that update Rust code Swift labels Jul 4, 2025
@github-actions github-actions bot added the Java label Jul 7, 2025
redsun82 added 13 commits July 8, 2025 10:52
This allows to mix different verb implementations in a single
invocation, for example:
```
just build ql/rust ql/java
just test ql/rust/ql/test/some/test ql/rust/ql/integrartion-test/some other
```
If a common justfile recipe is found, it is used for all arguments in
one go. If on the other hand no common justfile recipe is found, each
argument is processed separately in sequence.

This does require that any flags passed are compatible with all recipes
involved (like is the case for `--learn` or `--codeql=built` for
language and integration tests).
@redsun82 redsun82 marked this pull request as ready for review July 9, 2025 15:05
Copilot AI review requested due to automatic review settings July 9, 2025 15:05
@redsun82
Copy link
Copy Markdown
Contributor Author

redsun82 commented Mar 11, 2026

Rerun has been triggered.

  • Rust Integration tests MacOS (all) 🚀

redsun82 and others added 10 commits April 2, 2026 12:07
This avoids reinstalling dists when nothing changed, by leveraging
bazel test's caching behavior with the existing -as-test target
variants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace `npx tsx`-based scripts with standard Python 3:
- `codeql-test-run.ts` → `codeql_test_run.py`
- `language-tests.ts` → `language_tests.py`
- `forward-command.ts` → `forward_command.py`

Uses `shlex.split` and `pathlib` instead of hand-rolled parsing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ition

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add justfiles for C++ following the pattern of other ported languages
(go, rust, swift). Move consistency queries from semmle-code's
semmlecode-cpp-consistency-queries/ to ql/cpp/ql/consistency-queries/.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use env("python_version", "3") so that when the parent just process
exports the variable, nested just calls (via language_tests.py) pick
it up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@redsun82 redsun82 requested review from a team as code owners April 2, 2026 12:31
redsun82 and others added 3 commits April 2, 2026 15:17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use f-strings instead of `+` concatenation, remove `set unstable`
(all previously unstable features are now stable), and add `[parallel]`
to swift `extra-tests` recipe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants