Compiler and runtime tests

Deprecated language features: reusable and reusable[]. Use slot_pool<T> for slots and slice_pool<T> for contiguous runs. Builtin behavior and historical examples below remain relevant to existing code; removal is planned separately, after the remaining migration gaps.

Running the suite

The deterministic language server test sends saved call hierarchy items through Server::Handle directly after a document change. No background analysis runs between the messages. Build and run it with:

cmake -S . -B build-lsp -DGOOSE_BUILD_LSP_TESTS=ON
cmake --build build-lsp --target goose_lsp_test --config Debug
ctest --test-dir build-lsp -C Debug -R lsp_call_hierarchy --output-on-failure

Run python test/run_tests.py --profile baseline --cc native --require-clang after a normal compiler build. CI runs this on Windows, macOS and Linux.

The runners execute -j/--jobs jobs at once, defaulting to one per logical processor up to 32. Their threads mostly wait for Goose, the C compiler, and the programs they start. Each job buffers its output, and those buffers are printed in job order. Logs therefore stay comparable regardless of the number of jobs. -j1 runs everything in order on one thread.

Runs of the same fixture are sequential because one run may write a file that another would see (stdlib_os.goose, for example). The Goose-in-Goose chains and sample runner start first. The sample runner (samples/run_samples.py) is a separate process using the same number of jobs. The -O0 bootstrap chain is the longest job, and the whole suite takes about as long as that chain.

--gpu-jobs N limits concurrent gfx programs within each runner for GPU drivers that cannot handle many headless devices. There is no limit by default.

The log is divided into sections (parse, roundtrip and typecheck; generated C; JIT runs; errors_tc/; samples; ...). Each section ends with a line counting its checks, such as == typecheck errors (...): 1668 ok. Anything a section prints other than its ok lines is shown in place, under the section's title: a failure together with what led to it, and the skips. -v also prints a line for every check that passed. Under GitHub Actions, those ok lines are kept in a collapsed group per section, and the group is closed before anything else prints. After the log, the FAIL lines are repeated with the titles of their sections; under GitHub Actions they are printed as annotations.

Each fixture is compiled as few times as its checks allow. One run of goose --roundtrip --check parses the fixture, checks that its dump parses again to the same dump, and typechecks it. That run adds --bce-test where the fixture has annotations, uses --parse instead of --check for a parse-only fixture, and adds --dump-file to write the dump for a dump-runtime fixture. Each run has --stats, with which every run that gets past resolution prints parsed ok, so a single --check of an errors_tc/ fixture can tell a parse failure from a typecheck failure. Where an optimization level both builds the C and runs the program through TinyCC, one goose -O<n> --jit -o <file> does both: it writes the C, then runs the program in-process.

Most of those runs, and the runs of the errors/ and errors_tc/ fixtures, share a compiler process with other fixtures. goose --multi-test <flags> a.goose b.goose ... compiles each file in turn, exactly as a separate run on that file would. It ends each file's part of stdout and of stderr with a line ==== goose --multi-test: exit <code> <file>. From these lines the runner gives every fixture exactly the exit code and output that a run of its own would have produced. Starting a compiler process takes about as long as checking a small fixture, so a batch of 20 is checked in a tenth of the time. If a batch crashes or aborts, the results for the files before the one it was compiling are kept; that file and the rest are then run one per process, so the crash is reported for the right fixture. Runs that write a dump (dump-runtime), use runtime-define defines, or run a program (TinyCC) still use one process each.

The C of the fixtures that do not link a native layer is also built a batch at a time, at each optimization level and for the dump and GS_DEBUG programs. One C compiler run compiles each program's unit with its main renamed, together with a main that runs the program named by its first argument, and links them once against the runtime object (CC.compile_programs in scripts/toolchain.py). Every run of a program is still a separate process. Where a batch does not build, each program in it is built on its own, which shows why it failed. Starting and linking take most of a C compiler run's time for these programs, so a batch of 20 builds in a fifth of the time. --batch N sets how many files go to one compiler process and how many programs go into one executable (by default, a number that leaves every job some batches); --batch 1 gives each file its own process and each program its own executable.

Fixture layout

Test fixtures are grouped by category; run_tests.py stays at the root of test/:

Positive fixtures are discovered one level below test/; nested import helpers run through their entry programs. goose_in_goose/ is handled separately as one bootstrap program, rather than discovering its modules as standalone fixtures. Keep fixture stems unique across categories, since expectations and generated build artifacts use those names. The runner rejects duplicate names.

Configurations and backends

Run just the bootstrap with python test/run_tests.py --goose-in-goose-only --exe <goose> --cc native. The usual --profile, --no-jit, and --nocgen options apply. With no native C compiler, it still checks the source and, when available, self-compiles through the JIT, but reports the native chain as skipped. Sources and the exact stage contract are in goose_in_goose/README.md; artifacts and logs go in build/gen/<profile>/goose_in_goose/O<level>/.

The sanitizer job builds Goose with -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -fno-sanitize-recover=all, then runs python test/run_tests.py --exe <instrumented-goose> --profile sanitize --cc clang. Generated C uses native -O1, with UBSan's alignment check included for packed accesses as well as for the C++ compiler. ASan observes C allocations, but does not know logical object boundaries inside Goose's custom virtual-memory arenas.

Generated C is built as goose -o writes it, against the separate runtime (docs/implementation.md §7). GooseRuntime (scripts/toolchain.py) has the compiler under test write the runtime's C (--emit-runtime) once per run. It compiles that C once per toolchain configuration (compiler, optimization level, defines and extra flags, so the sanitize profile's runtime is instrumented as well) into build/gen/runtime/, or for the samples into their own directory. CC.compile(..., runtime=...) then links that object with each program built this way. Stages 2 and 3 of the bootstrap contain their own runtime and link none. Every TinyCC run compiles the single-unit form, which --standalone writes; the benchmarks build that form too.

The direct test/threads/runtime_threads_lifecycle.c regression checks allocations, mappings and Windows handles across worker churn, including unjoined workers, concurrent/repeated waits, children outliving their parents and contained workers that abort; and that gs_rt_stop ends the main program with every allocation and mapping released and the fault handling restored, after which a program starts and ends again. The host_exit task builds test/threads/host_exit.goose for its C host, host_exit.c, which starts, uses and ends the program three times over, each round over fresh globals and queues, and once ends it while a worker still runs, which is a runtime error. The direct test/runtime/runtime_trim_memory.c test checks that releasing unused data-stack memory decommits the pages on Windows and drops them from the resident set on Linux and macOS, while keeping live partial pages and other stacks. It also checks that discarded pages are zeroed, that Windows growth recommits through the fault handler behind a commit mark that each trim moves, and that macOS growth needs no runtime-handled faults, before or after a trim. Both run in both profiles. The normal suite also checks user-visible worker error diagnostics.

The gfx/ tests exercise the gfx package (https://github.com/aardappel/goose_gfx, its docs/gfx.md). They always parse, typecheck and generate C. When the compiler includes the gfx layer (configured with the package, GOOSE_BUILTIN_LIBS), goose build links the tests against it and they run at -O0 and -O2 and through TinyCC. They render headless into textures and read back only pixel-aligned results, so their output is the same on every backend and GPU. A machine with no GPU device (where the program prints gfx: no GPU device) reports them as skipped, as does a compiler without the layer. Linux CI runs them on Mesa's lavapipe. The runners set GOOSE_GFX_HEADLESS=1, so no test or sample opens a window. The hidden --compile-shader flag is tested on gfx/probe.frag and gfx/probe_badset.frag in every compiler, with or without the layer (Additional compiler checks).

The physics/ tests exercise the physics package (https://github.com/aardappel/goose_physics) in the same way. They always generate C. When the compiler was configured with the package (GOOSE_BUILTIN_LIBS), goose build links them and they run at -O0 and -O2 and through TinyCC. Box3D is deterministic across platforms and worker counts, so the tests print rounded physics results and compare them exactly. Together they call every function of the layer.

The ui/ tests exercise the ui module of the gfx package (its docs/ui.md) in the same way. goose build links the ui layer, and for the test that renders through gfx, the gfx layer too. A test needs the layers of its own category and of every module it imports, and is skipped where one of them is not built into the compiler. The tests drive a context with their own input and print what comes back and what was drawn; ui_render draws through headless gfx and reads back pixels only where the ui drew solid color. Together they call every function of the layer.

Each package's own test/api_check.py compares the functions, structs, and constants in its modules with their C headers. This catches interface mismatches that can compile successfully on both sides. It also rejects a struct the layer passes by value that TinyCC would pass differently from the C compilers on System V x86-64: a struct of at most 16 bytes that has a misaligned field, or that has only floats in one eightbyte and integers in the other.

The gfx package's own test/window/run_window_test.py (in https://github.com/aardappel/goose_gfx) is run by hand, on a machine with a display and a GPU. It uses the scripts/toolchain.py of a Goose checkout (the sibling goose directory, or whatever GOOSE_REPO names). It runs the showcase gfx_showcase.goose with a window, both in-process and built by every C toolchain found, on each GPU driver of the platform. It checks what the program reports about its own readbacks, and decodes the screenshots the program saves (under the package's build/gfx-window/) to check them for detail and compare them across runs.

The TinyCC runs are skipped under the sanitizers, because there the program runs uninstrumented inside an instrumented compiler and still holds its runtime allocations when that compiler exits. A test the backend cannot run yet is reported as a skip, not a failure: either because the compiler refuses the program (JIT mode does not support) or because the test's first line says no-jit. Prefer portable assertions to skipping a backend; the math tests allow small rounding differences in transcendental results. See docs/design/jit_backend.md for the backend's remaining limitations and possible solutions.

Expected output and diagnostics

An explicitly requested compiler must exist; CI does not silently skip it. Every runnable fixture requires an expected/<name>.out, including an empty file when the program should be silent. Expected aborts additionally require .aborts and a nonempty .stderr containing the expected runtime diagnostic substrings. Sanitizer reports fail even when a program was expected to abort. Resource-bound fixtures can set a limit in their first line, such as // runtime-define: GS_MAX_STACKS=1. The runner passes it as a -D to every compile: the compiler checks GS_MAX_STACKS against the program's static stack count, and the generated C takes care of the rest. Fixtures that use this:

• append_destination_storage.goose detects temporary result stacks that output-only comparisons would miss.
• cycle_scratch_locals.goose shows that a recursion's scratch locals use the same few stacks at every level.
• discarded_resizable_results.goose shrinks each stack's reservation (GS_STACK_RESERVE) so that its loops overflow a stack unless every discarded result is released. guard_flat_storage.goose does the same for what a guard's condition, its else and the rest of its block use on the data stacks (the rest of the block is emitted without braces of its own). continue_restores.goose and large_fixed_exits.goose do the same for the watermarks that a continue, a break and a return ... from restore.
• thread_cap.goose shrinks the address space budget (GS_STACK_BUDGET) so that hardware_threads() reports its minimum of one wherever it runs.
• stack_stats.goose has its program report its stack use (GS_STACK_STATS), which is checked through expected/stack_stats.stderr.

cycle_static_stacks.goose runs the shapes that a call into a recursive cycle allows (§7.8); the errors_tc/cycle_temporary_*, cycle_construction_* and cycle_result_* fixtures are the rejected shapes. Codegen fails with an internal error if any program holds a data stack across such a call. The runner also checks the shape, but not the counts, of the --stacks report of thread_cap.goose and cycle_scratch_locals.goose. For thread_spawn_discarded.goose, whose worker holds no regions, it checks that the report gives no thread cap. A fixture expecting a debug-only abort sets GS_DEBUG=1 the same way, which makes every run of it a debug build: cast_abort_location.goose and overflow_abort_location.goose check the location that a failing check reports, and thread_contained.goose checks that the debug checks end only the contained worker.

Parser/resolver errors live in test/errors/; semantic errors live in test/errors_tc/ and must get past parsing and resolution (parsed ok) before they fail. Each source declares one or more // error: <diagnostic substring> lines. The compiler must exit with 1, and every marker must occur in one of these: diagnostic headers, in const for expansion context lines, the instantiation chain's in fn lines (without their instantiated from <path>:<line> part), or the reasons on overload candidates' lines. Echoed source does not count. The compiler continues past the errors it can recover from, so a fixture may print more errors than it declares; test/errors_tc/recovery/ holds the fixtures that declare all of their errors, in order (see the table above). In a marker, use the specific rejection reason and relevant types/roots; omit source paths, line numbers and specification section numbers. errors_tc/function_value_chain.goose names a block's file, line and column on purpose, because what it checks is how the chain names a block. These inline assertions replace the old compiler-error .stderr files.

A fixture declares the warnings its --check prints in the same way, with one // warning: <message substring> line per warning; a positive fixture that declares none must print none. Markers and warning lines are paired one to one, and each marker must be part of the message of a different warning line. So a warning printed twice fails the fixture, just as a missing or an undeclared warning does (a warning is printed once for its node as written in the source, docs/implementation.md §3.12). Two &s on one line that produce the same warning print two identical lines and need two markers, as r6 in branch_reference_copies.goose does. The order of the markers does not matter. Without line numbers, a marker could not say where a warning is, and warnings print in the order in which checking reaches their nodes: a callee's body at its first call, a loop's or a construct's once its checks settle, and redundant casts by line after the whole program. A change to the checker may rearrange that order without changing what a program is told. The warnings of a rejection test are checked only if it declares some: checking stops at its error, so which warnings are printed first depends on how far checking got. For example, cycle_local_store.goose warns about its recursive call's &local before the error that the call reveals, at the store on the line above.

Additional compiler checks

The runner checks more than exit status and runtime output:

Language server protocol checks

lsp_protocol.py first runs --probe. A client in the runner then initializes the server with test/lsp/basics.goose as its entry file and the VS Code extension's commands. It opens the file, waits for diagnostics (there should be none), and sends the queries specified by its // lsp: lines. These check:

• Hover text; definitions at a position, in a file, or absent; document symbols; and signature help with the active parameter.
• Semantic token types at a position, including positions with no token; reference counts; and renames that either make the expected number of edits or are refused. Each rename edit must change only the name's leaf.
• Inlay hint text, position, and count per line. Defaults inserted by the checker must produce no hints. Folding ranges must cover the expected lines, and selection-range chains must contain the requested range and be nested.
• Code-lens positions and titles. An instantiation lens must have a count matching its data lines and the command that lists them. A "Run" lens must carry the file's URI; "not instantiated" must be a title alone.
• Completion labels that must be present or absent, and text an item's detail must hold. complete.goose ends with p., as source often does when an editor requests completion.
• Incoming and outgoing calls, including the function at the other end and the call lines. Uncalled functions must have no call results, and names that are not functions must have no call item.
• A workspace symbol from the imported helper.goose.

The test also checks that analysis asks the client to refresh code lenses, semantic tokens, and inlay hints. It then checks the following changes and additional roots in order:

1. A burst of edits introduces an error and removes it. Diagnostics must reflect the latest text, and a change must cancel an interrupted analysis. The test then inserts a line at the top and immediately requests a hover at the shifted position. The answer must use the new text. An incremental edit (a range plus its fragment, as a client with incremental sync sends) inserts a comment line above the file: the hint below it must stand, shifted down a line.
2. recovery.goose and broken_decls.goose are opened as separate roots. Every error must be published, and queries must still receive answers. Document symbols requested immediately after opening recovery.goose must come from its analysis, rather than return an empty result.
3. source_index.goose checks that global initializers are indexed as written, even though the checker rewrites them in place. Bindings in const for arms must resolve, and signature help must ignore commas in literals and comments. ufcs_ns.goose checks that signature help, and hover and definitions on a resolved call and on one that matches no overload, find a UFCS call's functions in the namespace of its receiver's type, not the global ones.
4. errors_tc/recovery/inside_braces.goose checks errors inside struct-literal and match braces. They must appear at their source locations, with no extra errors.
5. exemplar.goose and errors_tc/recovery/exemplar_unchecked.goose check queries on uncalled generic functions through their exemplars. In the second file, those bodies must produce no diagnostics even though no binding could make them correct. Its only error is in main, as reported by goose check.
6. Roots are opened, closed, and edited. After a request runs the due analyses, the test checks the last diagnostics published for each file. It opens roots_lib.goose, which contains an error, alone and then alongside its importer roots_app.goose. Closing that importer must leave hover and document symbols available again. Next it opens both roots_app.goose and roots_app2.goose as importers, then edits the first to stop importing the library. The library's error must appear at every step, only once when both roots import it. Separately, helper.goose stays open while basics.goose closes and ceases to be an entry file. The helper's hover must then use its own root analysis.

The first server shuts down cleanly. A second client starts with two_a.goose and two_b.goose as entry files. It renames a function in their shared import, two_shared.goose, then requests references and incoming calls. Results must include both programs, with each location listed once. They must also include callers of a function reached only by the second program's instantiation of a generic function. The second server then shuts down.

Run test/lsp_sweep.py by hand; it is separate from the main runner. It measures goose lsp robustness and latency across every sample, the goose_in_goose bootstrap, every stdlib module, and all positive, errors/, and errors_tc/ fixtures.

For each program, it opens a separate root and waits for diagnostics. At every identifier start and after every ., it requests hover, definition, completion, and signature help, plus incoming and outgoing calls for any function named there. It requests semantic tokens, symbols, folding, inlay hints, and code lenses once per program. It then makes ten random one-character edits in a burst and waits for diagnostics. After restoring the text, it checks that diagnostics match the initial ones.

The sweep fails if the server exits, returns an error, takes over 2 s for a request, or stops responding. It prints the slowest files and requests. --quick checks a handful of files in a second or two. The baseline is in docs/design/language_server.md §12.

Fixture audit

A fixture is load-bearing when some regression would fail it and nothing else. Two fixtures with similar diagnostics are not necessarily duplicates: they can exercise different expression visitors, specialization/cache states, root propagation, or generated layouts. Whole-program globals and call graphs are also part of many regressions, so combining two fixtures can change the property under test.

The October 2026 audit measured this. It ran the suite with a compiler built with clang's -fsanitize-coverage=trace-pc-guard and with an instrumented runtime object, and recorded, for every fixture, the compiler edges its runs hit and the runtime lines its programs ran. Coverage was counted only toward fixtures that would notice the same regression:

• a positive fixture's coverage toward other positive fixtures (a rejection does not notice the checker refusing a good program);
• a bce-annotated fixture's coverage in bce.h toward other annotated fixtures;
• a rejection's coverage in the checker's files toward other rejections of the same rule (a program accepted by mistake goes unnoticed by positive fixtures and by other rules' rejections).

No fixture's coverage was fully contained in that of any single other fixture. Even when coverage overlaps, a fixture may catch a failure that others miss. A fixture covered entirely by the rest of its class was therefore removed only if all of these conditions held:

• Named fixtures checked the same behavior through the same construct.

• It had not been added later as a regression test for a bug those fixtures missed.

• It checked nothing that coverage alone cannot show, such as an expected runtime diagnostic, an integer width or extreme value, a GS_DEBUG check, or output text.

A second review challenged every proposed removal. All removals together also had to preserve each class's coverage.

This removed 153 errors_tc/ fixtures and two positive ones. Most of the removed errors_tc/ fixtures tested a shrink, store or provenance rule reached through the same construct as a sibling. The two positive ones were threads/thread_spawn_bound.goose, whose generic thread_fn bindings thread_spawn_modules.goose spawns through the same generics with checked output, and typing/fill_counts.goose, whose computed fill counts named_const_readonly_refs.goose, appended_literals.goose and const_name_scopes.goose build and print. Nearly every positive candidate stayed: most abort with a diagnostic that no other fixture's .stderr pins, or pin values, types or output that run the same compiler edges. Small fixtures on one subject were merged. Each original fixture became a function of its own, called in the original order, and the expected output is their outputs concatenated:

The audit also compared each rejection's comment with the error it reports. Where a later rule caught a fixture first, the fixture was changed to reach the rule it names. Where its old program was the only test of the path it did reach, that program was kept as a separate fixture named for that path. For example, the growshrink_reach_* fixtures hold a second root, so that the caller's storage only bounds what they store, beside the growshrink_exact_* ones, whose holders' contents are known exactly; index_type_let_array.goose and index_type_fixed_array.goose pin the two halves of the index-type array rule; and the fnval_* fixtures are named for the function-value rule they reach. Markers that any rejection of their kind would match (cannot clear, read-only, a type name) were replaced by the specific message. Markers may also match the instantiation chain's frames and an overload candidate's reason, which some fixtures exist to pin.

The small return_from_cycle.goose remains beside return_from_cached_call.goose: the former checks replay through a recursion back edge without an additional cached-call path, which could otherwise mask a missing propagation step.

CI uses four jobs to cover these configurations without testing every combination of platform, sanitizer, optimization level, and runtime mode. Benchmarks run separately from CI.