Internal test inventory and the internal-tests.sh runner#

Scope: every project-internal test asset — the ones that check this repository's own code against its own expectations, not against a W3C test manifest. The W3C conformance suites (RDF, SPARQL, OWL, SHACL, CSVW, JSON-LD, XSLT, ...) already have a runner (w3c-tests.sh / formal/fstar/generate-report.sh for the F* tree; per-suite dispatch via tools/dispatch_test_suites.sh for suites whose manifest spec: field is a real W3C/OGC/IETF URL) and a score. This document and tools/internal-tests.sh cover what did not have either: internal regression scripts, Lean corpus probes, and the Node test suites.

Method: read the tree at the date of this note (2026-09-03, claude/main, darwin-arm64), plus one real run of every suite found. Where a suite could not run in this environment, the reason is stated; where it ran and failed, the cause is stated to the depth the task allowed (root-caused where cheap, named as an open question otherwise).

Reproduce with tools/internal-tests.sh --list (derives the same list from the tree) and tools/internal-tests.sh (runs it).

Result, this run#

internal-tests TOTAL: 70 pass, 40 fail, 16 skip (out of 126)

--quick (60s per-suite budget): 68 pass, 40 fail, 18 skip (out of 126) — the two extra skips are lean-probe-l4rdfs-semi (92s) and w3c-persisted-census (54s), both of which pass when given the full budget.

shellcheck is not installed in this environment; the shellcheck tools/internal-tests.sh gate could not be run.

Categories#

Total suite count this run: 126 (from tools/internal-tests.sh --list, 2026-09-03). This number moves as scripts are added or removed; treat the count in this document as a snapshot, and --list as the source of truth.

Findings#

Ordered by how many suites they explain, most first. None of these were fixed in this landing; each is either a filed defect (VC. GitHub issue numbers are named where one already exists) or a plain environment gap.

1. Wrong-platform binary at formal/fstar/ocaml-output/<tool> (17 of 52 tests/local/*.sh, plus local-graphs-api, local-jsonld-regressions, local-parquet-footer-version-gate, local-parser-unicode, local-sparql-parser, local-turtle-pretty, local-serializer-unicode, rdf-mt-generated). CLAUDE.md iron rule #9 states these symlinks "point at the current platform's bin/ dir". In this darwin-arm64 checkout they were committed pointing at bin/linux-x86_64/* (git show HEAD:formal/fstar/ ocaml-output/factoidal → ../../../bin/linux-x86_64/factoidal, an ELF binary; running it gives Exec format error). tools/sandbox-bootstrap.sh has the correct repair logic (lines ~73-79) but only acts if [[ ! -e ... ]] — it will not repoint a symlink that already exists and is simply wrong for this platform. Repointing the 11 of 12 affected symlinks that have a bin/darwin-arm64/ counterpart (working-tree only, not committed — see below) turned six previously-failing suites to PASS in this run. bin/darwin-arm64/factoidal-dump-nq is missing outright (no darwin-arm64 build of that consumer tool is committed), so local-serializer-unicode still fails for that reason alone. Separately, tools/negative-test-vacuity.py and tests/local/rdfs_schema_split_regressions.sh / turtle_undeclared_prefix_regression.sh hardcode bin/linux-x86_64/factoidal directly rather than going through the symlink, so repointing the symlink does not fix them — 17 of the 52 tests/local/*.sh scripts hardcode a linux-x86_64 path this way. Not fixed here: the committed symlink target is a CI-vs-local trade-off (CI runs ubuntu-latest), not a one-line "correct" value, and the hardcoded paths are a separate, larger cleanup.
2. _tmp.junk/pycottas-venv/bin/python not provisioned (16 of 52 tests/local/*.sh, plus local-backend-parity, local-backend-parity-full, local-cottas-ask-decode-failure, local-cottas-corpus, local-cottas-groupby-counts, local-cottas-row-order, local-graph-default-semantics, local-parquet-footer, dict_global_cache_parity, durable_update_stage3, and others). The session bootstrap hook provisions this venv; a worktree built only from tools/ensure-test-env.sh (this task's instructed setup) does not get it. Environment gap, not a defect — same root cause across all of these.
3. Query-plan open-mode descriptor strings have drifted from three blockengine-*-smoke.sh assertions. blockengine-ibk3-w3c-disk-query-smoke.sh expects open-mode=ibk3-paged-merkle-full-manifest(3) and gets open-mode=ibk3-sri2-tli1-subject-point(3); blockengine-sbm6-synthetic-smoke.sh expects open-mode=ibk3-sri2-tli1-oli2-object-subject-join(2) delta=base and does not find it in the actual output; blockengine-shard-merkle-scan-smoke.sh's final grep -q 'open-mode=predicate-selective-merkle-limit-prefix(1)' does not match either. Rows returned are correct in all three; the query planner now names its access path differently (a new subject-point/join index mode superseding what these scripts still name). Test staleness against a planner improvement, not a correctness regression — needs a decision on whether to update the three scripts' expected strings.
4. blockengine-shard-session-smoke.sh checks for store/manifest.sbm1; l4block-shard-pack now writes manifest.sbm2. Confirmed by running l4block-shard-pack directly: it reports format=predicate-ibk2-merkle-v2-streaming ... manifest=manifest.sbm2 wire-version=2. The script's test -s .../manifest.sbm1 assertion is checking a filename the packer stopped writing; per shardborough-storage, a wire-format version bump should update every consumer, and this script was missed.
5. tests/unit/run-all.sh fails at the ocamlfind ocamlopt build step for all 50 files with line 262: TEST_CMX[@]: unbound variable, even though TEST_CMX=() is set immediately before the loop that populates it (line 242) and bash 5.3 (confirmed on this host) does not reproduce the empty-array-under-set -u bug that affected older bash. Root cause not found within this task's time box — it is a harness bug (every file fails identically at the build step, before any file-specific compilation is attempted), not 50 independent regressions. Needs its own investigation.
6. tests/known-defects/run.sh: 3 of 5 tracked defects no longer reproduce (XPASS), 2 probes errored. XPASS on #324 (sameTerm case-folding), #334 (Turtle silently drops undeclared-prefix statements — now exits 126), and #275 (syntax-error message). ERROR on #336 (probe produced no rows — query or binary changed) and #337 (BGP itself did not match; probe invalid). The suite is designed to fail loudly on exactly this signal (its own header: "not automatically good news ... either somebody fixed it ... or the probe has drifted"). This needs an owner decision per issue, not a fix inside this task.
7. tests/local/cottas_corpus_regressions.sh is missing the executable bit (-rw-r--r--, siblings are -rwxr-xr-x). A direct exec ("$runner" args, the pattern tools/dispatch_test_suites.sh- style dispatch would use) fails with Permission denied; internal-tests.sh invokes every runner via bash "$runner" for this reason, which works. Not fixed in the tree (a one-line chmod +x, left for a dedicated hygiene commit).
8. local-check-pages-links needs docs/_site built (cd docs && npx @11ty/eleventy --output=_site) and local-sparql-negative needs a Jena ARQ checkout at /tmp/jena/jena-arq/testing/DAWG-Final — both external build/ fixture prerequisites, not provisioned by tools/ensure-test-env.sh. Environment gaps.
9. l4owl-probe, l4xslt, l4grddl fail with known, already-scored gaps, not new regressions: l4owl-probe reports TOTAL: 1131 pass, 316 fail, 2 skip, 8 unsupported (out of 1457) — closure gaps already tracked in the probe's own per-catalog breakdown against docs/test-results/latest.json; l4xslt reports 84 pass, 3 fail (out of 87 decided); l4grddl reports 19 pass, 22 fail (out of 41 decided), of which 7 failures are stated upstream http/https vendoring drift and 17 cases are unavailable (documents not fetched, by design — the runner makes no network request).

What the runner does NOT cover#

• The W3C conformance manifests (w3c-tests.sh, formal/fstar/generate-report.sh, and every .github/test-suites/ *.yaml whose spec: is a real spec URL) — those have their own runner and score already.
• tests/perf/* — a benchmark, not a pass/fail gate (skills/perf-benchmarking).
• tests/vc-di-eddsa/run.sh and tests/vc20-api/run.sh — registered in the W3C-suite dispatch registry (spec: is a real w3c.github.io test-suite URL), already excluded for the same reason as the conformance manifests. Both also need npm install in their vendored submodule directories, not run in this environment.

CI proposal (not wired in this commit)#

Add a internal-tests-quick job to a new or existing workflow, gated the same way verify-lean4.yml is (push to claude/main, PR, workflow_dispatch):

  internal-tests-quick:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { submodules: recursive }
      - run: bash tools/ensure-test-env.sh
      - run: bash tools/internal-tests.sh --quick

On ubuntu-latest the bin/linux-x86_64/* symlinks are already correct, so Findings 1 and most of Finding 2's provisioning gap would not appear there — CI would measure closer to the suites' real pass/fail than this darwin-arm64 run did. Left as a proposal, not landed, per this task's scope.