Skip to content

Feature: Support native node:test runner#6020

Open
willfarrell wants to merge 7 commits into
stryker-mutator:masterfrom
willfarrell:feat/node-test-runner
Open

Feature: Support native node:test runner#6020
willfarrell wants to merge 7 commits into
stryker-mutator:masterfrom
willfarrell:feat/node-test-runner

Conversation

@willfarrell

Copy link
Copy Markdown

Has been tested across ~12 different OSS repos I maintain, working really well. I've see 2-7x speed perf improvements and 50% reduction in CPU use compared to command.

Closes #5421

AI disclosure: Initial PoC by Claude Opus 4.8, Reviewed/Edited/Tested by human

cc @m-ronchi @bsene @bausmeier

Signed-off-by: will Farrell <willfarrell@proton.me>
Signed-off-by: will Farrell <willfarrell@proton.me>
Signed-off-by: will Farrell <willfarrell@proton.me>
Signed-off-by: will Farrell <willfarrell@proton.me>
Signed-off-by: will Farrell <willfarrell@proton.me>
@nicojs

nicojs commented Jul 4, 2026

Copy link
Copy Markdown
Member

@willfarrell, thanks for your work and idea.

Have you tried the tap-runner? It was the recommended way of running Stryker with the node-runner. It is a small wrapper and might even be slightly faster. A small PoC can be found in the e2e tests.

If there is a reason why the tap runner doesn't work or why your approach is better, I would be happy to review and merge this PR. 🙏

Signed-off-by: will Farrell <willfarrell@proton.me>
@willfarrell

Copy link
Copy Markdown
Author

Thanks for taking a look! Yes, I tried the tap-runner first (including the tap-using-node-test e2e PoC). It works, but there are two structural reasons I ended up writing a dedicated runner:

1. Per-test granularity instead of per-file. The tap-runner's unit of work is the test file: test ids are file names, perTest coverage is keyed by file, and bail can only stop between files. This runner drives node:test through its programmatic run({ isolation: 'none' }) API with a root beforeEach hook, so Stryker gets real per-test ids, per-test coverage attribution, and per-test bail. Reports name the exact killing test (e.g. math - should recognize a negative number instead of test/is-negative.spec.js).

2. One process per run instead of one per file. The tap-runner spawns a fresh node process for every covering test file on every mutant, handing results back via TAP-on-stdout plus a JSON temp file. This runner forks a single child per mutant run, executes all covering files in-process, and streams structured results over IPC, which also enables mid-run hit-limit checks and an immediate process-group kill on first failure.

I benchmarked both on the same projects (same machine, concurrency: 1; both runners produced identical mutation scores):

Project tap-runner node-test-runner
tap-using-node-test e2e (4 test files, 14 mutants) 3.4s 2.2s
20 test files sharing one mutated module (11 mutants, 4 survivors) 9.0s 1.8s

The gap grows with test-file count: every surviving mutant costs the tap-runner one Node startup per covering file, while this runner pays it once per mutant.

Why not extend the tap-runner instead? I considered it, but every feature above requires in-process node:test hooks and IPC rather than TAP-over-stdout. perTest coverage in particular is impossible from outside the process: something inside has to set the current test id on the Stryker global while coverage counters accrue, and TAP is a one-directional after-the-fact text stream. Retrofitting would have meant embedding a complete second runner inside the tap package behind a mode switch, while compromising what makes the tap-runner valuable as a simple, framework-agnostic fallback.

Happy to walk through any part of the implementation, thanks for reviewing! 🙏

PS just ran an audit using Fable 5, and added in support foe skip. Good to review.

…nner

Signed-off-by: will Farrell <willfarrell@proton.me>
@nicojs

nicojs commented Jul 15, 2026

Copy link
Copy Markdown
Member

Cool, thanks for your explanation. I now understand that this is a big improvement over the current tap-runner. I'm on vacation atm, will have a look when I get back (or @Andreas02-dev reviews this one).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native node:test runner support

2 participants