Feature request: Multiple runs of specific tests

[osiewicz]

Motivation

Over at Zed we'd love to have nextest have more opportunities for parallelization of our tests execution.
For context: Zed uses a home-grown wrapper around test executions. That wrapper lets us write asynchronous test code.
Crucially, since we own the executor for our GPUI framework, the tests can and do exercise. different execution orders via iterations parameter (example) - on each execution our test executor is seeded with a different value, which leads to different sequencing of events. This then lets our tests execute greater surface area of our app. We do find bugs that happen only on certain iterations of tests.

The problem with that is that all iterations are executed within execution of Rust test. This creates a long pole in test execution for us.

Proposal

We would like to integrate more deeply with nextest and have it execute each iteration in parallel. To do that, we would move from defining the iteration count in source code (as a parameter to gpui::test macro) to defining it in nextest config.
In order to make the switch, we would need something like retries option of nextest, but one that keeps going even if the tests are successful. Then, we could use the value of NEXTEST_ATTEMPT (or an equivalent of that) as a seed of a test.

Put another way, we would like to have an ability to run stress tests on a test-by-test basis and have some kind of consistent identifier (a monotonically increasing integer would do) for use as a seed. We don't need time-period-based limiting that nextest currently supports for this.

Alternatives

No response

Additional context

No response