Test Infrastructure Audit

[rachitnigam]

It is time once again to audit the test infrastructure and figure out how to reorganize things as the monorepo continues growing. A couple of big questions:

• Should we replace runt with turnt?
• How can we define separate test suites for the "core" infrastructure distinct from "frontends"?
• How do we test features like synthesis, AXI-generation, and tests that require complex testbenches?

runt vs. turnt

The original motivation for runt was running tests in parallel. turnt has since caught up and now supports parallel execution. Separate organizational reasons to consider here:

1. turnt supports more complex tests that can produce multiple inputs and outputs
2. turnt overall has more person-hours invested in maintenance since it's used for CS 6120 and most other compilers in the CAPRA group.

The other two questions (separate test suites and complex test benches) also influence whether we should make this decision. I would lean on folks who've had experience with both systems (@sampsyo or @anshumanmohan) for more technical arguments for whether we should make this change or not.

Separate Test Suites

The current runt.toml file uses ad-hoc tags like [core] to allow users to filter out particular tests but this is not particularly ergonomic or elegant and doesn't scale with multiple test suites. Can we either:

1. Make runt support test suites so we can define subsets of tests to run?
2. Rely on turnt to support test suites (not sure if this is an out-of-the-box feature).

Complex Testbenches

The work on AXI demonstrates the long-term need for more complex testbenches that can define multi-cycle interactions with designs. Supporting these kinds of tests with runt is slightly annoying because you end up having to define a custom test suite for each test at which point we might as well be using turnt's CHECK commands.

[rachitnigam]

Another interesting possibility, enabled by fud2, is generating a ninja manifest with all the steps needed to build, run, and check all the tests. Then, we can just use the parallelization, caching, and other nice stuff within ninja to do the test management. Something to ponder @sampsyo!

[sampsyo]

Super interesting idea! Seems worth spinning out to see how it would look…

[sampsyo]

Really interesting thoughts! Yeah, seems like there are some interesting trade-offs here. Just a couple of things to keep in mind while we think about this more deeply:

• One big feature that Turnt is still missing (w/r/t Runt) is a way to specify the "default set" of tests to run. It works fine if you want to do, like, turnt tests/*.futil or whatever. But if your tests are scattered over many subdirectories and many filename patterns, constructing that glob to select them all is annoying. There is no way to have just plain old turnt run everything by default, or perhaps turnt -e something run all the tests for a given environment. I have actually started implementing an idea around this, but it is surprisingly tricky because it somewhat fundamentally in conflict with the goal of having configurations associated implicitly with test files based on paths. I can elaborate on this sometime, but it is really nuanced and would take a while to explain…
• One thing that's tricky in both Runt and Turnt is managing the two-dimensional "grid" of tests and commands. Everything's easy if you want to run all of N commands on all of M tests, for N×M total runs. But if you want to be careful about only some "treatments" applying to only some test files (so only a subset of the N×M "grid" is meaningful), that is annoying to accomplish IMO. It's complicated for different reasons in Turnt and Runt because the configuration goes in opposite directions, in a way. But because this is something we clearly want for Calyx, it's maybe worth doing some deep thinking about what the ideal way to deal with this problem would be.

[rachitnigam]

Thanks for writing this up! I've added discussing this as a possible topic for a Calyx meeting in the future. The glob pattern support is hopefully not too bad to implement in turnt (and might even be possible to define it so that it can parse runt.toml files?).

The matrix-test specification problem is more interesting. I think there is some notion of orthogonal definitions for test recipes and test files (similar to how ninja rules can define pattern variables which are separately defined/overriden by rules). Maybe there is something there?