The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:
- Catch definitely wrong behaviour.
- Allow comparisons of fidelity between builds and simulators.
Proposal:
- Make the pass criterion for the tests quite lax, so that FAIL really means that something went wrong.
- Produce a 'standard' report output from the validation tests. This would follow a well defined schema and contain a list of derived values of interest, their pass thresholds (and less than, greater than, etc.), and metadata for sim and nsuite version.
- Manually running the validation tests in a new 'verbose' mode would output a human- and script- readable version of this data.
- Reports from multiple tests in the same validation run should be able to be combined and presented in a form that is easy for programmatic consumption, e.g. for presentation in Jenkins or other tools.
The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:
Proposal: