Skip to content

Richer validation output #79

@halfflat

Description

@halfflat

The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:

  • Catch definitely wrong behaviour.
  • Allow comparisons of fidelity between builds and simulators.

Proposal:

  • Make the pass criterion for the tests quite lax, so that FAIL really means that something went wrong.
  • Produce a 'standard' report output from the validation tests. This would follow a well defined schema and contain a list of derived values of interest, their pass thresholds (and less than, greater than, etc.), and metadata for sim and nsuite version.
  • Manually running the validation tests in a new 'verbose' mode would output a human- and script- readable version of this data.
  • Reports from multiple tests in the same validation run should be able to be combined and presented in a form that is easy for programmatic consumption, e.g. for presentation in Jenkins or other tools.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestvalidationRelates to nsuite validation component

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions