Added preliminary validation plan, added screenshots of the default dashboards

Author: bevalentin

docs/design/validation.md

@@ -0,0 +1,120 @@
+# Validation Plan
+
+## Introduction
+
+This validation plan defines the scenarios that must be executed to verify that the Application Quality Service, resulting from the work on the Application Quality BB, is behaving as expected.
+
+## Test Cases
+
+### User Authentication and Access Control
+
+This test case verifies the ability for users to access the Application Quality Service.
+
+#### Scenario: Access as anonymous user
+
+This scenario verifies the following:
+
+- Anonymous users visiting the Application Quality Web Portal:
+  - may see the home page, including the links to the user manual, the dashboards and the GitHub repository
+  - may access the login form
+  - may access the list of available analysis tools and display their properties
+  - may not access the list of available pipelines, nor the executions and reports
+- Anonymous users may not access the Application Quality Dashboards
+
+
+#### Scenario: Access as authenticated user
+
+- Users may login using the EOEPCA IAM BB (if integrated) to access the Application Quality Web Portal.
+- Users may login using the EOEPCA IAM BB (if integrated) to access the Application Quality Dashboards.
+- Users authenticated in the Application Quality Web Portal:
+  - may access the list of default analysis pipelines
+  - may define and edit custom analysis pipelines
+  - may access their custom analysis pipelines
+  - may not access analysis pipelines created by other users
+  - may execute analysis pipelines, monitor their execution and access the execution reports
+- Users authenticated in the Application Quality Dashboards:
+  - may access the dashboards displaying information about the pipeline executions, the execution-specific dashboard and the report-specific dashboards
+
+
+### Analysis Pipelines Creation, Editing and Deletion
+
+This test case focuses on the ability for authenticated users to create, edit and delete custom analysis pipelines.
+
+
+#### Scenario: Creation of Customer-Defined Pipeline
+
+This scenario verifies that an authenticated user may create new analysis pipelines.
+
+- The user must be able to select multiple analysis tools.
+- The user must be able to provide a default value for each analysis tool parameter.
+- The customer-defined pipelines must persist after a page refresh or a logout/login cycle.
+
+#### Scenario: Editing of Customer-Defined Pipeline
+
+This scenario verifies that an authenticated user may edit the analysis pipelines they have created.
+
+- The user must be able to add and remove analysis tools from the pipeline definition.
+- The user must be able to modify the default values.
+- The modified customer-defined pipelines must persist after a page refresh or a logout/login cycle.
+
+#### Scenario: Customer-Defined Pipeline Deletion
+
+This scenario verifies that an authenticated user may delete analysis pipelines they have created.
+
+- The user must be able to delete a pipeline they have created.
+- The user must not be able to delete a system-defined pipeline (provisioned by default).
+- The deleted pipelines must not re-appear after a page refresh or a logout/login cycle.
+
+### Manual Execution of Analysis Pipelines
+
+This test case focuses on the ability to execute analysis pipelines manually, via the Application Quality Web Portal.
+
+#### Scenario: Analysis Pipeline Execution
+
+This scenario verifies that an authenticated user may execute analysis pipelines using their own parameters.
+
+- The user must be able to select an analysis pipeline, enter execution parameters, and request its execution.
+
+#### Scenario: Analysis Pipeline Execution Monitoring
+
+This scenario verifies that an authenticated user may monitor the pipeline executions they have requested, and access the generated reports.
+
+- The user must be able to list the ongoing and past pipeline executions.
+- The user must be able to visualise the pipeline execution details, including the input parameters and the execution times.
+- The user must be able to visualise the resources consumed by a pipeline execution, after its completion.
+- The user must be able to visualise the generated analysis reports, after the completion of the pipeline execution.
+
+
+### Visualisation Dashboards
+
+This test case focuses on the access and use of the Application Quality Dashboards component.
+
+#### Scenario: Navigation from the Web portal
+
+This scenario verifies that an authenticated user may navigate from the Application Quality Web Portal to the Dashboards.
+
+- The user must be able to navigate from the Web portal to the Dashboards home page.
+- The user must be able to navigate from the Web portal to a dedicated pipeline execution dashboard.
+- The user must be able to navigate from the Web portal to a dedicated analysis report dashboard.
+
+#### Scenario: Default Visualisation Dashboards
+
+This scenario verifies that an authenticated user may access the default dashboards to visualise pipeline executions data and analysis reports.
+
+- The user must be able to access a dashboard that provides statistical information about the past and on-going pipeline executions.
+- The user must be able to list the past and on-going pipeline executions.
+- The user must be able to access a dashboard that provides information about a specific pipeline execution.
+- The user must be able to list the analysis tools executed in a specific pipeline execution.
+- The user must be able to access dashboards visualising the analysis reports.
+
+#### Scenario: Custom Visualisation Dashboards
+
+This scenario verifies that an authenticated user may create custom dashboards for visualising pipeline executions data and analysis reports.
+
+- The user must be able to use the Dashboards component (Grafana) to create new visualisation panels.
+- The user must be able to use the Dashboards component (Grafana) to create new dashboards and integrate default and custom visualisation panels.
+
+
+### Analysis Pipelines Execution Automation
+
+This test case will be specified when the integration with the Notification and Automation services will take place.

docs/usage/analysis-dashboards.md

@@ -38,6 +38,21 @@ A set of preconfigured library panels is provided. Users can import these panels
 
 ![Grafana settings - Variables](img/user-manual/grafana-settings-variables.png) <!-- This should show `executions` instead of `executions_admin` -->
 
+A number of dashboards are provided by default.
+
+The **Pipeline Executions Dashboard** provides statistics of past and ongoing executions, a timeline, and the list of executions:
+
+![Grafana pipeline executions dashboard](img/user-manual/grafana-pipeline-executions-dashboard.png)
+
+The **Pipeline Execution Dashboard** shows information about a specific pipeline execution, including the input parameters, the resources consumption, and the list of executed analysis tools:
+
+![Grafana pipeline execution dashboard](img/user-manual/grafana-pipeline-execution-dashboard.png)
+
+The **Analysis Report Dashboards** display the reports generated by the analysis tools (here: a PyLint report):
+
+![Grafana pylint report dashboard](img/user-manual/grafana-pylint-report-dashboard.png)
+
+
 ### Custom Dashboards
 
 Users have the ability to customise their dashboards by reorganising panels, creating new dashboards, or adding new visualizations based on their executed pipelines and job reports.

docs/usage/img/user-manual/app-authenticated.png

@@ -14,6 +14,7 @@ nav:
       - design/architecture.md
       - design/requirements.md
       - design/use-cases.md
+      - design/validation.md
     - Usage:
       - usage/user-manual.md
       - usage/analysis-dashboards.md