Add combined results and plot generation (#219)

* Add an option --combined-results to create list with all use cases

* Add options to draw comparison plots

* Handle n_jobs param correctly and add Summary 2 and All cases 2 pages

* Add footnote for plot and add metrics

* Update footnote

* Update docs

* Code format

* Address comments

Author: avolkov-intel

sklbench/report/README.md

@@ -18,6 +18,10 @@ Raw results are converted into a pandas dataframe and the final report is made b
 | `--report-file`                                | str    | report.xlsx                                         |                                                | Report file path.                                                                                                                |
 | `--report-type`                                | str    | separate-tables                                     | ('separate-tables',)                           | Report type ("separate-tables" is the only supported now).                                                                       |
 | `--compatibility-mode`                         |        | False                                               |                                                | [EXPERIMENTAL] Compatibility mode drops and modifies results to make them comparable (for example, sklearn and cuML parameters). |
+| `--performance-stability-metrics`</br>`-psm`   |        | False                                               |                                                | Adds performance stability metrics (`1st run time[ms]`, `1st-mean run ratio`, `median time[ms]`, `time CV`) to the report.       |
+| `--combined-results`                           |        | False                                               |                                                | [EXPERIMENTAL] Creates `All cases` and `Summary (for plots)` sheets combining time and speedup of all algorithms with per-dtype/total GEOMEAN rows (use with `--compatibility-mode`). |
+| `--draw-plots`                                 |        | False                                               |                                                | [EXPERIMENTAL] Draws Training/Inference speedup bar charts from the combined results (requires `--combined-results`). Footnote hardware parameters are hardcoded; edit them in the code for a production-ready plot.             |
+| `--plot-output`                                | str    | `plot.png`                                          |                                                | [EXPERIMENTAL] Output file path for plots (e.g. `plots.png`); if unset, plots are saved to `plot.png`. Only takes effect together with `--draw-plots`.                           |
 | `--drop-columns`</br>`--drop-cols`             | str    | []                                                  |                                                | Columns to drop from report.                                                                                                     |
 | `--diff-columns`</br>`--diff-cols`             | str    | ['environment_name', 'library', 'format', 'device'] |                                                | Columns to show difference between.                                                                                              |
 | `--split-columns`                              | str    | ['estimator', 'method', 'function']                 |                                                | Splitting columns for subreports/sheets.                                                                                         |

sklbench/report/arguments.py

@@ -93,6 +93,28 @@ def add_report_generator_arguments(
         help="Selects which part of one-vs-one difference to show "
         "(all matrix or one of triangles).",
     )
+    parser.add_argument(
+        "--combined-results",
+        default=False,
+        action="store_true",
+        help="[EXPERIMENTAL] Creates 'All cases' and 'Summary (for plots)' sheets "
+        "combining time[ms] and speedup of all algorithms into single pages with "
+        "per-dtype and total GEOMEAN rows. Use together with --compatibility-mode.",
+    )
+    parser.add_argument(
+        "--draw-plots",
+        default=False,
+        action="store_true",
+        help="[EXPERIMENTAL] Draw Training/Inference speedup bar charts from combined "
+        "results (requires --combined-results).",
+    )
+    parser.add_argument(
+        "--plot-output",
+        type=str,
+        default="plot.png",
+        help="[EXPERIMENTAL] Output file path for plots (e.g., plots.png). "
+        "If not specified, plots are saved to plot.png.",
+    )
     # color scale settings
     parser.add_argument(
         "--perf-color-scale",

sklbench/report/compatibility.py

@@ -34,6 +34,12 @@ def transform_results_to_compatible(results: pd.DataFrame):
                 "min_bin_size",
             ],
         )
+    if (
+        "n_jobs" in results.columns
+        and results["n_jobs"].isna().any()
+        and results["n_jobs"].notna().any()
+    ):
+        results.drop(inplace=True, columns=["n_jobs"])
     if results["environment_name"].unique().size > 1:
         # DBSCAN `eps` parameter drop for different CPUs
         results.drop(