RFC: A/B comparison on tree details (builds, boots, tests)

[alanpeixinho]

Problem

Users compare two kernel commits by opening two tabs and eyeballing tables. The dashboard shows one revision at a time and period-over-period metrics, but not a deliberate revision A vs revision B view.

Proposal

Users pick the revisions they want to compare (two commits on the same tree/branch). The UI may offer suggestions (previous commit, branch head, last visited, etc.) but does not auto-select sides.

We compare pass/fail counts (and inconclusive counts if shown) for builds, boots, and tests between the two revisions and surface count deltas only — no regression/fix labels for now, and not per-row status flips.

Same filters (diffFilter, origin, time window) apply to both sides. Comparison state is shareable via URL query params.

---

Decided (or proposed to decide in this RFC)

Topic	Proposal
Comparable pair	Two commits on the same (tree, repo, branch)
What we compare	Counts of passing and failing builds, boots, and tests (and inconclusive, if shown) — same buckets the dashboard already uses (PASS, FAIL, groupStatus inconclusive)
Delta semantics	Δ pass = pass_B − pass_A, Δ fail = fail_B − pass_A per entity and per group. Show deltas only; no regression/fix classification yet
Grouping	Tree-wide summary plus breakdown at an agreed grain (e.g. per config for builds, per path/platform for boots and tests) — exact keys TBD
Filter parity	Same filters on both sides before counts are computed
Scope	Tree Details: builds, boots, and tests

Example

Side A (abc1234)  →  Side B (def5678)

Builds:  400 pass · 12 fail  →  395 pass · 18 fail   (Δ -5 pass · +6 fail)
Boots:   380 pass ·  8 fail  →  372 pass · 14 fail   (Δ -8 pass · +6 fail)
Tests: 1,200 pass · 50 fail  → 1,180 pass · 72 fail  (Δ -20 pass · +22 fail)

path: kselftest · x86_64 (tests)
  A: 10 pass · 2 fail  →  B: 8 pass · 4 fail  →  Δ: -2 pass · +2 fail

---

Rough UX

Side A: [pick revision]  ↔  Side B: [pick revision]
        suggestions: previous · branch head · …

[Builds | Boots | Tests] tab

Tree summary: pass_A · fail_A · pass_B · fail_B · Δ pass · Δ fail
Grouped table (grain TBD per tab)

---

Success criteria

• Triage works without two tabs
• Comparison links are shareable and reproducible
• Acceptable latency on typical trees (target TBD after spike)

[mentonin]

Decided (or proposed to decide in this RFC)

Topic Proposal
Comparable pair Two commits on the same (tree, repo, branch)

1. Why limit to same branch/repo?. I think searching by commit hash without even specifying a tree should be allowed too.
2. What happens if I want to lookup tests in a commit with multiple checkouts?
3. What about no checkouts? As a user, I would expect to search by commit hash and be suggested the previous/next commits with checkouts in the branch
4. Suggestions for previous commit with checkouts would be easy for any commit - except for merge commits
5. Suggestions for next commit with checkouts in branch would be great too

What we compare Counts of passing and failing builds, boots, and tests (and inconclusive, if shown) — same buckets the dashboard already uses (PASS, FAIL, groupStatus inconclusive)

I think count diffs are the least useful info - easy to compare visually, and little information to gain. Why not go for per-row status flips (changes, only in A, only in B)?

[bhcopeland]

Will this show a comparison of the test cases, or the overall a/b comparsion?

What I'd like to see is an A/B comparison of say LTP-* or Kselftest-* (same with build|boot). The difficulty comes with A/B per board, (or config).

If it's an overall A/B comparison, thats fine but might not be too useful if its just showing the A/B of failed tests between X and Y.

[bhcopeland]

Here is photos of my "local" report I use. Nothing fancy

[alanpeixinho]

1. Why limit to same branch/repo?. I think searching by commit hash without even specifying a tree should be allowed too.
   No special reason to, just considered it would be the most usual case.
2. What happens if I want to lookup tests in a commit with multiple checkouts?
   We can decided, but right now, I think the last checkout would be the choice.
3. What about no checkouts? As a user, I would expect to search by commit hash and be suggested the previous/next commits with checkouts in the branch
   As I mentioned, the idea is that we can provide suggestions on checkouts for comparison, I believe this falls in that case.

[alanpeixinho]

Will this show a comparison of the test cases, or the overall a/b comparsion?

What I'd like to see is an A/B comparison of say LTP-* or Kselftest-* (same with build|boot). The difficulty comes with A/B per board, (or config).

If it's an overall A/B comparison, thats fine but might not be too useful if its just showing the A/B of failed tests between X and Y.

I think this is the best place to discuss. Lets throw suggestions.
What would be the most beneficial comparison here?