PART 2: perf: handle huge job groups gracefully - After #7058 by okurz · Pull Request #7026 · os-autoinst/openQA

okurz · 2026-02-21T16:32:22Z

Optimize compute_build_results by using database-level aggregation
instead of fetching and iterating all job objects.
Move job deduplication (per scenario) to the database level.
Implement a safety limit of 5,000 jobs per build to prevent timeouts.
Optimize comment/review tracking by only checking failed jobs.
Add t/61-job_group_aggregation.t to verify aggregation and limit enforcement.
Extract category mapping into _get_job_result_category helper.
Reuse helper in both count_job and _count_job_aggregated.
Ensure consistent result categorization across legacy and optimized paths.
Add job_group_overview_max_jobs to misc_limits in openqa.ini.
Pass this limit from web and API controllers to compute_build_results.
De-duplicate common job data in t/61-job_group_aggregation.t.
Add controller and API tests for limit enforcement.

Related progress issue: https://progress.opensuse.org/issues/196913

After:

feat: limit job results on group overview and dashboard #7058

codecov · 2026-02-21T17:57:55Z

Codecov Report

❌ Patch coverage is 98.74477% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.86%. Comparing base (594c7e5) to head (de7b0c6).
⚠️ Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
lib/OpenQA/WebAPI/Controller/API/V1/JobGroup.pm	60.00%	2 Missing ⚠️
lib/OpenQA/BuildResults.pm	98.91%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7026      +/-   ##
==========================================
- Coverage   99.87%   99.86%   -0.01%     
==========================================
  Files         418      420       +2     
  Lines       44000    44190     +190     
==========================================
+ Hits        43945    44132     +187     
- Misses         55       58       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Martchus

Look generally good. Have you tested this with production data and compared before and after? I think for bigger changes like that we should do that (like when I recently did a similar optimization for the test results overview page). This way we could also confirm whether it helps with the problem from the motivating ticket (https://progress.opensuse.org/issues/196709).

lib/OpenQA/Setup.pm

lib/OpenQA/BuildResults.pm

Martchus · 2026-02-23T13:16:37Z

lib/OpenQA/BuildResults.pm

    $job_result->{total} = 0;
 }

+sub _get_job_result_category ($state, $result) {


I'm wondering whether we can reuse some of the mapping functions we already have to avoid repetitiveness here. Of course this is already better than how it was before your change.

Martchus

I have just some style nitpicks but enough to say that at least some of them should be improved.

t/61-job_group_aggregation.t

Martchus · 2026-02-27T13:10:23Z

t/61-job_group_aggregation.t

+    my @jobs_data;
+    push @jobs_data, {%common, id => 600000 + $_, TEST => "test_$_"} for (1 .. $num_jobs);
+    $schema->resultset('Jobs')->populate(\@jobs_data);


Use map or avoid the intermediate array by calling create in a loop.

Martchus · 2026-02-27T13:11:43Z

t/61-job_group_aggregation.t

+    $t->get_ok("/group_overview/$group_id_ctrl" => form => {distri => 'distri', version => $version, build => $build})
+      ->status_is(400)->content_like(qr/exceeds the limit of 5/);
+};
+subtest 'API Controller limit enforcement' => sub {


This subtest is almost like the previous ones and they are all very verbose. It would make sense to avoid this kind of duplication.

Martchus · 2026-02-27T13:13:49Z

t/61-job_group_aggregation.t

+use Test::Mojo;
+my $test_case = OpenQA::Test::Case->new;
+my $schema = $test_case->init_data(fixtures_glob => '01-jobs.pl 03-users.pl');
+my $t = Test::Mojo->new('OpenQA::WebAPI');
+my $group_id = 1001;
+my $group = $schema->resultset('JobGroups')->find($group_id);
+subtest 'Aggregation with deduplication' => sub {


Please use blank lines to separate different sections.

Martchus · 2026-02-27T13:15:31Z

lib/OpenQA/BuildResults.pm

+    if ($state eq OpenQA::Jobs::Constants::DONE) {
+        my $meta = OpenQA::Jobs::Constants::meta_result($result);
+        return 'passed' if $meta eq OpenQA::Jobs::Constants::PASSED;
+        return 'softfailed' if $meta eq OpenQA::Jobs::Constants::SOFTFAILED;
+        return 'skipped' if $meta eq OpenQA::Jobs::Constants::ABORTED;
+        return 'failed' if $meta eq OpenQA::Jobs::Constants::FAILED || $meta eq OpenQA::Jobs::Constants::NOT_COMPLETE;
+    }
+    return 'skipped' if $state eq OpenQA::Jobs::Constants::CANCELLED;


It would look less noisy if the constants were imported.

lib/OpenQA/BuildResults.pm

d3flex

it seems to me that this will have impact in the performance. there are multiple $jobs_resultset->search which can stress the system with long results. is there anything better to avoid this? or do i miss something?

Martchus · 2026-03-02T13:31:14Z

lib/OpenQA/BuildResults.pm

+        if ($jr{children}) {
+            for my $child_id (keys %{$jr{children}}) {


Suggested change

if ($jr{children}) {

for my $child_id (keys %{$jr{children}}) {

if (my $children = $jr{children}) {

for my $child_id (keys %$children) {

mergify · 2026-03-06T13:08:40Z

This pull request is now in conflicts. Could you fix it? 🙏

mergify · 2026-03-23T16:51:46Z

This pull request is now in conflicts. Could you fix it? 🙏

- Optimize compute_build_results by using database-level aggregation instead of fetching and iterating all job objects. - Move job deduplication (per scenario) to the database level. - Implement a safety limit of 5,000 jobs per build to prevent timeouts. - Optimize comment/review tracking by only checking failed jobs. - Add t/61-job_group_aggregation.t to verify aggregation and limit enforcement. - Extract category mapping into _get_job_result_category helper. - Reuse helper in both count_job and _count_job_aggregated. - Ensure consistent result categorization across legacy and optimized paths. - Add job_group_overview_max_jobs to misc_limits in openqa.ini. - Pass this limit from web and API controllers to compute_build_results. - De-duplicate common job data in t/61-job_group_aggregation.t. - Add controller and API tests for limit enforcement. Related progress issue: https://progress.opensuse.org/issues/196913

- Introduce OpenQA::Error::LimitExceeded typed exception for robust error handling. - Decompose compute_build_results into smaller, testable helper functions. - Consolidate database queries to reduce O(N) round-trips. - Centralize job limit configuration using app config and internal defaults. - Implement graceful degradation for oversized builds instead of hard failure.

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch from b35dfa2 to afa425c Compare February 21, 2026 17:39

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch 2 times, most recently from 49f42c8 to 3345729 Compare February 21, 2026 19:58

Martchus requested changes Feb 23, 2026

View reviewed changes

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch from 3345729 to 38ddab6 Compare February 27, 2026 12:05

Martchus requested changes Feb 27, 2026

View reviewed changes

d3flex reviewed Feb 27, 2026

View reviewed changes

lib/OpenQA/BuildResults.pm Show resolved Hide resolved

d3flex requested changes Feb 27, 2026

View reviewed changes

Martchus reviewed Mar 2, 2026

View reviewed changes

okurz mentioned this pull request Mar 2, 2026

feat: limit job results on group overview and dashboard #7058

Open

okurz marked this pull request as draft March 2, 2026 14:59

okurz changed the title ~~perf: handle huge job groups gracefully~~ PART 2: perf: handle huge job groups gracefully - After #7058 Mar 3, 2026

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch 3 times, most recently from 61ecbbb to 1cee06d Compare March 23, 2026 14:00

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch from 1cee06d to 24914b1 Compare March 24, 2026 22:29

os-autoinst deleted a comment from mergify bot Mar 31, 2026

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch 3 times, most recently from 8754ad6 to 116ad1f Compare March 31, 2026 21:36

okurz added 2 commits April 1, 2026 13:08

okurz force-pushed the feature/019_poo196913_handle_huge_job_groups_gracefully branch from 116ad1f to de7b0c6 Compare April 1, 2026 11:50

		if ($jr{children}) {
		for my $child_id (keys %{$jr{children}}) {

Conversation

okurz commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Martchus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Martchus Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Martchus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Martchus Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Martchus Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Martchus Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Martchus Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

d3flex left a comment

Choose a reason for hiding this comment

Uh oh!

Martchus Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Mar 6, 2026

Uh oh!

mergify bot commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

okurz commented Feb 21, 2026 •

edited

Loading

codecov bot commented Feb 21, 2026 •

edited

Loading