Skip to content

feat: limit job results on group overview and dashboard#7058

Merged
mergify[bot] merged 1 commit intoos-autoinst:masterfrom
okurz:feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2
Apr 9, 2026
Merged

feat: limit job results on group overview and dashboard#7058
mergify[bot] merged 1 commit intoos-autoinst:masterfrom
okurz:feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2

Conversation

@okurz
Copy link
Copy Markdown
Member

@okurz okurz commented Mar 2, 2026

Implement a limit on the number of jobs fetched for job group overview,
parent group overview, and the dashboard (consistent with the tests
overview page). This prevents performance issues and 502 errors when
accessing job groups with a huge number of jobs (e.g. >10k).

A warning message is displayed to the user when results are truncated.

I had started in

until I realized that also the main index page aka "dashboard" of openQA
instances impacted. Also I found that we already have a limit for
/tests/overview that we can adapt here as well. And also regardless of
any optimization as done in #7026 we still benefit from a limit. I plan to
rebase #7026 on top and bump the default limit after optimization.

Related progress issue: https://progress.opensuse.org/issues/196913

Screenshot_20260302_144645_openqa_limit_job_group_results

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.87%. Comparing base (594c7e5) to head (96712f4).
⚠️ Report is 27 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #7058   +/-   ##
=======================================
  Coverage   99.87%   99.87%           
=======================================
  Files         418      418           
  Lines       44000    44075   +75     
=======================================
+ Hits        43945    44020   +75     
  Misses         55       55           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch from 9cd5f55 to 1bbeeb7 Compare March 3, 2026 22:08
@okurz
Copy link
Copy Markdown
Member Author

okurz commented Mar 3, 2026

Extended tests to cover the missing lines.

EDIT: … or so I thought

@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch 2 times, most recently from 57dc1c2 to d620f1c Compare March 11, 2026 13:33
@os-autoinst os-autoinst deleted a comment from mergify bot Mar 11, 2026
@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch 3 times, most recently from b876a9e to a509d1d Compare March 23, 2026 14:00
Comment on lines +190 to +193
if (defined $max_jobs_limit && $total_jobs_seen >= $max_jobs_limit) {
$limit_exceeded = $max_jobs_limit;
last;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (defined $max_jobs_limit && $total_jobs_seen >= $max_jobs_limit) {
$limit_exceeded = $max_jobs_limit;
last;
}
if (defined $max_jobs_limit && $total_jobs_seen >= $max_jobs_limit) {
$limit_exceeded = $max_jobs_limit;
last;
}
else {
$remaining = $max_jobs_limit - $total_jobs_seen if defined $max_jobs_limit;
}

increases the visibility/readability and decreases reduntant checks. it is cleaner

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand. Removing the indendation will be invalid format. Maybe you just wanted to add the "else". But why does this additional "else" mixing pre-condition and post-condition increase visibility/readability?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to say that many time I mark something as suggestion but it is not the exact code. it is more to show the intention. having said that, the purpose is to remove a second check as is in my $remaining = defined $max_jobs_limit ? $max_jobs_limit - $total_jobs_seen : undef; I think you can use only once the defined $max_jobs_limit and make it more easy to read and where the condition is involved.

Copy link
Copy Markdown
Contributor

@perlpunk perlpunk Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to say that many time I mark something as suggestion but it is not the exact code.

It makes it harder for everyone else to understand, though.

You could have just indented it to make it understandable. If you click on the suggestion icon, the code is already indented, so it's not even a lot of work.

Regarding the suggestion:

If you have

if ($xy) {
    something;
    last;
}

then an else branch is useless.

having said that, the purpose is to remove a second check as is in my $remaining = defined $max_jobs_limit ? $max_jobs_limit - $total_jobs_seen : undef;

And it doesn't really reduce the number of times if defined $max_jobs_limit is called.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have a point (I think you are talking about semantics) but it doesnt address my initial review. the defined $max_jobs_limit appears twice. the question is whether can remove the second check (as Tina said even without else) and make it more clean/obvious/readable. it is small thing, which it will end up in the same functionality or leave it as is.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as perlpunk also explained there is no benefit to the suggestion so leaving this as is

Copy link
Copy Markdown
Contributor

@perlpunk perlpunk Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@d3flex

but it doesnt address my initial review

I did.

the question is whether can remove the second check

Your suggestion does not remove a check.

Now this PR has been merged, so you can create another PR on top with your suggestion, then everyone can see what you mean.

@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch from a509d1d to c21f771 Compare March 24, 2026 22:29
@os-autoinst os-autoinst deleted a comment from mergify bot Mar 27, 2026
@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch 3 times, most recently from 8daa2c6 to 1dc79bd Compare March 31, 2026 21:36
Motivation:
- Performance issues and 502 errors occurred when accessing job groups with
  a huge number of jobs (e.g. >10k).
- Consistency with the tests overview page which already implements such limits.
- Missing test coverage for boundary conditions and JSON response consistency
  in the initial implementation.

Design Choices:
- Implemented a limit on the number of jobs fetched for job group overview,
  parent group overview, and the dashboard.
- Standardized limit reporting with a limit_exceeded flag in both HTML and
  JSON responses.
- Optimized compute_build_results to respect remaining job limits across
  multiple groups and builds.
- Refined limit-reached logic to correctly handle group boundaries on the
  dashboard.
- Extended t/22-dashboard.t with comprehensive coverage for JSON reporting,
  boundary cases, and invalid regex error handling.

Benefits:
- Prevents web UI timeouts and improves server responsiveness for large
  datasets.
- Clear user feedback via warning messages when results are truncated.
- More robust and fully tested implementation of performance-critical
  limits.

Related issue: https://progress.opensuse.org/issues/196913
@okurz okurz force-pushed the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch from 1dc79bd to 96712f4 Compare April 1, 2026 11:51
@os-autoinst os-autoinst deleted a comment from mergify bot Apr 9, 2026
@okurz
Copy link
Copy Markdown
Member Author

okurz commented Apr 9, 2026

rebased, fixed conflicts, ready for review

@mergify mergify bot merged commit 441b1f6 into os-autoinst:master Apr 9, 2026
51 checks passed
@okurz okurz deleted the feature/033_poo196913_make_openqa_handle_requests_gracefully_alternative_2 branch April 9, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants