KEDA does not work on runs having more than 100 Jobs

[jangraevell-pd]

Report

In the Logic for github_runner_scaler.go there is a hard-limit of "100" jobs per page. Those are also always returned in the same order, so as soon as the first 100 Jobs are done, no further Jobs are found to be executed.

While we could now discuss, if it makes sense to have so many jobs in one run, this still imposes a hard-limit, that we cannot come by.

As Github does neither provide a parameter for filtering by status nor a per_page-value > 100, there is no other way than implementing a merged result by doing multiple requests, one for each page.

Expected Behavior

When I have a Github-Run with more than 100 Jobs, then i want all Jobs to be considered for scaling.

Actual Behavior

As soon as the first 100 jobs of a run are completed, no further jobs of that run are considered "open"

Steps to Reproduce the Problem

Create a Workflow with more than 100 Jobs, and see, that scaling will stop after 100 Jobs:

name: Matrix 150 jobs demo

on:
  workflow_dispatch: {}

jobs:
  generate_matrix:
    name: Generate 150-entry matrix
    runs-on:
      labels:
        - your_scaling_label_here
    outputs:
      matrix: ${{ steps.build.outputs.matrix }}
    steps:
      - name: Build matrix with 150 indices
        id: build
        run: |
          i=1
          json='['
          while [ "$i" -le 150 ]; do
            if [ "$i" -gt 1 ]; then
              json="${json},"
            fi
            json="${json}${i}"
            i=$((i + 1))
          done
          json="${json}]"

          echo "matrix={\"index\":${json}}" >> "$GITHUB_OUTPUT"

  print_index:
    name: Job ${{ matrix.index }}
    needs: generate_matrix
    runs-on:
      labels:
        - your_scaling_label_here
    strategy:
      fail-fast: false
      matrix: ${{ fromJson(needs.generate_matrix.outputs.matrix) }}
    steps:
      - name: Output index
        run: |
          echo "${{ matrix.index }}"

Logs from KEDA operator

KEDA Version

2.20.2

Kubernetes Version

None

Platform

Microsoft Azure

Scaler Details

Github-Runner-Scaler

Would you be open to contributing a fix?

Maybe

Anything else?

No response