Infinite loop in BandwidthAllocator.allocate() causes CPU 100% under high load (stable-10431)

[semi-lion]

Description
Infinite loop in BandwidthAllocator.allocate() method causes CPU to reach 100% when handling large number of participants. The issue occurs when insufficient bandwidth causes the improve() method to return 0, but the while loop condition oldRemainingBandwidth != remainingBandwidth remains true, leading to infinite iteration. This problem is intermittent and occurs when stress levels exceed 1.0, even with the configured limit of 60 participants per JVB.

Current behavior
CPU usage reaches 100% under high load scenarios
Infinite loop in bandwidth allocation algorithm
System becomes unresponsive and load balancer redirects traffic to other JVB instances
Logs show 3,430+ WARNING messages related to TCC packet processing
High RTT values (6000ms+) and packet reordering issues
Intermittent stress level exceeding 1.0 (overloaded state)
Occurs even with 60 participants per JVB limit configured
Stress level fluctuates and occasionally spikes above threshold

Expected Behavior
Bandwidth allocation should complete within reasonable iterations
CPU usage should remain within normal limits even under high load
System should handle 60 participants per JVB without infinite loops
Bandwidth allocation should gracefully handle insufficient bandwidth scenarios
Stress level should remain below 1.0 consistently

Possible Solution
Add loop counter with maximum iteration limit and early termination logic:

var loopCount = 0
val maxLoops = 100
var totalConsumed = 0L

while (oldRemainingBandwidth != remainingBandwidth && loopCount < maxLoops) {
    loopCount++
    oldRemainingBandwidth = remainingBandwidth
    totalConsumed = 0L
    
    for (i in sourceBitrateAllocations.indices) {
        val sourceBitrateAllocation = sourceBitrateAllocations[i]
        if (sourceBitrateAllocation.constraints.isDisabled()) {
            continue
        }

        val consumed = sourceBitrateAllocation.improve(remainingBandwidth, i == 0)
        remainingBandwidth -= consumed
        totalConsumed += consumed
        
        if (remainingBandwidth < 0) {
            oversending = true
        }

        if (sourceBitrateAllocation.isOnStage() && !sourceBitrateAllocation.hasReachedPreferred()) {
            break
        }
    }
    
    // Early termination if no bandwidth was consumed
    if (totalConsumed == 0L) {
        logger.debug("No bandwidth consumed in iteration $loopCount, breaking loop")
        break
    }
}

if (loopCount >= maxLoops) {
    logger.warn("Bandwidth allocation loop exceeded maximum iterations: $loopCount")
}

Steps to reproduce
Set up a Jitsi Meet conference with 60 participants per JVB (configured limit)
Have participants turn on video simultaneously or in quick succession
Monitor CPU usage and JVB logs
Observe the following symptoms:
Intermittent stress level spikes above 1.0
CPU usage reaching 100%
Repeated WARNING messages in logs
High RTT values and packet processing issues
Stress level exceeding 1.0 even with participant limit

Environment details
Jitsi Videobridge version: stable-10431 (Docker image)
Deployment: Docker container
Participants: 60 users per JVB (configured limit)
Scenario: Participants turning on video simultaneously or in quick succession
Stress level: Intermittently exceeds 1.0 (overloaded: true)
Endpoints: 60 (configured limit)
Log evidence: 3,430 WARNING messages, TCC packet processing errors
File location: jvb/src/main/kotlin/org/jitsi/videobridge/cc/allocation/BandwidthAllocator.kt lines 251-272
Method: allocate() method in BandwidthAllocator class
Configuration: 60 participants per JVB limit is set
Docker image: jitsi/jvb:stable-10431
Root cause: The infinite loop occurs when sourceBitrateAllocation.improve() returns 0 due to insufficient bandwidth, but the while loop condition oldRemainingBandwidth != remainingBandwidth remains true, causing infinite iteration. This happens intermittently even with the configured participant limit, causing stress levels to spike above 1.0.

[JonathanLennox]

I don't understand how oldRemainingBandwidth != remainingBandwidth can stay true if improve returns 0, given that the first line of the loop is oldRemainingBandwidth = remainingBandwidth, and improve is the only thing that changes it. Can you explain this more?

[semi-lion]

Thank you for the quick follow-up and the clarification.

To be precise, we have not yet confirmed that improve() returns exactly 0 in the problematic state. What we do observe in production is a tight loop in the allocator with repeated “insufficient bandwidth → suspended” messages, bridge stress > 1.0, CPU near 100%, and participants losing both video and audio.

Would you be able to run a quick test under conditions similar to ours to see if you can reproduce the allocator spin?

Environment / config we used

Image: jitsi/jvb:stable-10431 (Docker)

Topology: 1×(web+prosody+jicofo), 8×JVB (Octo enabled)

.env file settings:
ENABLE_OCTO=1
ENABLE_SCTP=1
ENABLE_OCTO_SCTP=1
JICOFO_OCTO_REGION=asia
OCTO_BRIDGE_SELECTION_STRATEGY=RegionBasedBridgeSelectionStrategy
MAX_BRIDGE_PARTICIPANTS=60

UDP buffer size settings:
net.core.rmem_max=10485760 # 10 MB
net.core.wmem_max=10485760 # 10 MB

Workload: 4 large conferences in parallel (3×50+, 1×100+), all cameras ON;

Symptoms we observe

Example of the repeating log:

[confId=ada79541f6832bbb conf_name=XXXXXXXXXXXXXX meeting_id=d29e8c7e epId=f7c31c1c stats_id=Ofelia-PBR]
BandwidthAllocator.allocate#266: Sources suspended due to insufficient bandwidth (bwe=373159 bps): [df883782-v1]

JVB stress > 1.0, CPU ~100%, users lose both video & audio.

What we changed (mitigation)

We added (1) a max-iteration guard and (2) an early exit when a full pass consumes zero bandwidth. After this change, the “suspended due to insufficient bandwidth” line no longer repeats indefinitely and the bridge recovers instead of pegging CPU.

var loopCount = 0
val maxLoops = 100
var totalConsumed = 0L

while (oldRemainingBandwidth != remainingBandwidth && loopCount < maxLoops) {
    loopCount++
    oldRemainingBandwidth = remainingBandwidth
    totalConsumed = 0L

    for (i in sourceBitrateAllocations.indices) {
        val s = sourceBitrateAllocations[i]
        if (s.constraints.isDisabled()) continue
        val consumed = s.improve(remainingBandwidth, i == 0)
        remainingBandwidth -= consumed
        totalConsumed += consumed
        if (remainingBandwidth < 0) oversending = true
        if (s.isOnStage() && !s.hasReachedPreferred()) break
    }

    if (totalConsumed == 0L) {
        logger.debug("No bandwidth consumed in iteration $loopCount, breaking loop")
        break
    }
}

if (loopCount >= maxLoops) {
    logger.warn("Allocation loop hit max iterations: $loopCount (remaining=$remainingBandwidth, sources=${sourceBitrateAllocations.size})")
}

Thanks again for looking into this.

[bgrozev]

Was your "Allocation loop hit max iterations" log ever printed?

I am trying to understand how the outer loop can run an infinite (or very high) number of times.

It seems the totalConsumed check you added is equivalent to the oldRemainingBandwidth != remainingBandwidth condition of the loop (it's an invariant that oldRemainingBandwidth == remainingBandwidth + totalConsumed). So, if all the inner improve() calls return 0 we break out of the outer loop -- if this was not the case we would see an infinite loop any time there's sufficient bandwidth for all layers, and we don't see that.

The remaining option is for at least one of the inner improve() calls to return non-zero repeatedly. This shouldn't happen as there's a finite number of layers, and SingleSourceAllocation.improve() always increases targetIdx when it returns non-zero. Usually this should converge within just a few steps, but it should be bounded by sourceBitrateAllocations.size * number_of_layers or ~60*9=540. Note that the less available bandwidth there is, the quicker this will terminate. In our profiled tests BandwidthAllocator.allocate hasn't shows up as a hotspot.

I might be missing a bug in SingleSourceAllocation. If that's the case, you should see the Allocation loop hit max iterations you added printed, and it should remain printed if you increase the limit to say 600.

I'm also curious how you zeroed in on this particular function as causing an issue. The logs and observations you describe point to an overloaded bridge (which is somewhat expected if you load a lot of endpoints and enable video at the same time), but I don't see anything to suggest an "infinite loop" or a performance issue in improve().