Increasing MaxInFlight does not increase utilization

[martin-sucha]

The MaxInFlight is divided by the number of connections and the remainder after division is not taken into account:

https://github.com/kiwicom/go-nsq/blob/796abbcb19fbb48d95321ecb41d54a25cf35949c/consumer.go#L288-L290

This leads to the actual in-flight being lower than the configured MaxInFlight value, for example:

len(r.conns()): 6
r.getMaxInFlight(): 9
perConnMaxInFlight() returns 1

So the actual concurrency limit is 6 instead of 9.

We are using ChangeMaxInFlight to dynamically limit the concurrency of the consumer, (between 0 and the number of concurrent handlers) so that our database has time to scale up when workload increases. We monitor the actual concurrency and adjust the limit accordingly. The issue is that since go-nsq lowers the actual limit, setting ChangeMaxInFlight slightly higher than it was before does not have the desired effect of the consumer doing more work.

We have implemented a workaround on our side so that each pod calls ChangeMaxInFlight with multiples of connection count, but it would be better to fix the root cause in go-nsq.

What do you think about distributing the remainder of the limit after division between random connections? Respectively:

• increment the per-connection limit by one if maxInFlight%len(r.conns)<connectionIndex
• shuffle the connections list periodically, so that over time messages are read from nsqd servers equally