Add cooperative rebalance support

[thynson]

No description provided.

[thynson]

incremetalAssign and incrementalUnassign are already exposed to Node, and anyone can call them in a rebalance callback when partition.assignment.strategy is cooperative-sticky.
Does the KafkaConsumer need to be also modified to provide a default rebalance_cb for cooperative-sticky?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[johncsnyder]

I would love to see this feature added. 🙏

As far as I can tell, the cooperative-sticky partition assignment strategy can't work without adding the bindings for incremental assign/unassign.

I've tried but always received the error application *assign() call failed: Changes to the current assignment must be made using incremental_assign() or incremental_unassign() when rebalance protocol type is COOPERATIVE.

@thynson are there any plans to re-open this PR?

[thynson]

As far as I can tell, the cooperative-sticky partition assignment strategy can't work without adding the bindings for incremental assign/unassign.

They were implemented in this PR, 21a2226#diff-e574f6617d5a1bd9fb35fab6b02e906b120024e0dcae87b010eed716e23820d4R45, but you need to set partition.assignment.strategy to cooperative-sticky. Or if you're using custom rebalance_cb, call incrementalAssign and incrementalUnassign.

[johncsnyder]

@thynson Yeah, I saw, looks great.

Any chance this will be merged into master? I'd prefer not to maintain my own fork.

[thynson]

Any chance this will be merged into master? I'd prefer not to maintain my own fork.

If @iradul has time to take a look.

[johncsnyder]

Sweet, thanks 🙏

[iradul]

@thynson, thank you for the PR

[iradul]

Missing items:

• assignment_lost implementation
• e2e test(s)

[thynson]

I'll try to implement the missing parts, but I'm still busy for some days.

[micheleangioni]

Any hope this will get implemented? @thynson :)
It would be extremely useful for us

[neuralspin]

@micheleangioni
Did you find a way to solve this? We also really need it @thynson )

[spalax]

and we need it a lot :)

[serj026]

We are eagerly awaiting this feature 🙏

[notgosu]

we need it also :)

[votar1408]

It can save the world, great the feature! @micheleangioni

[yaroslav-fedyshyn-playson]

Thanks for the PR. It will significantly reduce our struggling and help us!

[thynson]

I did not have any experience of writing an e2e test for rdkafka. So any help would be great, or I have to take some time to finish the e2e test.

[ofekdeitch-oligo]

Hi @thynson , why can't we use the e2e tests that @serj026 implemented in his PR?
https://github.com/Blizzard/node-rdkafka/pull/1081/files

Maybe we can merge the two PRs and get this feature finally merged