Support for KIP-932

[omnilight]

Hi!

Are there any plans or roadmap to support KIP-932 (queues for kafka)

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434

Thank you!

[twmb]

This was only just released. I no longer pre-implement code from KIPs because Confluent has changed how things are actually implemented -- and not updated KIPs-- repeatedly in the past. That said, yes I would like to have this implemented. I'm not sure what the API will look like.

[twmb]

I've checked out Kafka 4.0; the APIs introducing share groups are currently marked unstable and are not exposed unless explicitly opted into via a configuration option. I'm not going to implement share group support until Confluent GAs it, since Confluent has pretty regularly changed the shape of Kafka responses in the past before GA.

[twmb]

I'm going to close this for now and reopen this in a new issue (or just implement it) once Kafka itself stabilizes share queues. The main reason to close is: I don't want an issue sticking around to add support for a currently unstable feature, since open issues kinda distract me a bit (I probably open them at least a couple dozen times while preparing for / going over what remains for releases).

[kmrgirish]

@twmb Given that this KIP is now marked stable for the Kafka 4.2.0 release planned for January 2026, are there any plans to introduce support for it in the library?

[twmb]

Candidly, I have little interest in supporting KIP-932 while the foundation it's built on - KIP-848 - has what I believe to be fundamental flaws.

KAFKA-19233 is the most egregious.
KAFKA-19235 is bad.

Confluent has a history of ignoring my bug reports. I believe the only way to get eyes on these is to actually setup a proxy between the Java client OR librdkafka and the broker and forcefully recreate the scenario I'm running into by dropping ConsumerGroupHeartbeat responses. If that recreates the problem, it definitively points to this being a broker issue, and since it'd be against Confluent's own clients, perhaps they'd take it more seriously. If it does not recreate the problem, bad for me.

Since recreating the problem is a lot of work, it's very low on my priority list. I'd be very open to help.

I may check if 932 is possible to use without 848, but literally the first line in 932 describing share group membership says "This KIP builds upon the new consumer group protocol in KIP-848". Given how bad 19233 is, I don't want to support a feature that ultimately will lead to end-user problems that ultimately come back to me.

[kmrgirish]

Since recreating the problem is a lot of work, it's very low on my priority list. I'd be very open to help.

I’ll check on my end and see if I can reproduce the issue.

[ijuma]

A question was posted on KAFKA-19233 and KIP-1251 is meant to address KAFKA-19235.

[twmb]

Thanks for checking, Ismael. I'll loop test franz-go against Kafka 4.2 once 4.2 is released soon since it was released today. I've added a few fixes to the 848 implementation that were found while adding 848 support to kfake itself, with any luck, enough improvements happened in both Kafka and franz-go that there will be no failures (although I'll likely run into what KIP-1251 is meant to fix and hold off making "next gen" the default until 4.3).

As far as your question:
"""
The request has partitions 24, 25 and 26 with epoch 10. The response returns partitions 24 and 25 with epoch 11. This is not a valid transition as the member must remain in epoch 10 until partition 26 is fully revoked. I cannot explain how this happened based on the code yet. Would you have a way to reproduce it?
"""

No clue - the integration tests rarely reliably fail. My flow was:

docker compose down
docker compose up
sleep 3 (roughly)
<run full suite of tests in debug mode, also the broker in debug mode>

I wish I uploaded all broker's logs to the issue.

I may be a bit salty in my statement above due to ...some history (here, in KIPs, and in non-Kafka/Confluent projects)..., I'll see about resetting the salt a bit :)