Skip to content

[Feature] Support Virtual Threads#908

Open
devingryu wants to merge 3 commits intoconfluentinc:masterfrom
devingryu:features/enable-virtual-threads
Open

[Feature] Support Virtual Threads#908
devingryu wants to merge 3 commits intoconfluentinc:masterfrom
devingryu:features/enable-virtual-threads

Conversation

@devingryu
Copy link

@devingryu devingryu commented Jan 12, 2026

Adds option to use Virtual Threads for worker pool. (See also: #896)

  • VT support: Added the useVirtualThreads option and execution logic to support Java 21+ Virtual Threads (Project Loom).
  • VT Pinning prevention: Replace synchronization method from synchronized blocks to ReentrantLock to prevent VT pinning.
  • ExecutorService Abstraction: Generalized the return type of setupWorkerPool to ExecutorService and improved queue management and backpressure logic accordingly.

Concerns

  1. Test Environment Compatibility
    The newly added VT-related tests require JDK 21+ and are configured to skip execution using Assumptions if the environment does not support it. It seems that current CI environment is based on JDK 17, if so the tests I added will be skipped in CI.

  2. Concurrency in isPoolQueueLow
    When VT mode is enabled, we cannot directly query the queue size, so WorkerManager.numberRecordsOutForProcessing is used to calculate the load. Please review if this approach creates any potential concurrency issues in a multi-threaded environment.

  3. Defensive Logic
    To ensure the stability of the existing logic, if VT is disabled (!useVirtualThreads), setupWorkerPool is enforced to return a ThreadPoolExecutor (throwing an exception otherwise), and type checks have been added. I am concerned this might introduce a breaking change (compatibility issue) for existing users who inherit AbstractParallelEoSStreamProcessor to use custom Executors.

Checklist

  • Documentation (if applicable)
  • Changelog

* Added boolean useVirtualThreads in ParallelConsumerOptions

* Generalize workerThreadPool logics to handle Virtual Threads

* Added tests
* prevents virtual threads pinning
@devingryu devingryu requested a review from a team as a code owner January 12, 2026 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant