feat: enable shard-awareness with Proxy Protocol v2 in PrivateLink mode#839
Open
nikagra wants to merge 3 commits intoscylladb:scylla-4.xfrom
Open
feat: enable shard-awareness with Proxy Protocol v2 in PrivateLink mode#839nikagra wants to merge 3 commits intoscylladb:scylla-4.xfrom
nikagra wants to merge 3 commits intoscylladb:scylla-4.xfrom
Conversation
Adds `proxyProtocol` support to `ClientRoutesConfig` so users can declare that the NLB uses Proxy Protocol v2 (PP2) when forwarding connections to ScyllaDB. With PP2, the NLB prepends a binary header carrying the original client source IP and port to each connection it opens to ScyllaDB; ScyllaDB reads the header and routes using the original source port, restoring shard-aware routing end-to-end through the NLB. Changes: - `ClientRoutesConfig`: add `proxyProtocol` field + `withProxyProtocol()` builder method; update equals/hashCode/toString - `DefaultDriverOption`: add `CLIENT_ROUTES_PROXY_PROTOCOL` enum constant - `TypedDriverOption` / `OptionsMap`: add typed wrapper and default value - `reference.conf`: add `advanced.client-routes.proxy-protocol = false` - `DefaultDriverContext`: parse `proxy-protocol` from HOCON; warn (not fail) when `proxyProtocol=true` but shard awareness is disabled - `TcpProxy` / `RoundRobinProxy`: add PP2 header injection mode — after connecting to the target, write the 28-byte TCP4 PP2 binary header carrying the original client IP and source port - `NlbSimulator`: add `proxyProtocol` constructor parameter; pass flag to both per-node (`TcpProxy`) and discovery (`RoundRobinProxy`) proxies - `ClientRoutesIT`: add `should_use_shard_awareness_through_pp2_nlb` test - `PRIVATELINK.md`: document the PP2 mechanism, header format, configuration, and infrastructure requirements Jira ID: DRIVER-391 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ea58854 to
4cf0c72
Compare
There was a problem hiding this comment.
Pull request overview
Adds a proxy-protocol flag to the driver’s client-routes configuration to support shard-awareness end-to-end when traffic goes through a PrivateLink/NLB that injects Proxy Protocol v2, and extends the integration-test NLB simulator to inject PP2 headers.
Changes:
- Add
proxy-protocolto client-routes config surface area (API builder + HOCON option + typed option + defaults + reference.conf). - Parse/validate the new option in
DefaultDriverContext(warn when advanced shard awareness is disabled). - Extend integration-test proxies (
TcpProxy,RoundRobinProxy,NlbSimulator) to optionally prepend a PP2 v2 header; add an end-to-end IT that exercises the flow.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/TcpProxy.java | Adds optional PP2 header injection on proxied connections. |
| integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/RoundRobinProxy.java | Adds optional PP2 header injection for discovery round-robin proxying. |
| integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/NlbSimulator.java | Threads proxyProtocol flag through simulator into per-node and discovery proxies. |
| integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/ClientRoutesIT.java | Adds an end-to-end integration test for shard-awareness through a PP2-enabled NLB. |
| core/src/main/resources/reference.conf | Documents and defaults the new advanced.client-routes.proxy-protocol option. |
| core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultDriverContext.java | Reads the new option from config and warns on incompatible settings. |
| core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java | Exposes a typed option for CLIENT_ROUTES_PROXY_PROTOCOL. |
| core/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.java | Adds default false for CLIENT_ROUTES_PROXY_PROTOCOL in driver defaults. |
| core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java | Introduces CLIENT_ROUTES_PROXY_PROTOCOL option path. |
| core/src/main/java/com/datastax/oss/driver/api/core/config/ClientRoutesConfig.java | Adds proxyProtocol field + builder method + equality/toString updates. |
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/TcpProxy.java
Outdated
Show resolved
Hide resolved
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/TcpProxy.java
Outdated
Show resolved
Hide resolved
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/RoundRobinProxy.java
Outdated
Show resolved
Hide resolved
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/RoundRobinProxy.java
Outdated
Show resolved
Hide resolved
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/ClientRoutesIT.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultDriverContext.java
Outdated
Show resolved
Hide resolved
|
@nikagra , Couple of things:
|
nikagra
added a commit
to nikagra/java-driver
that referenced
this pull request
Mar 19, 2026
- Rename proxyProtocol → nlbShardAwareness throughout (ClientRoutesConfig, DefaultDriverOption, TypedDriverOption, OptionsMap, DefaultDriverContext). The config key path (advanced.client-routes.proxy-protocol) is unchanged. - Remove PP2 integration test infrastructure (ClientRoutesIT, TcpProxy, RoundRobinProxy, NlbSimulator) per reviewer request. - Fix warning log to say "has no effect" instead of "will be ignored". - Fix reference.conf comment to match actual behaviour (logs warning, does not silently ignore). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
|
@dkropachev All three points addressed in the latest commit (
|
Adds PrivateLink implementation notes to the project's standard documentation tree (manual/core/private_link/README.md), following the same structure as other core-driver feature docs. Content covers: - The PrivateLink problem and abstract routing-table model - Architecture overview and implementation deep dives - Shard-awareness through NLBs using Proxy Protocol v2 (PP2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename proxyProtocol → nlbShardAwareness throughout (ClientRoutesConfig, DefaultDriverOption, TypedDriverOption, OptionsMap, DefaultDriverContext). The config key path (advanced.client-routes.proxy-protocol) is unchanged. - Remove PP2 integration test infrastructure (ClientRoutesIT, TcpProxy, RoundRobinProxy, NlbSimulator) per reviewer request. - Fix warning log to say "has no effect" instead of "will be ignored". - Fix reference.conf comment to match actual behaviour (logs warning, does not silently ignore). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fdd0dd2 to
cd5d3c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
proxyProtocolflag toClientRoutesConfig(+ builder, HOCON key,reference.confdefault,TypedDriverOption,OptionsMap)DefaultDriverContextparses the new HOCON key and warns (does not fail) whenproxyProtocol=truebut advanced shard-awareness is disabledTcpProxyandRoundRobinProxygain a PP2 injection mode: after connecting to the target they write the 28-byte TCP4 PP2 binary header carrying the original client IP and source portNlbSimulatorexposes aproxyProtocolconstructor parameter and passes the flag to both per-node and discovery proxiesClientRoutesITaddsshould_use_shard_awareness_through_pp2_nlbend-to-end testBackground
ScyllaDB's shard-awareness works by binding a shard-specific local (source) port for each connection (
localPort % shardCount == shardId). When connections pass through an NLB the NLB terminates TCP and opens a new connection to ScyllaDB with its own ephemeral source port — the shard hint is silently lost.When the NLB is configured with Proxy Protocol v2 it prepends a binary header to every connection it opens to ScyllaDB carrying the original client source IP and port. ScyllaDB reads the header and uses the original source port for shard routing, restoring shard-awareness end-to-end.
The driver's role is only to declare that PP2 is in use (
withProxyProtocol(true)); the actual header is injected by the NLB. The test infrastructure (TcpProxy/NlbSimulator) simulates this injection so the end-to-end path can be exercised without a real NLB.Test plan
mvn test -pl core)MapBasedDriverConfigLoaderTest.should_fill_default_profile_like_reference_filepasses (addedCLIENT_ROUTES_PROXY_PROTOCOLtoTypedDriverOptionandOptionsMap)ClientRoutesIT.should_use_shard_awareness_through_pp2_nlb— requires ScyllaDB Enterprise ≥ 2026.1 with PP2 configuredFixes: DRIVER-391 ✔️
🤖 Generated with Claude Code