Skip to content

feat: enable shard-awareness with Proxy Protocol v2 in PrivateLink mode#839

Open
nikagra wants to merge 3 commits intoscylladb:scylla-4.xfrom
nikagra:enable-shard-awareness-privatelink
Open

feat: enable shard-awareness with Proxy Protocol v2 in PrivateLink mode#839
nikagra wants to merge 3 commits intoscylladb:scylla-4.xfrom
nikagra:enable-shard-awareness-privatelink

Conversation

@nikagra
Copy link

@nikagra nikagra commented Mar 18, 2026

Summary

  • Adds proxyProtocol flag to ClientRoutesConfig (+ builder, HOCON key, reference.conf default, TypedDriverOption, OptionsMap)
  • DefaultDriverContext parses the new HOCON key and warns (does not fail) when proxyProtocol=true but advanced shard-awareness is disabled
  • TcpProxy and RoundRobinProxy gain a PP2 injection mode: after connecting to the target they write the 28-byte TCP4 PP2 binary header carrying the original client IP and source port
  • NlbSimulator exposes a proxyProtocol constructor parameter and passes the flag to both per-node and discovery proxies
  • ClientRoutesIT adds should_use_shard_awareness_through_pp2_nlb end-to-end test

Background

ScyllaDB's shard-awareness works by binding a shard-specific local (source) port for each connection (localPort % shardCount == shardId). When connections pass through an NLB the NLB terminates TCP and opens a new connection to ScyllaDB with its own ephemeral source port — the shard hint is silently lost.

When the NLB is configured with Proxy Protocol v2 it prepends a binary header to every connection it opens to ScyllaDB carrying the original client source IP and port. ScyllaDB reads the header and uses the original source port for shard routing, restoring shard-awareness end-to-end.

Driver                NLB                   ScyllaDB (shard 3 of 8)
──────                ───                   ──────────────────────
bind localPort=10011  │                     │
(10011 % 8 == 3)      │                     │
connect(NLB:29043) ───┤── new TCP + PP2 hdr─► src_port=10011 → shard 3 ✓

The driver's role is only to declare that PP2 is in use (withProxyProtocol(true)); the actual header is injected by the NLB. The test infrastructure (TcpProxy / NlbSimulator) simulates this injection so the end-to-end path can be exercised without a real NLB.

Test plan

  • All core unit tests pass (mvn test -pl core)
  • MapBasedDriverConfigLoaderTest.should_fill_default_profile_like_reference_file passes (added CLIENT_ROUTES_PROXY_PROTOCOL to TypedDriverOption and OptionsMap)
  • ClientRoutesIT.should_use_shard_awareness_through_pp2_nlb — requires ScyllaDB Enterprise ≥ 2026.1 with PP2 configured
  • Manual verification against a real ScyllaDB Enterprise cluster with NLB PP2 enabled

Fixes: DRIVER-391 ✔️

🤖 Generated with Claude Code

Adds `proxyProtocol` support to `ClientRoutesConfig` so users can declare
that the NLB uses Proxy Protocol v2 (PP2) when forwarding connections to
ScyllaDB. With PP2, the NLB prepends a binary header carrying the original
client source IP and port to each connection it opens to ScyllaDB; ScyllaDB
reads the header and routes using the original source port, restoring
shard-aware routing end-to-end through the NLB.

Changes:
- `ClientRoutesConfig`: add `proxyProtocol` field + `withProxyProtocol()`
  builder method; update equals/hashCode/toString
- `DefaultDriverOption`: add `CLIENT_ROUTES_PROXY_PROTOCOL` enum constant
- `TypedDriverOption` / `OptionsMap`: add typed wrapper and default value
- `reference.conf`: add `advanced.client-routes.proxy-protocol = false`
- `DefaultDriverContext`: parse `proxy-protocol` from HOCON;
  warn (not fail) when `proxyProtocol=true` but shard awareness is disabled
- `TcpProxy` / `RoundRobinProxy`: add PP2 header injection mode —
  after connecting to the target, write the 28-byte TCP4 PP2 binary header
  carrying the original client IP and source port
- `NlbSimulator`: add `proxyProtocol` constructor parameter; pass flag to
  both per-node (`TcpProxy`) and discovery (`RoundRobinProxy`) proxies
- `ClientRoutesIT`: add `should_use_shard_awareness_through_pp2_nlb` test
- `PRIVATELINK.md`: document the PP2 mechanism, header format, configuration,
  and infrastructure requirements

Jira ID: DRIVER-391

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@nikagra nikagra force-pushed the enable-shard-awareness-privatelink branch from ea58854 to 4cf0c72 Compare March 18, 2026 17:01
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a proxy-protocol flag to the driver’s client-routes configuration to support shard-awareness end-to-end when traffic goes through a PrivateLink/NLB that injects Proxy Protocol v2, and extends the integration-test NLB simulator to inject PP2 headers.

Changes:

  • Add proxy-protocol to client-routes config surface area (API builder + HOCON option + typed option + defaults + reference.conf).
  • Parse/validate the new option in DefaultDriverContext (warn when advanced shard awareness is disabled).
  • Extend integration-test proxies (TcpProxy, RoundRobinProxy, NlbSimulator) to optionally prepend a PP2 v2 header; add an end-to-end IT that exercises the flow.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/TcpProxy.java Adds optional PP2 header injection on proxied connections.
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/RoundRobinProxy.java Adds optional PP2 header injection for discovery round-robin proxying.
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/NlbSimulator.java Threads proxyProtocol flag through simulator into per-node and discovery proxies.
integration-tests/src/test/java/com/datastax/oss/driver/core/clientroutes/ClientRoutesIT.java Adds an end-to-end integration test for shard-awareness through a PP2-enabled NLB.
core/src/main/resources/reference.conf Documents and defaults the new advanced.client-routes.proxy-protocol option.
core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultDriverContext.java Reads the new option from config and warns on incompatible settings.
core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java Exposes a typed option for CLIENT_ROUTES_PROXY_PROTOCOL.
core/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.java Adds default false for CLIENT_ROUTES_PROXY_PROTOCOL in driver defaults.
core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java Introduces CLIENT_ROUTES_PROXY_PROTOCOL option path.
core/src/main/java/com/datastax/oss/driver/api/core/config/ClientRoutesConfig.java Adds proxyProtocol field + builder method + equality/toString updates.

@nikagra nikagra self-assigned this Mar 18, 2026
@dkropachev
Copy link

@nikagra , Couple of things:

  1. It does not make sense to have integration tests for it, to much code and too little bennefits
  2. Remove Protocol Proxy v2 from everywhere, except commets to the configuration field
  3. Names, exceptions, logs should be shard-awarness-centric

nikagra added a commit to nikagra/java-driver that referenced this pull request Mar 19, 2026
- Rename proxyProtocol → nlbShardAwareness throughout (ClientRoutesConfig,
  DefaultDriverOption, TypedDriverOption, OptionsMap, DefaultDriverContext).
  The config key path (advanced.client-routes.proxy-protocol) is unchanged.
- Remove PP2 integration test infrastructure (ClientRoutesIT, TcpProxy,
  RoundRobinProxy, NlbSimulator) per reviewer request.
- Fix warning log to say "has no effect" instead of "will be ignored".
- Fix reference.conf comment to match actual behaviour (logs warning,
  does not silently ignore).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@nikagra
Copy link
Author

nikagra commented Mar 19, 2026

@dkropachev All three points addressed in the latest commit (fdd0dd2):

  1. Integration tests removedshould_use_shard_awareness_through_pp2_nlb deleted from ClientRoutesIT, and all PP2 infrastructure reverted from TcpProxy, RoundRobinProxy, and NlbSimulator.
  2. "Proxy Protocol v2" removed from all names/symbols — renamed to shard-awareness-centric names: proxyProtocolnlbShardAwareness, isProxyProtocol()isNlbShardAwareness(), withProxyProtocol()withNlbShardAwareness(), CLIENT_ROUTES_PROXY_PROTOCOLCLIENT_ROUTES_NLB_SHARD_AWARENESS. The config key path (advanced.client-routes.proxy-protocol) and its reference.conf comment block (which is the appropriate place for the PP2 explanation) are unchanged.
  3. Names/logs shard-awareness-centric — warning log now reads nlbShardAwareness=true … NLB shard awareness has no effect without advanced shard awareness enabled.

nikagra and others added 2 commits March 19, 2026 14:51
Adds PrivateLink implementation notes to the project's standard
documentation tree (manual/core/private_link/README.md), following the
same structure as other core-driver feature docs. Content covers:
- The PrivateLink problem and abstract routing-table model
- Architecture overview and implementation deep dives
- Shard-awareness through NLBs using Proxy Protocol v2 (PP2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename proxyProtocol → nlbShardAwareness throughout (ClientRoutesConfig,
  DefaultDriverOption, TypedDriverOption, OptionsMap, DefaultDriverContext).
  The config key path (advanced.client-routes.proxy-protocol) is unchanged.
- Remove PP2 integration test infrastructure (ClientRoutesIT, TcpProxy,
  RoundRobinProxy, NlbSimulator) per reviewer request.
- Fix warning log to say "has no effect" instead of "will be ignored".
- Fix reference.conf comment to match actual behaviour (logs warning,
  does not silently ignore).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@nikagra nikagra force-pushed the enable-shard-awareness-privatelink branch from fdd0dd2 to cd5d3c4 Compare March 19, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants