Adaptive server selection support for multi-stage engine

[dang-stripe]

Is adaptive server selection supported for the multi-stage engine? https://docs.pinot.apache.org/operators/operating-pinot/tuning/query-routing-using-adaptive-server-selection

cc @Jackie-Jiang

[Jackie-Jiang]

cc @gortiz @vrajat @vvivekiyer

Per my code reading:
Adaptive routing is built within the broker routing manager, which is shared across SSE and MSE. It is applied for MSE leaf stage, and if we limit the fanout to leaf stage servers (#16084), we can leverage adaptive routing.
There seems to be gaps on how to reflect back the server stats to adaptive routing for MSE because seems the stats are only gathered for SSE. I think we'll need to revisit that logic.

[ankitsultana]

@dang-stripe fyi: this is supported with the v2 query optimizer. But it is likely that if you have multiple leaf stages, the server selection can vary (we had run into this with our tests).

V2 optimizer will choose the left-most table scan's servers for all intermediate stages.

[Jackie-Jiang]

@ankitsultana Can you elaborate more on this? The gap is actually server reflecting stats when solving MSE queries. Seems right now broker only gathers server stats for SSE queries

[ankitsultana]

Oh yeah that's the same for all MSE query optimizers. I was only referring to worker assignment.

We were seeing adaptive server selector change servers across leaf stages for the same table and had decided to disable it because we prefer keeping a query within a replica or isolation group.

(note that the cluster in question also serves V1 engine queries so I suppose V1 engine was still playing with the server rank list)

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

[jadami10]

We do plan to continue work on this one in the near future

[timothy-e]

Changes:

• Observability
  • Export Adaptive Routing Stats as broker metrics #18134
  • Allow dynamic toggling of adaptive routing metrics export #18286
  • Add pinot_server_mse_queries server metrics #18287
• Bug fixes
  • Add configurable queue size offset to adaptive routing hybrid score #18288
• MSE Enhancements
  • Include MSE inflight_requests in adaptive query stats #18289
  • TODO: Include MSE latency in adaptive routing stats

[timothy-e]

@vvivekiyer @siddharthteotia @jasperjiaguo are any of you still interested in Adaptive Routing? I'd love your expertise on the linked PRs above.