feat: support env vars in heartbeat - #8064
Conversation
Add `heartbeat_env_vars` config option for datanode and frontend. When configured, the specified environment variable values are read at startup and sent to metasrv in every heartbeat via the `extensions` map. Metasrv extracts and stores them in `NodeInfo` for use in routing decisions (e.g. AZ-aware region placement). - Add `EnvVars` helper in `common/meta/src/datanode.rs` following the existing `GcStat` extension pattern with `into_extensions`/`from_extensions` - Add `env_vars: HashMap<String, String>` field to `NodeInfo` in `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat - Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`, `FrontendOptions`, and `StandaloneOptions` - Inject env vars into heartbeat `extensions` in both datanode and frontend heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`) - Extract env vars from `req.extensions` in all three metasrv `CollectXxxClusterInfoHandler`s - Update `NodeInfo` construction sites in `meta-client`, `discovery/lease.rs`, and `standalone/information_extension.rs` - Update expected TOML output in `tests-integration/tests/http.rs` - Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
github-actions
Bot
added
size/S
docs-not-required
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism for nodes to report specific environment variables to the meta service via heartbeat messages. Key changes include the addition of a heartbeat_env_vars configuration option, the EnvVars utility struct for managing these variables, and updates to the NodeInfo structure and heartbeat handlers to process and store the reported data. Feedback suggests refactoring duplicated extraction logic in the cluster info handlers into a shared helper function and optimizing the Datanode heartbeat loop by pre-serializing the static environment variables to avoid redundant JSON serialization on every iteration.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds support for configuring heartbeat-reported environment variables across node options, serializing them in heartbeat extensions, and persisting them into metasrv NodeInfo. This fits into the existing cluster-info/heartbeat pipeline by extending node metadata without breaking older serialized NodeInfo records.
Changes:
- Added
heartbeat_env_varsconfig to datanode, frontend, and standalone options, including env-list parsing from environment overrides. - Added
EnvVarsheartbeat-extension encoding/decoding and persisted reported values into metasrvNodeInfo. - Added backward-compatibility/default-field updates plus config-loader and serialization tests.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests-integration/tests/http.rs |
Updates integration test config fixture with the new standalone option. |
src/standalone/src/options.rs |
Adds standalone config field and forwards it into derived frontend/datanode options. |
src/standalone/src/information_extension.rs |
Extends standalone NodeInfo construction with the new env_vars field. |
src/meta-srv/src/handler/collect_cluster_info_handler.rs |
Extracts env vars from heartbeat extensions and stores them in cluster NodeInfo. |
src/meta-srv/src/discovery/lease.rs |
Adjusts tests for the expanded NodeInfo struct. |
src/meta-client/src/client/util.rs |
Adjusts test helper/default NodeInfo construction. |
src/meta-client/src/client.rs |
Fills default env_vars for metasrv node conversions. |
src/frontend/src/heartbeat.rs |
Reads configured env vars and includes them in frontend heartbeat extensions. |
src/frontend/src/frontend.rs |
Adds frontend config field and env-list parsing support. |
src/datanode/src/heartbeat.rs |
Reads configured env vars and includes them in datanode heartbeat extensions. |
src/datanode/src/config.rs |
Adds datanode config field and env-list parsing support. |
src/common/meta/src/datanode.rs |
Introduces EnvVars extension type plus serialization tests. |
src/common/meta/src/cluster.rs |
Adds persisted env_vars to NodeInfo with backward-compatibility tests. |
src/cmd/tests/load_config_test.rs |
Adds config-loader coverage for heartbeat_env_vars from environment variables. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Why does Metasrv require these env vars, or why do Datanodes and Frontends need to submit them to the Metasrv? |
We will implement a affinity allocator for regions that selects datanodes according to their env vars, like availability zone. |
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
38a7283 to
d8e87e7
Compare
|
@codex review |
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Summary
heartbeat_env_varsconfig for datanode, frontend, and standalone, including env-list parsing from environment overrides.NodeInfo.Test Plan
cargo test -p cmd --test load_config_test test_load_heartbeat_env_vars_from_env