🚀 Describe the new functionality needed
In run.yaml configurations, the conditional syntax ${env.VAR:+value} works for provider_id (entries with None/empty provider_id are filtered out before validation), but does NOT work for benchmark_id in registered_resources.benchmarks.
When KUBEFLOW_LLAMA_STACK_URL is not set, this config:
registered_resources:
benchmarks:
- benchmark_id: ${env.KUBEFLOW_LLAMA_STACK_URL:+trustyai_garak::owasp_llm_top10}
dataset_id: garak
scoring_functions:
- garak_scoring
provider_id: trustyai_garak_remote
provider_benchmark_id: owasp_llm_top10Causes this validation error:
pydantic_core._pydantic_core.ValidationError: 6 validation errors for StackRunConfig
registered_resources.benchmarks.0.benchmark_id
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
💡 Why is this needed? What if we don't build it?
Upon adding Garak provider , I had this PR blocked
Other thoughts
No response
🚀 Describe the new functionality needed
In
run.yamlconfigurations, the conditional syntax${env.VAR:+value}works forprovider_id(entries with None/empty provider_id are filtered out before validation), but does NOT work forbenchmark_idinregistered_resources.benchmarks.When
KUBEFLOW_LLAMA_STACK_URLis not set, this config:Causes this validation error:
💡 Why is this needed? What if we don't build it?
Upon adding Garak provider , I had this PR blocked
Other thoughts
No response