Skip to content

Dockerfile from "adk deploy agent_engine" ignores "AGENT_GATEWAY_ROOT_CERTIFICATES" build arg, causing runtime SSL handshake failures #6427

Description

@KenTandrian

🔴 Required Information

Describe the Bug:
When deploying an agent to Gemini Enterprise Agent Runtime (fka. Vertex AI Agent Engine) with Agent Gateway enabled (agent_gateway_config.agent_to_anywhere_config and identity_type="AGENT_IDENTITY"), adk deploy agent_engine generates a Dockerfile that does not declare or consume the AGENT_GATEWAY_ROOT_CERTIFICATES build argument passed by Cloud Build.

Consequently, the custom TLS inspection root CA certificate for the Agent Gateway is not installed into the container trust store (/etc/ssl/certs/ca-certificates.crt), causing all outbound HTTPS and gRPC calls (Gemini API, BigQuery, MCP Toolbox, etc.) routed through the Gateway to fail at runtime with SSL certificate verification errors.

Steps to Reproduce:

  1. Configure an agent with .agent_engine_config.json specifying an Agent Gateway:
    {
      "agent_gateway_config": {
        "agent_to_anywhere_config": {
          "agent_gateway": "projects/PROJECT_ID/locations/LOCATION/agentGateways/GATEWAY_NAME"
        }
      },
      "identity_type": "AGENT_IDENTITY"
    }
  2. Deploy the agent using the ADK CLI:
    adk deploy agent_engine --project=PROJECT_ID --region=LOCATION agent_dir
  3. Check Cloud Build logs and observe the unused build-arg warning.
  4. Send a query to the deployed Reasoning Engine that invokes any outbound tool, database query, or model API call.

Expected Behavior:
The Dockerfile generated by adk deploy agent_engine should accept ARG AGENT_GATEWAY_ROOT_CERTIFICATES, write the certificate to /usr/local/share/ca-certificates/, and run update-ca-certificates. Outbound TLS traffic through the Agent Gateway should complete successfully without SSL verification errors.

Observed Behavior:

  • Cloud Build Warning:
    [Warning] One or more build-args [AGENT_GATEWAY_ROOT_CERTIFICATES] were not consumed
    
  • Runtime Error:
    [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
    

Environment Details:

  • ADK Library Version: 2.5.0
  • Desktop OS: macOS / Linux
  • Python Version: 3.11+

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-3.5-flash

🟡 Optional Information

Regression:
N/A

Logs:

Step 13/13 : CMD adk api_server --port=8080 --host=0.0.0.0 --session_service_uri=...
[Warning] One or more build-args [AGENT_GATEWAY_ROOT_CERTIFICATES] were not consumed
Successfully built container

... at runtime ...
google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 failed to connect to all addresses; last error: FAILED_PRECONDITION: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)

Additional Context:
Deploying via the native Vertex AI Python SDK (vertexai.agent_engines.AdkApp via client.agent_engines.create(agent=..., config=...)) works as expected because Vertex AI's managed source package template natively consumes ARG AGENT_GATEWAY_ROOT_CERTIFICATES and executes update-ca-certificates. The issue is strictly isolated to the Dockerfile generated in google/adk/cli/cli_deploy.py (_DOCKERFILE_TEMPLATE).

Possible Fix in google/adk/cli/cli_deploy.py:
Add the build argument and CA update step to _DOCKERFILE_TEMPLATE:

ARG AGENT_GATEWAY_ROOT_CERTIFICATES
RUN if [ -n "$AGENT_GATEWAY_ROOT_CERTIFICATES" ]; then \
      mkdir -p /usr/local/share/ca-certificates && \
      echo "$AGENT_GATEWAY_ROOT_CERTIFICATES" > /usr/local/share/ca-certificates/agw-ca.crt && \
      update-ca-certificates; \
    fi

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions