SNMP Discovery request timeout

[lukassavukaitis]

I'm getting request timeout errors with SNMP Discovery backend although I can do snmpwalk manually both from within the host and the container itself.

Log messages from orb agent -

orb-agent-1  | Starting orb-agent : /usr/local/bin/orb-agent with args 4
orb-agent-1  | {"time":"2025-11-06T08:33:59.474597851Z","level":"INFO","msg":"backends loaded","backends":{"common":{"diode":{"agent_name":"netbox-snmp-agent","client_id":"${DIODE_CLIENT_ID}","client_secret":"${DIODE_CLIENT_SECRET}","target":"grpc://<netbox_url>:8080/diode"}},"snmp_discovery":null}}
orb-agent-1  | {"time":"2025-11-06T08:33:59.475026724Z","level":"INFO","msg":"no secrets manager specified or invalid type, skipping"}
orb-agent-1  | {"time":"2025-11-06T08:33:59.475052763Z","level":"INFO","msg":"agent started","version":"2.5.0","routine":"agentRoutine"}
orb-agent-1  | {"time":"2025-11-06T08:33:59.475058666Z","level":"INFO","msg":"requested backends","values":{"common":{"diode":{"agent_name":"netbox-snmp-agent","client_id":"${DIODE_CLIENT_ID}","client_secret":"${DIODE_CLIENT_SECRET}","target":"grpc://<netbox_url>:8080/diode"}},"snmp_discovery":null}}
orb-agent-1  | {"time":"2025-11-06T08:33:59.475069162Z","level":"INFO","msg":"registered backends","values":["pktvisor","worker","device_discovery","network_discovery","opentelemetry_infinity","snmp_discovery"]}
orb-agent-1  | {"time":"2025-11-06T08:33:59.475151637Z","level":"INFO","msg":"snmp-discovery startup","backend":"snmp_discovery","arguments":["--host","localhost","--port","8070","--diode-target","grpc://<netbox_url>:8080/diode","--diode-client-id","${DIODE_CLIENT_ID}","--diode-client-secret","********","--diode-app-name-prefix","netbox-snmp-agent"]}
orb-agent-1  | {"time":"2025-11-06T08:34:00.47594288Z","level":"INFO","msg":"snmp-discovery process started","backend":"snmp_discovery","pid":19}
orb-agent-1  | {"time":"2025-11-06T08:34:00.477489584Z","level":"INFO","msg":"snmp-discovery is not ready, trying again with backoff","backend":"snmp_discovery","backoff backoffDuration":"1s"}
orb-agent-1  | {"time":"2025-11-06T08:34:00.85001188Z","level":"INFO","msg":"starting snmp-discovery server","backend":"snmp_discovery","address":"localhost:8070"}
orb-agent-1  | {"time":"2025-11-06T08:34:01.479001163Z","level":"INFO","msg":"snmp-discovery readiness ok, got version ","backend":"snmp_discovery","network_discovery_version":"1.10.0"}
orb-agent-1  | {"time":"2025-11-06T08:34:01.479116349Z","level":"INFO","msg":"managing agent policy from core","action":"manage","name":"policy_1","dataset":"69a76bad-d2f3-4dd2-94c9-9939c8ba3c6a","backend":"snmp_discovery","id":"a49158dd-ff75-53d4-b3d2-78a0d6bf841c","version":1}
orb-agent-1  | {"time":"2025-11-06T08:34:01.479151993Z","level":"DEBUG","msg":"snmp-discovery policy apply","backend":"snmp_discovery","policy_id":"a49158dd-ff75-53d4-b3d2-78a0d6bf841c","data":{"config":{"defaults":{"device":{"comments":"Automatically discovered via SNMP","description":"SNMP discovered device"},"interface":{"description":"Auto-discovered interface"},"ip_address":{"description":"SNMP discovered IP","tenant":"<Some tenant>","vrf":"<Some VRF>"}},"retries":3,"schedule":"* * * * *","snmp_timeout":60,"timeout":600},"scope":{"authentication":{"auth_passphrase":"${AUTH_PASSWORD}","auth_protocol":"SHA","priv_passphrase":"${PRIV_PASSWORD}","priv_protocol":"AES","protocol_version":"SNMPv3","security_level":"authPriv","username":"${USERNAME}"},"targets":[{"host":"<NW_DEVICE_IP_ADDRESS>"}]}}}
orb-agent-1  | {"time":"2025-11-06T08:34:01.481357243Z","level":"INFO","msg":"Received policies","backend":"snmp_discovery","policyCount":"1"}
orb-agent-1  | {"time":"2025-11-06T08:34:01.519915338Z","level":"INFO","msg":"Loaded device lookup extensions","backend":"snmp_discovery","directory":""}
orb-agent-1  | {"time":"2025-11-06T08:34:01.520712705Z","level":"INFO","msg":"Starting policy runner","backend":"snmp_discovery","policy":"policy_1"}
orb-agent-1  | {"time":"2025-11-06T08:34:01.521184325Z","level":"INFO","msg":"policy applied successfully","policy_id":"a49158dd-ff75-53d4-b3d2-78a0d6bf841c","policy_name":"policy_1","backend":"snmp_discovery"}
orb-agent-1  | {"time":"2025-11-06T08:34:01.521228382Z","level":"DEBUG","msg":"Startup of agent execution duration","Start() execution duration":"2.0461759s"}
orb-agent-1  | {"time":"2025-11-06T08:35:00.058195088Z","level":"INFO","msg":"Starting SNMP crawl of targets","backend":"snmp_discovery","targetCount":"1"}
orb-agent-1  | {"time":"2025-11-06T08:35:00.058217994Z","level":"INFO","msg":"Querying targets","backend":"snmp_discovery","objectCount":"13","targetCount":"1"}
orb-agent-1  | {"time":"2025-11-06T08:35:00.058227566Z","level":"INFO","msg":"Scanning","backend":"snmp_discovery","host":"<NW_DEVICE_IP_ADDRESS>"}
orb-agent-1  | {"time":"2025-11-06T08:39:00.07400314Z","level":"WARN","msg":"Error walking ObjectID","backend":"snmp_discovery","error":"request timeout (after 3 retries)","objectID":".1.3.6.1.2.1.1.5.0"}
orb-agent-1  | {"time":"2025-11-06T08:39:00.074052847Z","level":"WARN","msg":"Error crawling host","backend":"snmp_discovery","error":"request timeout (after 3 retries)","host":"<NW_DEVICE_IP_ADDRESS>"}
orb-agent-1  | {"time":"2025-11-06T08:39:00.07406126Z","level":"INFO","msg":"SNMP crawl complete","backend":"snmp_discovery","entityCount":"0","policy":"policy_1"}
orb-agent-1  | {"time":"2025-11-06T08:39:00.074067823Z","level":"INFO","msg":"No entities to ingest","backend":"snmp_discovery","policy":"policy_1"}
orb-agent-1  | {"time":"2025-11-06T08:40:00.044292365Z","level":"INFO","msg":"Starting SNMP crawl of targets","backend":"snmp_discovery","targetCount":"1"}
orb-agent-1  | {"time":"2025-11-06T08:40:00.044326897Z","level":"INFO","msg":"Querying targets","backend":"snmp_discovery","objectCount":"13","targetCount":"1"}
orb-agent-1  | {"time":"2025-11-06T08:40:00.044336175Z","level":"INFO","msg":"Scanning","backend":"snmp_discovery","host":"<NW_DEVICE_IP_ADDRESS>"}
orb-agent-1  | {"time":"2025-11-06T08:44:00.056774427Z","level":"WARN","msg":"Error walking ObjectID","backend":"snmp_discovery","error":"request timeout (after 3 retries)","objectID":".1.3.6.1.2.1.2.2.1.2"}

Agent config -

orb:
  config_manager:
    active: local
  backends:
    snmp_discovery:
    common:
      diode:
        target: grpc://<netbox_url>:8080/diode
        client_id: "${DIODE_CLIENT_ID}"
        client_secret: "${DIODE_CLIENT_SECRET}"
        agent_name: netbox-snmp-agent
  policies:
    snmp_discovery:
      policy_1:
        config:
          schedule: "* * * * *" 
          timeout: 60 # Timeout for policy in seconds (default 2 minutes)
          snmp_timeout: 10 # Timeout for SNMP operations in seconds (default 5 seconds)
          retries: 3 # Number of retries
          defaults:
            ip_address:
              description: "SNMP discovered IP"
              tenant: "<Some Tenant>"
              vrf: "<Some VRF>"
            interface:
              description: "Auto-discovered interface"
            device:
              description: "SNMP discovered device"
              comments: "Automatically discovered via SNMP"
        #  lookup_extensions_dir: "/opt/orb/snmp-extensions" # Specifies a directory containing device data yaml files (see below)
        scope:
          targets:
            - host: "<NW_DEVICE_IP_ADDRESS>"
          authentication:
            protocol_version: "SNMPv3"
            username: ${USERNAME}
            security_level: "authPriv"
            auth_protocol: "SHA"
            auth_passphrase: ${AUTH_PASSWORD}
            priv_protocol: "AES"
            priv_passphrase: ${PRIV_PASSWORD}

docker-compose for orb-agent:

services:
    orb-agent:
        volumes:
            - /opt/orb-agent-snmp:/opt/orb/
            - /opt/orb-agent-snmp/ca.crt:/usr/local/share/ca-certificates/ca.crt
        image: netboxlabs/orb-agent:latest
        env_file:
            - .env
        user: root
        network_mode: host
        command:  "run -c /opt/orb/agent.yaml -d"

Running snmpwalk manually -

snmpwalk -v3 -l authPriv -u $USERNAME -a SHA -A $AUTH_PASSWORD -x AES -X $PRIV_PASSWORD $NW_DEVICE_IP_ADDRESS .1.3.6.1.2.1.1.5.0
SNMPv2-MIB::sysName.0 = STRING: <HIDDEN_HOSTNAME>

I'm testing this against Fortinet FortiGate 1500D firewall.

[leoparente]

hi @lukassavukaitis I think this issue may be related to this other one that we have just fixed: #260

Could you try to validate the fix candidate for that?

Pull the latest develop image with

docker pull netboxlabs/orb-agent:develop

and run discovery using it. We expect the fix candidate to be in snmp discovery version 1.13.1, you can check in the container logs like below

{"time":"2025-12-22T15:54:08.07429322Z","level":"INFO","msg":"snmp-discovery readiness ok, got version ","backend":"snmp_discovery","snmp_discovery_version":"1.13.1"}

if the issue persists, try to run the orb-agent with debug enabled and share a anonymized version of the logs with:

... run -d -c /opt/orb/agent.yaml

[leoparente]

hi @lukassavukaitis the fix candidate is present in latest release as well. Your feedback on that would be appreciated.

[leoparente]

Hi @lukassavukaitis as this issue is staled, I will close it. But feel free to reopen it if the issue persists.