BGP with BFD cannot recover from AWS BGP bring down test: incoming conn rejected - no AF activated for peer

[kyleli666]

Background

FRR Version 10.2.1-0~ubuntu24.04.1
Linux 6.8.1-1014-realtime #15-Ubuntu SMP PREEMPT_RT Fri Dec 6 22:24:34 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I created a BGP with BFD to AWS Direct Connection and would like to test the failover to AWS VPN by a test function from AWS "Bring down BGP".
Both BGP and BFD establish well.
However, when testing, the failover works well, but when recovering, BGP keeps Idle and BFD keeps down.

Jan 17 14:21:15 node1 bfdd[3055688]: [SEY1D-NT8EQ] state-change: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] up -> down reason:neighbor-down
Jan 17 14:21:15 node1 bgpd[3055678]: [Q4BCV-6FHZ5] zclient_bfd_session_update: 169.254.66.2/32 -> 169.254.66.1/32 (interface vlan1622) VRF default(0) (CPI bit yes): Down
Jan 17 14:21:15 node1 bgpd[3055678]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Up -> Down
Jan 17 14:21:15 node1 bgpd[3055678]: [HZN6M-XRM1G] %NOTIFICATION(Hard Reset): sent to neighbor 169.254.66.1 6/10 (Cease/BFD Down) 0 bytes
Jan 17 14:21:15 node1 bgpd[3055678]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 17 14:21:15 node1 bgpd[3055678]: [ZWCSR-M7FG9] 169.254.66.1 [FSM] BGP_Stop (Established->Clearing), fd 25
Jan 17 14:21:15 node1 bgpd[3055678]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Clearing established_peers 2
Jan 17 14:21:15 node1 bgpd[3055678]: [HKWM3-ZC5QP] 169.254.66.1 fd -1 went from Established to Clearing
Jan 17 14:21:15 node1 bgpd[3055678]: [ZWCSR-M7FG9] 169.254.66.1 [FSM] Clearing_Completed (Clearing->Idle), fd -1
Jan 17 14:21:15 node1 bgpd[3055678]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Idle established_peers 2
Jan 17 14:21:15 node1 bgpd[3055678]: [HKWM3-ZC5QP] 169.254.66.1 fd -1 went from Clearing to Idle
Jan 17 14:22:08 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer
Jan 17 14:22:31 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer
Jan 17 14:22:33 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer

Current configuration:
!
frr version 10.2.1
frr defaults traditional
hostname node1
log syslog
service integrated-vtysh-config
!
debug bgp bfd
debug bfd peer
debug bfd zebra
!
ip router-id xxx.xx.xxx.xx
!
router bgp 65031
 no bgp ebgp-requires-policy
 no bgp hard-administrative-reset
 neighbor 169.254.44.1 remote-as 64512
 neighbor 169.254.44.1 description BGP VPN tunnel1
 neighbor 169.254.44.1 update-source 169.254.44.2
 neighbor 169.254.55.1 remote-as 64512
 neighbor 169.254.55.1 description BGP VPN tunnel2
 neighbor 169.254.55.1 update-source 169.254.55.2
 neighbor 169.254.66.1 remote-as 65030
 neighbor 169.254.66.1 bfd
 neighbor 169.254.66.1 description BGP Line
 neighbor 169.254.66.1 update-source 169.254.66.2
 !
 address-family ipv4 unicast
  network 169.254.44.0/30
  network 169.254.55.0/30
  network 169.254.66.0/30
  neighbor 169.254.44.1 weight 100
  neighbor 169.254.55.1 weight 200
  neighbor 169.254.66.1 weight 300
 exit-address-family
exit
!
end

Current Workaround

If I do not enable BFD, BGP can recover well, but I believe it is slower when failover in real cases.

Try 1

AI suggests me to config "neighbor 169.254.66.1 activate" under "address-family ipv4 unicast", but it seems default, not show up in "show run" and not solving the issue, and still "no AF activated for peer"

Try 2

I had a very similar setup with VyOS which uses FRR 9.1.1, and that recovers well.
I looked into the frr.conf of this VyOS, and manually update my frr.conf accordingly, and finally neighbor activate is there by vim...

router bgp 65031
 no bgp ebgp-requires-policy
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor 169.254.44.1 remote-as 64512
 neighbor 169.254.44.1 description BGP VPN tunnel1
 neighbor 169.254.44.1 update-source 169.254.44.2
 neighbor 169.254.55.1 remote-as 64512
 neighbor 169.254.55.1 description BGP VPN tunnel2
 neighbor 169.254.55.1 update-source 169.254.55.2
 neighbor 169.254.66.1 remote-as 65030
 neighbor 169.254.66.1 bfd
 neighbor 169.254.66.1 description BGP Line
 neighbor 169.254.66.1 update-source 169.254.66.2
 !
 address-family ipv4 unicast
  network 169.254.44.0/30
  network 169.254.55.0/30
  network 169.254.66.0/30
  neighbor 169.254.44.1 activate
  neighbor 169.254.44.1 weight 100
  neighbor 169.254.55.1 activate
  neighbor 169.254.55.1 weight 200
  neighbor 169.254.66.1 activate
  neighbor 169.254.66.1 weight 300
 exit-address-family
exit

Now, debug log does not show "no AF activated for peer" anymore, but still cannot recover BGP until I remove BFD configuration...

Jan 18 01:27:28 node1 bgpd[3466715]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Down -> Up
Jan 18 01:27:28 node1 bgpd[3466715]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 18 01:27:39 node1 watchfrr[3466696]: [WFP93-1D146] configuration write completed with exit code 0
Jan 18 01:30:09 node1 bfdd[3466725]: [SEY1D-NT8EQ] state-change: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] up -> down reason:neighbor-down
Jan 18 01:30:09 node1 bgpd[3466715]: [Q4BCV-6FHZ5] zclient_bfd_session_update: 169.254.66.2/32 -> 169.254.66.1/32 (interface vlan1622) VRF default(0) (CPI bit yes): Down
Jan 18 01:30:09 node1 bgpd[3466715]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Up -> Down
Jan 18 01:30:09 node1 bgpd[3466715]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 18 01:34:08 node1 bfdd[3466725]: [MSVDW-Y8Z5Q] ptm-del-dest: deregister peer [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default cbit:0x00 minimum-ttl:255]
Jan 18 01:34:08 node1 bfdd[3466725]: [NYF5K-SE3NS] ptm-del-session: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] refcount=0
Jan 18 01:34:08 node1 bfdd[3466725]: [NW21R-MRYNT] session-delete: mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622
Jan 18 01:34:17 node1 watchfrr[3466696]: [WFP93-1D146] configuration write completed with exit code 0
Jan 18 01:34:43 node1 bgpd[3466715]: [JG0WZ-7X009][EC 33554504] 169.254.66.1 unrecognized capability code: 128 - ignored
Jan 18 01:34:44 node1 bgpd[3466715]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 169.254.66.1 in vrf default

Try 3

I finally tried graceful-restart-disble, in case of confliction, but still cannot recover BGP peering, it keeps in Idle.

Question

What could be the possible cause?

[kholderbach]

Exact same issue here. When adjacencies are broken, only way to get BGP peers and BFD peers back is to remove 'neighbor x.x.x.x BFD' from the config. Then put it back and everything is great again. Been looking all over for solutions - haven't found anything so far.

[ton31337]

This should be fixed already for stable + master branches.

[robergeandre]

I did get the same issue using frr 10.3 on Debian 13 peering with Juniper QFX-5120. My BFD will run fine when created but as soon as something goes down, or I do a restart or reload, the BFD session goes down.

Here is my frr config:
frr version 10.3
frr defaults traditional
hostname nd3srv10
log syslog
service integrated-vtysh-config
!
debug bgp neighbor-events
debug bgp updates in
debug bgp updates out
debug bgp bfd
debug bfd peer
debug bfd network
!
router bgp 65100
bgp router-id 10.14.0.17
no bgp ebgp-requires-policy
no bgp network import-check
neighbor fe80::117 remote-as 65513
neighbor fe80::117 bfd
neighbor fe80::117 interface enP4p1s0f0np0
neighbor fe80::117 update-source enP4p1s0f0np0
neighbor fe80::217 remote-as 65513
neighbor fe80::217 bfd
neighbor fe80::217 interface enP4p1s0f1np1
!
address-family ipv6 unicast
neighbor fe80::117 activate
neighbor fe80::217 activate
exit-address-family
exit
!
end

BFD session:
BFD Peers:
peer fe80::217 vrf default
ID: 1895239496
Remote ID: 0
Active mode
Status: down
Downtime: 3 minute(s), 20 second(s)
Diagnostics: ok
Remote diagnostics: ok
Peer Type: dynamic
RTT min/avg/max: 0/0/0 usec
Local timers:
Detect-multiplier: 3
Receive interval: 300ms
Transmission interval: 300ms
Echo receive interval: 50ms
Echo transmission interval: disabled
Remote timers:
Detect-multiplier: 3
Receive interval: 1000ms
Transmission interval: 1000ms
Echo receive interval: disabled

    peer fe80::117 local-address fe80::1 vrf default
            ID: 843952575
            Remote ID: 0
            Active mode
            Status: down
            Downtime: 3 minute(s), 20 second(s)
            Diagnostics: ok
            Remote diagnostics: ok
            Peer Type: dynamic
            RTT min/avg/max: 0/0/0 usec
            Local timers:
                    Detect-multiplier: 3
                    Receive interval: 300ms
                    Transmission interval: 300ms
                    Echo receive interval: 50ms
                    Echo transmission interval: disabled
            Remote timers:
                    Detect-multiplier: 3
                    Receive interval: 1000ms
                    Transmission interval: 1000ms

Does anyone have a clue on this error ?

[ton31337]

Latest 10.3 and 10.4 have a fix for this, could you try?

[robergeandre]

I tried 10.3 I had the issues, but with 10.4.1 it is resolve and it is running great.

Thank You