I recently ran across an interesting error with one of my Static Virtual Tunnel Interface (SVTI) point-to-point links running IPSec/IKEv2. The link runs between Maryland and Alaska over an independent ISP. This past weekend, I received a notification from my SolarWinds server telling me that the tunnel interface, and its associated crypto, had gone down. When I logged in to see what was up, I was greeted with the following output when I ran a “show log” on my Cisco 2901 router running 15.5(3)M1 code:
000282: Jan 31 2016 01:12:09.930 UTC: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel64621 – looped chain attempting to stack
000283: Jan 31 2016 01:12:13.418 UTC: %TUN-5-RECURDOWN: Tunnel64621 temporarily disabled due to recursive routing
000284: Jan 31 2016 01:12:13.418 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64621, changed state to down
000285: Jan 31 2016 01:12:13.418 UTC: %OSPF-5-ADJCHG: Process 64621, Nbr 126.96.36.199 on Tunnel64621 from FULL to DOWN, Neighbor Down: Interface down or detached
000286: Jan 31 2016 01:16:53.891 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64621, changed state to up
000287: Jan 31 2016 01:16:54.335 UTC: %OSPF-5-ADJCHG: Process 64621, Nbr 188.8.131.52 on Tunnel64621 from LOADING to FULL, Loading Done
As you can see from the time line above, things were down for just under 5 minutes in all. What I could not figure out though, was why I was getting an error that indicated that I had “recursive routing”. It made no sense. This tunnel, and the associated crypto, had been up and operational for over 2 months and no changes had been made. If there was a recursive routing issue, it surely would have manifested itself long ago and OSPF (which is running over the tunnel) would never have come up had that been the case. I did what I often do, and that was to visit the Cisco Support Forums to see if anyone else had run across this problem.
I was very fortunate to run across a post by Richard Burt (who is a prolific and excellent source of information in the Cisco Support Forums) and he pointed out that, for whatever reason, you will see errors similar to these (and especially the first line #000282) when a physical interface or the ISP has an issue somewhere in the path. Ironically, we had just migrated (3 months ago) from dedicated T1s for this link to a provider MPLS L3 VPN solution. I reached out to the provider who quickly pointed out that they had done a JUNOS upgrade on their MX-series router in Seattle that links us into Alaska and that they had to reboot – BINGO! Problem solved! Here is the link to the original post on the Cisco Support Forums:
So, hopefully you never run into this issue, but if you do you now know what one possible cause (and a validated one at that) might be…especially if you are going across a provider/ISP network. Okay, that is all on this one and best of luck from the dark side….