Interface administratively shut down

show ip interface brief

Make sure that the interface is not administratively shut down.

 

Bring up the interface using the no shutdown command, if needed.Use the show interface command to verify that the interface is up.

RIP not enabled on the interface

show ip rip interface

Verify that RIP is enabled for the interface.

Enable RIP on the interface

Interface configured as a passive interface

Make sure that the interface is not configured as a passive interface using the show run command

If the interface is configured as passive, give the no passive interface command

RIP advertisements not sent and received on the interface

Use a packet sniffer (such as Ethereal or TCP dump) or log messages to verify the RIP advertisements. To turn on logging:

One router configured as RIPv1 and the other router as RIPv2

Configure the router running RIPv2:

IBGP neighbors not coming up

BGP sessions not formed

IBGP neighbors stuck in active/idle state

Check for IP reachability between the BGP neighbors using ping command. If ping fails, configure IGP in the network for IBGP reachability or configure static route.

If IP reachability is there, check for the BGP configuration on both sides. Ensure that the AS numbers on both sides (peers) and the local AS / remote AS are same for IBGP. Use “show run bgp” to verify the configuration.

 

Verify that the bgp router ids are different on both sides using “show run bgp” or show ip bgp Summary” command.

 

If the IBGP peering is over any address family other than IPV4/unicast, explicit neighbor activation need to be configured for that address-family.

Eg:- neighbor < peer address> activate (for address-families other than default/ipv4 unicast)

 

If configurations are correct, check for tcp connection status using “netstat –tpln “command at shell prompt. If tcp connections are not in CONNECTED State, check whether the BGP neighborship is configured over loopback interface or normal/physical interface.

 

If the neighborship/peering is over loopback interface on both sides, then explicit “neighbor <peer loopback address> update source loopback” need to be configured on both peers. If loopback interface is used only on one side (peer), the above configuration need to be applied on the other side (peer).

 

If bgp md5 authentication is enabled, ensure that the passwords match on both sides. Verify that “debug ip bgp “does not throw any message like “No md5 digest ****“ or “Invalid md5 digest “.

EBGP neighbors not coming up

EBGP sessions not formed

EBGP neighbors stuck in Active/idle state

Check for IP reachability between the BGP neighbors using ping command. If ping fails, verify IP address config and configure static route if the ebgp peers are not directly connected.

 

Verify the EBGP config is correct using the “show run bgp” command. For EBGP the AS numbers are different on both side and the local AS/remote AS should be reverse on both peers.

 

Verify that the bgp router ids are different on both sides, using “show run bgp” or show ip bgp Summary” command.

 

If the EBGP peering is over any address family other than IPV4/unicast, explicit neighbor activation need to be configured for that address-family.

Eg:- neighbor < peer address> activate (for address-families other than default/ipv4 unicast).

 

If configurations are proper, check for tcp connection status using “netstat –tpln” command at shell prompt. If tcp connections are not in CONNECTED State, check whether the BGP neighborship is configured over loopback interface or normal/physical interface.

 

If the neighborship/peering is over loopback interface on both sides then explicit “neighbor <peer loopback address> update source loopback” need to be configured on both peers. If loopback interface is used only on one side (peer), the above config need to be applied on the other side (peer).

 

If EBGP neighbors are not directly connected, configure the bgp multihop config with appropriate ttl value. The “ttl“ value should be greater than or equal to number of hops, the bgp peer is away.

Eg: neighbor < peer address> ebgp-multihop <ttl value>.

 

If bgp md5 authentication is enabled, ensure that the passwords match on both sides. Verify that “debug ip bgp “does not throw any message like “No md5 digest ****“ or “Invalid md5 digest “.

EBGP session should come up and verify the EBGP neighbors are in Established state using the command “Show ip bgp summary” or “show ip bgp neighbors”.

BGP peering flaps

BGP session flaps

BGP establishes and drops

Check for the BGP peer flap cause/reason from the notification field in “show ip bgp neighbor < peer address>.

If the reason is “Hold timer expiry” then keepalives are not received/ processed.

If keepalives are not received, check the peer router “show ip bgp summary” and verify that the sent msg field is getting incremented.

 

If Keepalives are received but not processed then keepalives are getting stuck behind the Update messages. In other words the update messages are taking longer time to process or the system is configured beyond the claimed BGP limits.The keepalives getting queued can be verified from the “show ip bgp summary” output In Q/OutQ field.If the system is configured with in the scalability limits, please call up support team for help.

 

If the reason is “Administer configured” then the administrator has modified the fields which could trigger BGP session reset. Eg:- BGP Graceful restart configuration.If the flapping is continuous, check for interface flap, bad connectors and bgp process unstable at other end and traffic and rate limiting parameters. Check for interoperability issues too.

 

If the reason is “Malformed packet” then BGP has received a invalid packet from the peer. This is mostly caused by oversized BGP packet as BGP TLVs are variable length. Please call up the support team for help.

 

If none of these, then the issue could be PATH MTU related.Try pinging the peer with various size packet like above 1500 bytes etc. Normal ping succeeds but extended ping fails.Trouble shooting the path MTU issues is beyond the scope of this document.Please call up the support team.

Missing routes

Routes not seen in BGP table

Irregular network connectivity

For IBGP peers, the topology should be fully meshed if no Route Reflectors are configured. If Route Reflectors are used, ensure IBGP is established between all iBGP peer and the RRs and between the RRs.

If confederations are configured to avoid IBGP mesh, ensure that the confederation configuration is proper.

 

Route origination issues - If BGP is not originating the route and If the route is injected via “network prefix” command, Verify that the exact match (not best match) for the route exists in the RIB (IP RIB) using the “show ip route | include prefix “. If exact match does not exist, add a static route for the exact prefix and verify the static route is active in the RIB. BGP should originate the route now.

 

Aggregate route origination issue - If route aggregation is configured and aggregate route is missing in BGP table (“Show ip bgp”) but available in IP RIB (“show ip route”), then the configuration is not complete. For route aggregation, route needs to be there in BGP table.Add the aggregated route in BGP via the “network <aggregate prefix/mask>. Now the aggregated route can be seen in the “Show ip bgp” command also.

 

Originator/cluster id collisions - If Router reflectors are used and topology is proper, still routes not populated. The problem could be clashing cluster ids and router/originator ids. To debug this issue, check for the BGP table from the originator of the route using “show ip bgp “

Check whether the route is advertised to the peer using the “show ip bgp neighbor <peer address> advertised-routes” command.

Check whether the route is received at the peer using the “show ip bgp neighbor <peer address>“ at the peer router.

If route is not received, check for the access-list configured on the peer router.

If access list are also permitting the route, enable bgp debugs and clear the bgp session using “clear ip bgp <peer-address> out “and figure out the reason for the denial of the route.

If the reason for route ignore/drop is “originator id same as router id” Change the router ids in one of the router.

If the reason for route ignore/drop is “reflected from same cluster” then the All RRs are not peered with each other. Ensure ibgp peering between all RRs.

 

Filtering issues - If Routes are not seen with proper configuration, look for the filters/route-maps configured at the inbound and outbound of each peer. Verify that there are no typos in the filters or in the expressions used for filtering.Verify that access-list configured are for exact match.

 

Community config/mismatch issues – verify the configuration has “neighbor <peer address> send community“ to advertise the community strings. Ensure that the community match strings configured via route-map are matching/correct/proper.

 

IBGP meshing issues – Routes are marked as valid and internal but not marked as the best in the “show ip bgp” output. The nexthop for the route (mostly ebgp route) is not reachable and hence not advertised to peer. Configure static route or IGP for nexthop reachability.Another reason for missing route in an IBGP network is that not fully meshed. Ensure that IBGP is fully meshed or use RR + peer group config for ease of maintenance

High CPU utilization

In case of recursive BGP routes, ie where BGP nexthop is also learned via BGP, recursive look up may end up in invalid/illegal. IT is advisable to learn the BGP Nexthop via IGP also.

Route flapping could be one cause for High CPU utilization.

Routing loops caused by lack of IBGP meshing. Use RR or ensure IBGP full meshing.

OSPF not enabled on the interface or the corresponding interface is down.

Use network command to enable ospf on a particular network address.

Do “no shutdown” on the corresponding interface.

Use these commands:

OSPF Interface network address mismatch or subnet mask mismatch

Use these commands to find the output/find the mismatch:

Hello packet attribute mismatch I.e. interface hello interval or dead interval mismatch

Use these commands to find the output/mismatch:

OSPF interface configured as passive interface. Passive interfaces will not send hello packets.

Use these commands:

OSPF option field mismatch i.e. Ospf External Routing Capability mismatch due to stub area mismatch.

Configure same stubby-area.

Use these commands:

OSPF gets struck in two-way state when there is no DR/BDR elected i.e. both routers configured with priority 0

One router should be configured with a priority other than 0.

Use these commands:

OSPF can get stuck in Exstart/exchange state when the maximum transmission unit (MTU) do not match

Use these commands:

OSPF network type mismatch can cause timers (hello-interval) mismatch

Use these commands:

OSPF same router-id and loopback address

Use these commands:

 

OSPF area mismatch for virtual-link

Use these commands:

 

OSPF virtual-link router-id mismatch causing virtual-link down

Use these commands:

 

OSPF not enabled on the interface or the corresponding interface is down causing vlink down

Use these commands:

 

OSPF Multi-Area Adjacency happens only if the adjacency over the primary link is up.

Use these commands:

 

Multi-area configuration mismatch, such as area-id or process-id mismatch.

Use these commands:

 

Multi-Area link remains down, if the Router is not an "ABR"

Use these commands:

 

Multi-Area link will be removed if the primary link is down

Use these commands:

 

For multi access networks, it is required to specify the neighbor's interface address.

 

Multi-area configuration mismatch such as when the neighbor's address does not match any of the OSPF networks or the neighbor’s address is incorrect.

Use these commands:

 

Multi-area adjacency not applicable for the area on which virtual-link is enabled and vice-versa.

Use this command:

 

Daemon restarted after maximum router dead interval of all OSPF interfaces on restarting node

Use this command on restarting node:

 

Capability mismatch between restarting and helper node

Enable capability lls in router node.

Use these commands on both nodes:

 

Configurations not saved before restart

Use any one of these commands and check the saved file in /usr/local/etc:

 

Two routers on same segment get restarted at same time

Use these commands on restarting node:

 

Helper router got its LSDB changed while helping

Use these commands on restarting node:

 

“Resync timeout timer” is not sufficient to sync the whole database

Use these commands on restarting node:

 

OSPF not enabled on the interface or the corresponding interface is down

Use network command to enable OSPFv3 on a particular network address. Do no shutdown on the corresponding interface. Use these commands:

 

Hello packet attribute mismatch i.e. interface hello interval or dead interval mismatch

Use these commands to find the output/mismatch:

 

OSPF option field mismatch i.e. Ospf External Routing Capability mismatch due to stub area mismatch

Configure the same stubby-area. Use these commands:

 

OSPF gets stuck in two-way state when there is no DR/BDR elected i.e. both routers configuration with priority 0

One router should be configured with a priority other than 0. Use these commands:

 

OSPF can get stuck in Exstart/Exchange state when the maximum transmission unit (MTU) does not match

Use these commands:

 

OSPF network type mismatch can cause timers (hello-interval) mismatch

Use these commands:

 

OSPF same router-id and loopback address

Use these commands:

 

Ospf area mismatch for virtual-link

Use these commands:

 

OSPF virtual-link router-id mismatch cause virtual-link down.

Use these commands:

 

OSPF vlink Global remote address is NULL causing virtual-link adjacency down.

Use these commands:

 

OSPF not enabled on the interface or the corresponding interface is down causing vlink down.

Use these commands:

 

Incorrect VRRP States

Make sure the interfaces are up and running by using the show interface command. If the interface is down, give the no shutdown command in interface mode to bring up the interface.

Or

Use the ifconfig IFNAME up command to bring up the interface.

 

Make sure the interface has an IP address in the that matches the subnet of the VRRP session’s virtual-ip.

If not exist, add the ip address in interface mode using the command “ip address A.B.C.D/M”

The configured IP address is displayed in the output of “show running-config interface <IFNAME>”

 

Make sure that both VRRP routers can reach each other by pinging.

If both routers cannot reach each other, check the network connections for the default Master and default Backup routers.

 

Check the advertisement interval on Master and Backup routers. The advertisement interval must be the same on both. The default advertisement interval = 1 second.

 

Use the advertisement-interval command in router mode to configure the advertisement interval.

BFD session not created

Use the show run command to make sure that the interface is not administratively shutdown. If shutdown is configured, remove this configuration with the no shutdown command.

Use the show interface command to make sure that the interface is up.

 

Check if minimum BFD configuration exists for the respective client. Client list includes static route, RIP, OSPF, BGP, ISIS. For configuration refer BFD command reference guide.

 

Check if BFD is administratively down. If BFD is admin-down, session will show state as Admin-Down for 1 minute and then session information is deleted. No session will be seen after that.

 

The next hop for which BFD is enabled should be reachable. For example in OSPF, OSPF neighbor adjacency should be formed for BFD session to be created to monitor the neighbor next hop. This applies to all clients.

 

If configuration is as expected, then enable BFD debug logs and check if packets are received, transmitted for that session. To turn on logging, enter the debug bfd all command

BFD session down

Check the diagnostic reason which can be Neighbor Signaled Down (Remote admin Down) or Control Detect Expiry (non-receipt of hello packets from neighbor for detect multiplier time).

Expected adjacencies not formed or not seen in the output of show clns neighbors

 

The interfaces might have been administratively shut down

The IS-IS configurations are incorrect.

The levels configured on the interfaces might be mismatching

The IP subnet configured is incorrect.

Check for link failures. We can verify whether all the interfaces on which IS-IS adjacency is enabled are in UP state. This can be confirmed by checking the output of sh ip interface brief command. Once the state of the interface is verified to be in UP state, do a ping to the other end and check the connectivity is proper. If the ping fails, that means the physical connectivity problem exits, which should be resolved before going further.

 

If the link is fine, check the IS-IS configurations. IS-IS is enabled in two steps, first is creating isis process as shown below:

 

router isis

net 49.0010.0000.0001.00

 

A misconfigured NET command could also lead to IS-IS adjacency problem. Next step is to attach this process to the appropriate interfaces with command ip router isis.

 

By default, IS-IS router processes have Level 1-2 capability. We can configure IS-IS routers to be level-1 only or level-2 only by using the is-type command. If the levels are mismatching between two directly connected routers adjacency is not formed. By correcting the levels the adjacency issue can be resolved.

 

If the directly connected routers are not present in the same IP subnet, adjacencies are not formed. In this case, the hellos received will be rejected as the interface addresses are not in same subnet. Hence make sure that the interfaces are configured in the same IP subnet.

 

The system ID configured is unique throughout the network. If same system ID is configured for any two IS-IS routers adjacency will not be established. By enabling debug logs we can find the interface which is sending hellos with duplicate system ID. Hence make sure that system IDs are different across the network.

The adjacency state of some or all neighbors is struck in INIT state in show clns neighbors

 

Authentication problems.

The MTU (Maximum Transmission Unit) mismatch has occurred.

If IS-IS authentication is enabled, first resolve this issue. If IS-IS authentication is not enabled, the problem might be with mismatched MTUs. First verify the authentication configurations on the routers. Make sure that proper authentication method is configured and the passwords are consistent. Sometimes, authentication may be enabled on only one side in which case the other side will not be able to process the hellos and hence the neighbor state will be stuck in INIT state.

 

If the authentication problem is resolved and still the neighbor state is stuck in INIT, the possible cause will be because of mismatched MTU. The show isis interface command displays the MTU size of the link. You can also enable debug logs and check the MTU of the packets being sent on the interfaces.

Route advertisement

Most route advertisement problems are caused because of incorrect configurations. If some routes are missing, use the show commands mentioned above to find more information about topology and LSPs. Using the knowledge about topology you can narrow down the problem to a single router.

 

As IS-IS is a link-state protocol, the routers depend on LSPs to learn the topology and routing information. If a route is missing, it might be because the routers did not receive the original LSP or the LSP was received corrupted and hence was purged. In such cases, enabling debug logs for LSPs will help to determine the problem.

Route flaps

Route flaps occur because of unstable links between the routers. As the links flap between UP and DOWN state, they induce SPF calculation resulting in high CPU utilization which might lead to crashes

Route flaps can be identified by checking the Uptime for the neighbor:

 

In these cases physical connectivity should be stabilized to resolve the problem.

 

Route redistribution

IS-IS allows external routes like static, connected, or routes from other protocols to be distributed into IS-IS levels. The main reasons for external routes missing in routing table could be because of the wrong configuration. The source of the route may not be active in which case the external route will not be installed in the routing table.

Verify that the source of the route is active.

 

In case of static routes, the nexthop might not be reachable as the interface is down and hence it is not present in the routing table.

Check the interface state.

 

External routes might be filtered in the route-map or distribute-list at the source, in which case it is the correct behavior to not install the route in the routing table.

Restarting node does not send RR bit in Hello packet when "graceful restart" is performed.

Use this command on restarting node:

Check if "no capability restart graceful" is not configured.

graceful restart capability is enabled by default.

Helper node does not respond with RA bit for the Restart request received from neighbor.

Use this command on helper node:

 

Check if "isis restart helper" is configured on the helper node.

 

Configurations not saved before restart

Use any one of these commands and check the saved file in /usr/local/etc:

 

Restarting router does not remove the stale marked routes from RIB (after coming up).

Use these commands on restarting node:

Check the show command output. Timer should not be running. On successful DB sync, this timer should be cancelled.

The starting router does not send SA bit in hello packet.

Use these commands on starting node:

 

Check whether "isis restart suppress-adjacency" is configured.