Cisco 7206VXR FA-GE= port adapter performance
High CPU utilization is not uncommon, especially when a router is struggling to process a packet and punts it between switching processes.
The graphs below show the output from a production Cisco 7206VXR (NPE-G1, PA-GE=, PA-2FE-TX) router which is the primary for an IPv6 HSRP pair. Whenever the primary was active CPU utilisation would hit 100% by midday and the trend followed the working day and therefore the volume of traffic being passed. This would make network monitoring alarms light up as the router became unreachable via the management interface, VTY console sessions would be laggy and SNMP requests were not fulfilled.
Interestingly whenever the secondary router was used which was fitted with the lesser NPE-400 the same problems were not observed.
Cisco TAC were contacted who indicated that bugs CSCta73296 and CSCtj77285 in the currently running IOS would account for the high CPU utilisation and an upgrade to 12.4(24)T7 was carried out. This fixed the problem to the extent that CPU utilisation more accurately followed traffic flow, but still topped 100% for most of the working day.
I continued my own trouble shooting looking in particular at the packet processing processes believing that the high CPU combined with the interface input drops was caused by the buffers being full due to slow switching. In doing so I tidied up some of the routing tables and CEF adjacencies which reduced some of the error counters over the course of the day.
Eventually I came across this document:
Sure enough the upstream link to providers BGP peer was connected to the PA-GE=, so I took a look at:
…and in particular:
|The PA-GE offers an additional choice of high-speed LAN connection to Cisco 7200 VXR and Cisco uBR7246 VXR routers. The PA-GE does not offer wire rate performance on Cisco 7200 VXR or Cisco uBR7246 VXR routers but does offer performance throughput suitable to WAN aggregation applications. Cisco 7200 VXR and Cisco uBR7246 VXR routers, when deployed as WAN aggregation routers, can be integrated into a Gigabit Ethernet campus backbone using the PA-GE.|
This port adapter’s poor switching performance was clearly the cause of the problem. During the next at-risk period the uplink was moved onto one of the Gigabit interfaces on the NPE-G1, and quite quickly the benefits were apparent. The remaining input errors are a result of the unicast RPF running on the interface.