Archive for the ‘Troubleshooting’ Category
I’ve mentioned regular expressions throughout a few my articles over the last few years, usually as part of a configuration. However something I wanted to throw out there, is the fact you can use regular expressions with show commands. This can save quite a bit of time when bouncing between devices all day.
A few examples that I commonly use:
Let’s say you want to easily to pick out and view individual routes in the routing table or BGP table, here I used the underscore “_” regular expression to pick out only the route I wanted. This also works quite well to quickly see the default route.
Something I got tired of real quick, was checking interface errors since we all know issuing “show int” displays an awful lot of information, here I used the pipe “|” to look for multiple strings.
Let’s also say you need to quickly see the status of multiple interfaces on a chassis or stackable switch, we can use the brackets “[” “]” to look at a range of numbers, so you can easily view a particular interface across the stack or chassis, and/or even to see a range of interfaces across the stack/chassis. (I don’t have a functional stack in my lab yet but that RegEx would work across the first 2 stack members)
Now regular expressions are very powerful and there are quite a few I did not mention here. I highly recommend taking some time to go over them and see you can come up with. Feel free leave those findings in the comments below!
You can read more about the RegEx’s here.
For years I’ve been a fan of the reload in command it has always been a useful safety net when making changing that could essentially remove my ability to manage the router. Only recently have I found a feature that will actually roll back the configuration changes I make during a session without the need to reload the router! I don’t know about you but this is just an awesome feature due to the fact it is much less intrusive than the old way of reloading a routing and waiting for it to boot back up. Let’s quickly review this feature:
First we will need to configure a configuration archive, this is actually a prerequisite of the feature want to utilize for reverting our configuration.
The above configuration simply does the following:
- Keeps a copy of the configuration backup on the local flash card with the directory ‘Archives’
- Keeps the last 8 copies of the configuration
- Takes a copy of the configuration when it is saved (Either using the wr mem command or copy run start)
- The configuration will also be saved automatically every 525600 minutes. (This is entirely optional I just included it)
And I used the following commands:
Now, that the configuration is enabled we can start using this configuration revert feature.
To use this feature all you have to do is use the following command when entering configuration mode config t revert timer x once you enter this command it take a backup of the current configuration of the router and places you in configuration mode:
If you try to utilize this feature without first configuring your config archive:
Now you can make any changes that you need. If you do not confirm your changes when you are finished the configuration will be rolled back to the snapshot taken.
You will want to enter the command config confirm to keep the router from rolling the configuration back assuming the change was implemented successfully:
I would like to add, it is possible to enter config with the revert feature, make your changes save the configuration to the start-config and then not confirm your changes. This will cause the running-config to revert back to its previous state but the startup-config will contain any changes made. So you have to be careful with this feature.
Now, I’ve been trying to beat this feature up in my lab all day, and so far it has not been perfect and I’ve seen some errors rolling back for some Frame-Relay configurations. I’ve went as far as to upload a router config from one of INE’s labs and then enter configuration mode with the revert feature enable and then post an entirely different router’s configuration over the existing configure just to see how the revert feature works. So far with the exceptions of some frame relay features it has been solid. It looks like I will start incorporating this new method into my normal day-to-day operations now I depending on the change I might still put in a reload in since you can’t be too careful especially if you are working devices that are in remote datacenters or located physically across an ocean. At least until I start feeling more confident with this feature.
By default syslog run over UDP port 514, UDP as well all know is unreliable. Now lets say you have a couple of core devices and you wanted to ensure the syslog messages from these devices successfully arrived to your syslog server or NMS well in that case I would say your best bet would be to configure syslog to use TCP to send syslog messages to that destination. This is conveniently configured in on simple line:
So now Syslog messages to 10.1.1.1 will be handled by TCP over port 8080, with that said you can even specify a different port number if you see it necessary, just remember to configure your syslog server to accept the syslog messages over the specific port that you can configure.
You can also use this configuration to change the default UDP port used by syslog if your environment calls for it. So instead of the default port UDP 514, syslog messages will be sent over UDP port 5514 in this example:
I while back I mentioned it is possible to debug a single IPSec tunnel using crypto conditions, this functionality also extends outside of crypto conditions. IOS routers offer the functionality to create debug conditions and limit debug output to specific interfaces, ip addresses, and more see the following list:
You can also stack these debug conditions on top of each other:
You will see as you specify additional conditions it tells how many conditions have been set. Issuing the command show debug condition will display a list of debug conditions that been set.
One thing you will want to keep in mind if you do not remove these conditions is when you finished troubleshooting your problem you should remove these debug conditions are else they will affect any other debugs you run in the future. You can remove these debug conditions by issuing the command no debug condition. One thing to keep in mind is that you have the ability to remove the debug conditions in any order, using the above configuration as an example lets say you want to keep seeing debug messages pertaining to IP address 192.168.1.254 and also pertaining to any interface and not just when it includes interface fa0/0, you can issue the command the no debug condition condition 1 or no debug condition interface fa0/0 depending which method you find simplier.
While conditional debugging is a very powerful tool when troubleshooting, you will still want remember that debugging can be very processor intensive and still might hinder performance on a production router, after all the debug is still running using debug conditions just limit the messages you will see.
The fact that Windows tracert operates differently from the traceroute command in Cisco devices will almost always lead to an interesting discussion. Mainly because many people are not aware of the real differences between these two utilities, they are merely aware of the spelling differences and how could you blame them both commands give you very similar results and perform the same function.
Let’s look at a tracert from a Microsoft Windows workstation:
What to know here is the Window’s tracert utility is relying on ICMP echo requests. (So this is nothing more then an extension of a simple ping)
The other key difference between Windows and Cisco, is when the destination is reached it replies back with an ICMP echo reply:
Now when we look at traceroute utility from a Cisco device:
Now the packets:
What we see here is a UDP packet with a destination port of 33434 (The source port is almost always random), not an ICMP echo packet.
Now with Cisco the destination will not answer back with reply packet but surprising enough a destination unreachable packet. See below (Notice the source of the IP packet, it’s the destination of our traceroute)
So to recap, Windows Tracert utility relies on ICMP Type 8 (Echo Request) and Type 0 (Echo Reply) packets, while Cisco replies on a UDP probe packet with a destination port of 33434, and ICMP Type 3 (Destination Unreachable) packet.
Apparently Cisco switches have the capability of performing TDR tests on cables to test cable length and pin-outs. This is something I stumbled upon while browsing through the Cisco Learning Network and I just had to give it a whirl. After all sooner or later you are going to run into a cable issue, it’s just a matter of time. Whether it was not crimped, punched down correctly, or someone ignored the IEEE standard and made a cable too long. The bottom line is some user somewhere will not connect to the network and then it will be up to you to troubleshoot why (and to fix it).
To test a cable simply issue the following command: test cable-diagnostics tdr int %IntefaceType/Number% from privileged mode, and include the interface in question.
After a few seconds issue the following command to see the results: Sh cable-diagnostics tdr int %IntefaceType/Number%
The results are as follows:
Pretty straight forward, it tells you the interface the cable is plugged into, the speed of the connection, the local and remote pair of the cables, cable length (notice the “+/-” of a few meters), and finally the status.
Here is a description of the results from Cisco’s website (Found here):
This command is currently available many different switch models 2900’s, 3500’s, 3700’s, and further (Including both 10/100 MB and 1 GB ports). I would also like to mention if you perform this test on a live port, communication is not interrupted at all. The Cable Length measurement is actually very accurate, don’t let the “+/-” factor confuse you, I’ve ran this on multiple cables and the length calculation is right on the mark. This is a definitely a nifty feature to keep in your when you run into those wired connectivity issues.