Archive for May 2012
Who’s got the time? NTP – Network Time Protocol
Anyone who has ever had to troubleshoot any type of issue on a Cisco router knows they have to look through log messages, and anyone who has ever done this knows what’s it’s like to look through a log where the time is off by minutes, hours, days, or even years. It tends to add on another layer of complexity in itself & typically you are going to be in this type of situation:
- Ok, I’ve got a problem let’s troubleshoot
- Lets look at the logs
- HHmm those dates/times are all wrong
- Ok let’s issue the sh clock command to see what time the router thinks it is
- Now how far off is the router’s internal clock from the real world current time
- Alright now what what time did the problem occur in the real world and what time did the problem occur in the router’s time zone
- There we go!
It’s not overly complex but it can be a little time consuming and it can cost you a good amount of time depending on the type of issue you are troubleshooting. In my mind this is completely unnecessary and should just be corrected. How do we correct this though? Well we configure our devices to check in with a time server and get the correct time using NTP (Network Time Protocol).
The first thing we will want to do is pick one (or more) of your routers to be the NTP master for the network, and the other routers will connect to this NTP master for the correct time, this will provide us with some type of hierarchical NTP design. Before we configure any clients to check in with the NTP master we decide on we’ll need to get the correct time on the master. To do this we can peer to some of the Stratum 1 public NTP servers. In terms of NTP and Stratum levels, Stratum 1 will be the most accurate time sources, these Stratum 1 servers are directly attached to some type of GPS, WWV, or CDMA device which are stratum 0 devices and are known to have the most accurate time. (Note: Cisco devices do not support the use of these devices if directly attached)
Here is our current topology:
Now let’s configure our router:
From here we configured a timezone for our router, I named the timezone EST I figured I’d keep it simple and set it to -5 which is where the Eastern time zone fits in the world. Then I set daylight saving time with the clock summer-time EST recurring command. After that I start configuring some NTP parameters I start by turning on NTP logging, I did this just for the sake of showing off some of the NTP logging messages. Then I configure the router to update it’s hardware clock with the time it receives from the server with the command ntp update-calendar, the hardware clock runs off an internal battery and will keep the correct time even when the router loses power. Now we configure our router to check in with ntp.you.org this is just one of the stratum 1 servers I pulled off the webpage mentioned earlier.
Now, shortly after I finish with those commands I see some of the NTP logging messages flash through the console:
The messages are simply telling us the clock was reset, and synchronized successfully with the server we configured. We can verify this by looking at the sh ntp status & sh ntp associations output.
You can see from the sh ntp status output, that our time is synchronized, our stratum level is 2, and when the last update was. From the sh ntp associations we can see we are peered with the NTP server 204.9.54.119 (Confirmed with the * before the IP address) and the NTP server we are peering with is using a CDMA device for it’s clock source. Also notice the 127.127.1.1 address, which is the local router.
So now in SW1 and SW2 I am going to issue the following commands:
This tells the other devices to check in with the router we configured as the NTP master, very similar to the commands we issued earlier on the master. We can verify this again by issuing the sh ntp status & sh ntp associations:
Notice there is less information, but it still tells us what we need to know. We see our clock is synchronized and to whom, as well as our stratum level. Our stratum level here is 3, since we got our time from a stratum 2 source and we are one more “time hop” away from the stratum 1 source.
So essentially our NTP design looks like this:
We can do the same thing with even more complex designs:
However NTP would be behave like this:
RFC 1305 goes into NTPv3 much more in depth if you want to read about it, also keep in mind NTP runs off UDP Port 123. (I just had to throw that in there somewhere before I ended this post)
Let’s configure HSRP – Hot Standby Router Protocol.
I covered VRRP a few weeks back, which is a vendor neutral FHRP, but now let’s take a look at HSRP which is more or less a Cisco proprietary version of VRRP.
With the exception of the standby keyword this configuration is almost identical to our previous VRRP configuration, and the joy of this is the fact these commands pretty much mean and do the same thing. Standby 1 specifies our HSRP group ID or instance, Standby 1 ip 192.168.1.254 is the HSRP virtual address. Standby 1 timers 2 7 specifies how often HSRP will send out hello packets every 2 seconds in this case and if hello packets are not received within 7 seconds that HSRP speaker is removed from the HSRP group. Standby 1 priority 110 sets this device’s priority within the HSRP group the default priority is 100, and the device with the higher priority is the device that will be the HSRP active router. Standby 1 preempt delay minimum 15 tells this router to take the role of the active router if it were a standby router or if the router has a higher priority and is entered into a group, it also tells the router to wait 15 seconds before taking back the role of the active router. Those 15 seconds should provide enough time for the router to accept any routing updates from its neighbors and have a fully converged routing table so it does not blackhole traffic when it becomes the active HSRP router again. Standby 1 track 1 decrement 100 tells the router to decrement the HSRP priority by 100 if the tracking status of tracking object 1 goes down, and tracking object 1 is configured to track the line protocol of fastethernet0/1/0.
You can issue the show standby command to verify all the HSRP configurations on the router. Just reading this output will tell you everything you need to know about the HSRP configuration, from this you can verify the following:
- What interface is in which HSRP Group
- State of this particular device in the HSRP group
- The Virtual MAC and IP address
- Unless manually configured the MAC will always start with 0000.0c07.acXX – Where XX is the group ID (Remember MAC addresses are in hex format)
- Hello & holddown timers
- Who the active and standby routers are
- Priorities of the active and standby routers
- Any tracking parameters configured on the HSRP group
Above is some debug output of the a standby router becoming the active router, from the above output you see the last hello packet received was 19:31:21, and at 19:31:28, 7 seconds later the holdtime expires and the local router becomes the active router.
Then I connected the original active router back into the network, and you can see the coup message received from 192.168.1.1 stating it has a higher priority, the coup message is what tells the current active router that another router has joined the group with a higher priority and it’s assuming the role of the new active router.
Onward to CCIE!!
Well, I’ve decided to go down the path and become a CCIE! After all I have my CCNP, CCIP, & CCDP I’ve really only got one more level to go to Expert! So while I am still building my lab, I only need a few more routers, a bunch of WICs, and even more cables I have enough to keep studying and moving forward.
So I’ll be trying to get out even more blog posts covering various topics, or whatever topics I stumble upon while tackling the CCIE (and I’m sure their will be a lot of them)
So here goes nothing! (and I’m sure I’ll be saying that 6+ months later when I’m still studying for this)
Debugging IPSec VPN’s.
As promised I mentioned we were going to go over some debug output from 2 Cisco ISRs establishing an IPSec VPN. Now I’m not going to go over every line in the debug but I’ll touch on some of the things to look out for. Now, from the beginning.
From the first line you can see ISAKMP is enabled and it starts looking for it’s peer (172.17.1.1 in this case), the router realizes it needs to use main mode and it locates the PSK for this particular peer, so right off the bat we know the peer we are establish a IPSec VPN with, along with what PSK/Keyring we are going to be using. Next we see some mention of NAT-T and RFC 3947, this is essential so both devices know if there is any NAT”ing done in between the two devices and if so where it is being done. (If you want a bit more information on that I’d say give RFC3947 a glance over, it tells you why this is done) Next you will see the line New State = IKE_I_MM1 meaning the first packet in the IKE Phase I process has been sent out, following by the router so nicely telling us (with line beginning Main Mode exchange). Next you see the ISAKMP state change to MMO_NO_STATE, this is because we have sent out a packet but we have not received a response yet, so we are unable to continue along with the process. Then in the second to last line we see Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH meaning we received a response from the peer (the second packet in the process).
Now that we received the second packet (which contains the remote device ISAKMP polices) this router will compare the remote peers ISAKMP policies to it’s own, you’ll notice right after it says the atts are acceptable, so our policies match! You’ll also see the last 3 lines mention the lifetime: 86400 this is default ISAKMP lifetime in seconds you will want these to match on both sides of the tunnel, it’s not something to be really concerned about when building VPN’s between two Cisco devices but I would pay attention to it when building VPNs between different vendors.
Next we see the ISAKMP state proceed to MM_SA_SETUP which is another confirmation that the ISAKMP policies match and the 2 peers are going to continue along with the process. Also notice the the progressions of Old State = IKE_I_MM2 New State = IKE_I_MM3 and Old State = IKE_I_MM3 New State = IKE_I_MM4 telling us the third and fourth packets have been exchanged, shortly after that you see the router processes the NONCE payload which is used to generate the DH secret.
Now that we have exchanged the first four packets it starts authenticating the peer using the configured method (PSK in this case), and we see this in the line Sending packet to 172.17.1.1 my_port 500 peer_port 500 (i) MM_KEY_EXCH, and the line Old State = IKE_I_MM4 New State = IKE_I_MM5 telling us the fifth packet has been sent. Right after we see that we received the response packet from the peer in line Received packet from 172.17.1.1 dport 500 sport 500 global (i) MM_KEY_EXCH, then after it processes the payload, we see the line SA authentication status: authenticated, telling us the peer successfully authenticated phase I.
The above output just verifies everything we saw and said in the previous output, it shows us we received the sixth packet (Old State = IKE_I_MM5 New State = IKE_I_MM6), and Main Mode has been complete in the following lines. Input =IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE and Old State = IKE_I_MM6 New State = IKE_P1_COMPLETE.
These next group of lines tell us that phase II quick mode is starting and ISAKMP status is QM_IDLE. Then we see the router sends the first packet in the process and receives the second packet in the quick mode process from the remote device, after that it begins to process the payloads.
Now that we received the second packet from the remote device which contained its IPSec transform-sets the router compares it to its own transform-sets. The router then accepts the transform-set, as long as it matches its own transform-set and, it does. We know this because of the line atts are acceptable and it creates IPSec SAs with the peer (in the line Creating IPSec SAs).
Now, that the IPSec SAs have been established the process is pretty much complete and the IPSec VPN (both phase I and II) is negotiated and formed. There is still some more output displayed. The first 10 of so lines tell us the SPI’s associated with these IPSec peers and the IPSec security association lifetimes, like ISAKMP lifetimes you will want the IPSec lifetimes to match as well. Now there are another dozen of so lines left but the key line in middle of that Old State = IKE_QM_R_QM2 New State = IKE_QM_PHASE2_COMPLETE is the one that tells Phase II quick mode is complete and was successful. As if forming IPSec SAs were not enough.
So there we have the typical debug output of an IPSec VPN, assuming everything is configured properly on both ends, now hopefully after going this and my post from a few weeks ago going over IKE in general you’ll have a deeper understanding of what really goes on when a routers try to form an IPSec VPN.
Passed CCDP!!
After 4 months of getting lost in design guides/books I finally decided to take a shot at the CCDP exam, and passed! I’m sure the experience I gained at worked helped out even more since a good portion of job is network design.
Much like the CCDA it covered the best practice and design for a broad array of different technologies IP addressing, remote access, security, virtualization, multicast, & various other topics. Just from my own experience the difficulty of the questions varied I found the layer 3 routing protocol and IP address questions to be quite simple, what I consider along the lines of just good common sense for any network design. Compared to the data center/virtualization technologies where I found them a little bit more difficult because I don’t typically find myself playing with NX-OS often. I will say however my previous experience managing and maintaining a few VMWare ESX clusters did come in handy.
I am looking forward to a small break before I start going down another path, whether it be CCNP: Security or towards my CCIE: R&S. So for now it’s back to blogging, labbing, working, and living life!
Now to figure out what more I need to complete my CCIE lab!
CCIE or Null! 