Archive for the ‘Wireshark’ Category
I recently found myself troubleshooting some SNMP connectivity between a particular set of devices and an NMS. Connectivity did not appear to be the problem as IP Connectivity was there and MIB walks were successful, however some interesting errors were still getting reported on the NMS. As I captured the packets to verify this connectivity, I said to myself ‘If only I can see what the NMS was asking for specifically and what device in question was replying back with’. This led me to check out the SNMP protocol settings in Wireshark, I mean Wireshark can de-crypt HTTPS traffic (with the private key) and wireless WPA traffic surely it can de-crypt SNMPv3. Behold it was true!! I was able to de-crypt SNMPv3 packets, and see what was really going on.
To add SNMPv3 information into Wireshark:
Access your Wireshark preferences: Edit -> Preferences -> Protocols -> SNMP
Where you see ‘Users table’ choose edit:
From here we can enter the SNMPv3 settings we need:
- Engine ID
- SNMP USer
- Authentication & Password – MD5 or SHA1
- Privacy & Password- DES, AES-128, AES-192, or AES-256
Once you enter the correct information and choose ‘ok‘ Wireshark will automatically de-crypt any relevant packets.
I feel like this is something I should have known about for a while now, but I supposed I don’t find myself troubleshooting SNMP connectivity too often. Figured I would get the word out there!
Well, I finally took the time to buckle down and take the WCNA – Wireshark Certified Network Analyst. Once I finished up with the exam I can happily say I successfully past the exam which I have to admit is pretty cool. Protocol Analysis is definitely an interesting set of technologies to learn & know, it is also extremely beneficial for troubleshooting certain types of issues.
How I studied:
Having a few years a packet analysis behind me certainly helped, however the Wireshark WCNA Books from Wireshark University are absolutely fantastic!
The network analysis book, while it is expensive is definitely worth it. To me, this book is to protocol analysis; is what Routing TCP/IP Vol I & II is to a CCIE: R/S candidate. It is a large book with great material, a book you can keep on your book shelf at an arms length for years and still use for reference. This book will also cover all the WCNA Exam Objectives, making it an important resource if you are studying for this exam. The other great piece I loved about this book was all the real world case studies, it’s one thing for a book to teach you topic but it’s completely differently for a book to show you how this knowledge is applied in the real world. At the end of each the chapter the book points you toward PCAPs to test your newly learned knowledge which are available for free off the Wireshark book website.
Once you finish with the Network Analysis book, this is where the Prep Guide comes in. I bought the prep guide the weekend before my exam and went through all the questions, using that book to judge where I stand with the objectives. I did pretty well with the Prep Guide, missing maybe 10% of all the 300 questions so I figured it was time to schedule the exam and took it later that week.
I also read through the Wireshark 101 book, which in my opinion is a good book for anyone just starting out with Wireshark or if you want to start customizing wireshark. Which I highly recommend, however if you already familiar with Wireshark I’d skip over this one.
There are also a few great YouTube channels out there, with some great Wireshark videos and even some Sharkfest videos.
Just because I finished the WCNA, does not mean I will stop posting my Wireshark Tid-Bits I’ve still got plenty more of those in store.
Many of us are familiar with the GUI version of Wireshark, but believe it or not but there are also a slew of other command line based utilities that enhance Wireshark and also aid us in capturing and analyzing data. Let’s take a quick look at some of these tools.
- tshark – This is pretty much the CLI equivalent of Wireshark. Allowing you to capture packets like you are using tcpdump, specifing interfaces, filters, etc. It’s definitely worth taking the time to get familiar with tshark.
- dumpcap – This is another CLI equivalent of Wireshark, however this utility writes directly to a file and is less feature-rich then its ‘tshark‘ equivalent. Think of this as the cheap and dirty Wireshark, hop into a system and initiate a dumpcap then boom you have your capture.
- mergcap – As the name implies, this tool allows you to merge multiple captures files into a single capture. Since, Wireshark does have a limitation on processing large file sizes you also have the ability to truncate packets after so many bytes. Similar to what we will do with editcap shortly.
- editcap – This is very nifty, allowing you to do many different things:
- Pick out specific time frames of a packet capture.
- Remove duplicate packets. In case you accidentally captured at multiple locations or fubar-ed your SPAN or TAP locations.
- Truncate packets after so many bytes. This is very handy incase you only want to look at packet headers.
In the below example I am taking an existing PCAPNG file and limit every packet to 40 bytes into a new file filter.pcapng. So you can decrease the file size making it easier for Wireshark process while still keeping the header information. 40-bytes is a bit much but hey it gets the point across.
- capinfos – Provides detailed information about the packet capture in question.
- Average Packet Size
- Time stamp information
- Data rate or packet rate
In the GUI you can get most of this information from the ‘summary‘ -> ‘statistics‘ page which I covered in a previous post, but the CLI version can provide quick and easy access to this information without the need to even launch Wireshark.
Sample output from capinfo is below:
Well, my first draft got lost in the cloud so let’s try this again!
The more you use Wireshark and the more familiar you get with protocols / packet analysis the quicker you realize what you may need to for. Luckily for us, if we know what we are looking for we can configure Wireshark to turn that needle in a haystack into a firework on the middle of Halloween. It does this by giving us the flexibility to control what information Wireshark displays to us and how Wireshark displays that information. The two most useful features we have are profiles and coloring rules, both of these are very powerful features and using both of these features together allows you to take your analyzing skills to the next level.
Profiles – Profiles give us the ability to control what information Wireshark displays to us, and how the information is displayed.
- Affecting the complete layout of the Wireshark display
- What columns are displayed in the Wireshark display
- Which coloring rules are in affect
Now, that we know some of the ways profiles affect Wireshark lets consider a few good use cases for profiles, below are a few profiles I have.
- Wired-VoIP – This profile will call out the DSCP field as specific column to easily keep an eye on QoS marking.
- Remote-Site-VPN – Calls out specific columns for the DF-Bit, IP & TCP length, and more fragment field.
- Wired-Transaction-Time – Contains specific columns for relative time & absolute time, etc.
Those are just a few ways profiles can be leveraged, and remember it is easy enough to flip from one profile to the next. There is no need to even close the current capture or restart Wireshark. This allows to quickly scroll through a single capture looking for key characteristics.
Coloring Rules – These coloring rules define how Wireshark displays the individual packets, it’s these same coloring rules that make re-transmissions show up in red. It’s important to remember that coloring rules are match from top-to-bottom and they will by specific criteria found in the packet. These coloring rules are tied to specific profiles, so you definitely want to keep in mind what profile you are working under.Below are a few of my coloring rules:
- WLAN-RETYU-PACKET – This filter looks to see if the packet is a retransmission of the RF medium. Might be useful if you are troubleshooting a WLAN performance issue.
- FRAG-PACKET – This rule calls out any fragmented packets by keeping an eye out on the ‘More Fragments’ bit. Could be a useful statistics if you working on performance issues in remote IPSec VPN locations.
- Kerberos_MSG – This filter actually picks any kerberos related packets, cause sometimes when Windows says the login failed due a network timeout it might really be due to a kerberos authentication issue. (FYI: Kerberos type 30 messages are errors. So you can be a bit more specific with this filter if desired)
- PC-1500-MTU – This Filter actually matches on two packet fields. First we make sure the packet is a ‘SYN’ packet, and then look to see if the TCP Max Segment Size is at 1460 which ideal for Ethernet networks. Sequentially, there is also a coloring rule for when the PC advertises a MSS that is not 1460. (PCI-NOT-1500-MTU)
Those are just a few examples to show how powerful the coloring rules can be, we can match on any field within the packet regardless of whether it is the Layer 2 MAC address or a piece of data with the application payload. Not mention we can match by multiple fields at the same time, talk about potential! The only thing I want to re-iterate is the matching is top to bottom, so in this example when Wireshark finds a Kerberos message it will hit the first coloring rule and no other even it is a retransmission. That is just something to keep in mind.
You can verify why a coloring rule is in affect by looking at the ‘Frame’ portion of the packet:
From the above, you can see which coloring rule we hit and why we matched this specific coloring. Very useful in the event we ever need to troubleshoot our own coloring rules.
So, now that we spent all this time creating profiles and coloring rules how do we back them up or transfer them to another laptop/desktop? Well, all these configurations can be transferred and backed up by copying only a few folders. If you are running Windows 7, you’ll find this under AppData\Roaming\Wireshark for your specific windows account.
It’s the Profiles folder we really want, once we take a look in there we see our specific profiles. Although you will probably be better off just copying the entire ‘Wireshark’ directory.
I don’t know about you but when I find myself performing packet captures and analyzing PCAPs I usually only know the symptoms of the issue I am attempting to troubleshoot. IE: Connection timeouts, slow response, long transfer times, etc. I usually don’t know much more than that, only in rare occasions do I get a heads up and insight into the behaviors of the application I am trying to troubleshoot. For all the other situations I need to rely on the PCAPs and interpret what and how the applications are communicating. Whether or not the application is behaving properly and performance is as it should be or if there is indeed something amiss somewhere.
Now for me the easiest way to do this is by using the reviewing the ‘Summary’ page under the ‘Statistics’ menu. A sample summary page is below:
A few great call-outs from this screen:
- Packet Size Limit -Knowing whether or not the packets within the capture were sliced after the first so many bytes is important to know, as sometimes you might not see the entire TCP header or wireshark will start classifying the packets as malformed. Although you will also see a ‘Truncated’ message within the packet indicating the packet was sliced.
- First Packet, Last Packet, & Elapsed time -Matching up the time of a packet capture with when the particular issue occurred is crucial, after all you don’t want to find yourself analyzing the wrong capture. The Elapsed time is important to make note of as this give you the ability to establish a baseline, knowing how long a process takes can you help you identify an issue or identify expected behavior in the future.
- Avg. Packet Size – Depending on what you are trying to troubleshoot the average packet size can be a quick indicator in regards to whether or not your fully using the MTU an your network. If you are troubleshooting data transfers normally you would expect the Avg. Packet to be quite large. If you see exceptionally small packet sizes data transfers may take a lengthy amount of time due to the increase TCP overload and normal L3 forwarding. Same goes for the Avg Mbit/sec, if you have large packets flowing you can expect to see a higher throughput rate, and the opposite for lower packet size rate.
The next spot that is worth checking out is the ‘Conversations’ which is also found under ‘Statistics’ this quaint little window gives you a brief overview of any Source/Destination devices identified within the capture. From an L2 Ethernet perspective up to a L4 TCP/UDP Perspective allowing you see what end points are really involved with this communication along with how much data was sent, the length of time the connection, etc. It’s not completely unheard of for applications to communicate with other devices (Web Servers, DB Servers, File Servers, Other App Servers) to perform whatever tasks it is trying to perform and it could be very possible this third server may or may not be slowing down the process.
So by using these two windows in Wireshark you’ve identified the following:
- The length of time the process take. – Found in the elapsed time of the capture, as long as the entire process was captured that is.
- The endpoints involved with this communication. – Remember it is important to cut down as much background noise as possible.
- How much data is transferred and at what size & rate. – This can very helpful when working data transfers.
There are many different fields in the various headers we get to examine during packet analysis, one of the most overlooked field is the IP Identification field. This simple 16-bit field is displayed in Hex and has a few different uses, most importantly:
- Identifies fragmented packets.
- Identifies the individual packets that the sender transmits.
How does this help us?
- Well, by reviewing the IP Identification numbers you can easily identify which packet was dropped in the conversation, by comparing the packet captures from two different capture points.
- This field can also give us a glimpse at how busy the end-devices are. The IP Identification field will increase by ‘1’ for every packet from the sender. Remember the IP ID Value is specific to each individual and not to a specific conversation. If you are following a specific conversation we may see consecutive IP ID #’s or we could see large jumps in the IP ID # intervals. Depending on the numbers this could tell us if the end-devices could be overloaded, or under-utilized and depending on the situation that could point us to a smoking gun.
- If the packets get fragmented they will have the same IP ID number, the Fragment Offset field will also be set as well. This is helpful in following a conversation over particular link changes.
- Seeing the same IP ID #’s in the same packet capture could also identify switching or routing loops within our network. The IP ID #’s will always increase, seeing the duplicate numbers means were are seeing the same packet more than once. The first thing you want to do is verify your capture point is functioning properly and make sure your capture point is in the right spot. Once you verified that it’s time to go hunting for the loop.
By reviewing the IP ID numbers of the packets what can we tell about this conversation with Wireshark.org?
- All the IP ID #’s are unique, no routing/switching loops
- The IP ID #’s are pretty consecutive on both sides of the conversation. Showing both endpoints are not being highly utilized at this point in time. In fact there are one or two gaps on the 192.168.1.4 side of the conversation showing that endpoint is a little busier than 18.104.22.168
Ever so often when I was doing some packet analysis I would come across systems that were sending packets larger the Ethernet MTU of the segment. Or so I thought those packets were getting transmitted, eventually I finally figured out why I was seeing packets with an increased packet size.
The answer was large segment/send offload (LSO) – When this feature is enabled it is the responsibility of NIC Hardware to chop up the data ensuring why it conforms to the MTU of media/network segment.
Now that we know why we are seeing these large packets, the next part of the question is how are we seeing these large packets in Wireshark. Well, Wireshark relies on WinPCAP or LibPCAP depending on your platform, these two tools capture the packets just before the packets hit the NIC Card and get transferred to the actual network.
The above image is from Winpcap.org, showing the kernel level NPF just above NIC Drive, thus explaining how Wireshark is able to see the larger traffic. Before it hits the NIC Driver and gets segmented due to its LSO capabilities.