CCIE or Null!

My journey to CCIE!

IKE main mode, aggressive mode, & phase 2.

with 13 comments

Just like GRE tunnels, IPSec is found in every single network, whether it’s in the form a Lan2Lan tunnel or a client side remote access VPN. We all know IPSec secures communication between two endpoints using ISAKMP, Diffie-Hellman, and various other encryption and hashing algorithms but how exactly do these protocols work together. We know IPSec will form its tunnel after IKE Phase 1 and Phase 2 so let’s take a look at what goes on during this process:

IKE Phase 1

IKE Phase 1 works in one of two modes, main mode or aggressive mode now of course both of these modes operate differently and we will cover both of these modes.

Main Mode:

IKE Phase 1 operating in main mode works with both parties exchanging a total of 6 packets, that’s right 6 packets is all it takes to complete phase 1.

  1. The first packet is sent from the initiator of the IPSec tunnel to its remote endpoint, this packet contains the ISAKMP policy
  2. The second packet is sent from the remote endpoint back to the initiator, this packet will be the exact same information matching the ISAKMP policy sent by the initiator.
  3. The third packet is sent from the initiator to the remote endpoint, this packet contains the Key Exchange payload and the Nonce payload, the purpose of this packet is generate the information for the DH secret key
  4. This fourth packet as you would expect comes from the remote endpoint back to initiator and contains the remote endpoints Key Exchange and Nonce payload.
  5. The fifth packet is from the initiator back to the remote endpoint with identity and hash payloads, the identity payload has the device’s IP Address in, and the hash payload is a combination of keys (including a PSK, if PSK authentication is used)
  6. The sixth packet from the remote endpoint to the initiator contains the corresponding hash payloads to verify the exchange.

First packet in the IKE Phase 1

Second packet of the IKE Phase 1 process

Third packet in IKE Phase 1

Fourth packet in the IKE Phase 1 process

Fifth packet in the IKE Phase 1 process

Sixth and final packet in IKE Phase 1

Aggressive Mode:

IKE Phase 1 operating in aggressive mode only exchanges 3 packets compared to the 6 packets used in main mode. One downside in aggressive is the fact it not as secure as main mode.

  1. The first packet from the initiator contains enough information for the remote endpoint to generate its DH secret, so this one packet is equivalent to the first four packets in main mode.
  2. The second packet from the remote endpoint back to the initiator contains its DH secret
  3. The third packet from the initiator includes identity and hash payloads. After the remote endpoint receives this packet it simply calculates its hash payload and verifies it matches, if it matches then phase one is established.

IKE Phase 2

Now let’s look at IKE Phase 2, IKE Phase 2 occurs after phase 1 and is also known as quick mode and this process is only 3 packets.

  • Perfect Forward Secrecy PFS, if PFS is configured on both endpoints the will generate a new DH key for phase 2/quick mode.
  1. Contained in this first packet from the initiator to the remote device are some of the hashes/keys negotiated from phase 1, along with some IPSec parameters IE: Encapsulation (ESP or AH), HMAC, DH-group, and the mode (tunnel or transport)
  2. The second packet contains the remote endpoint’s response with matching IPSec parameters.
  3. The last packet is sent to the remote device to verify the other device is still there and is an active peer.

That last packet concludes the forming an IPSec tunnel and the phase 1/2 process.

  • Note: These 3 quick mode packets are encrypted

Quick Mode packets with encrypted payload.

All 9 packets of the Phase 1 (Main Mode) and Phase 2 process.

Written by Stephen J. Occhiogrosso

March 26, 2012 at 7:24 AM

13 Responses

Subscribe to comments with RSS.

  1. Well done my friend! Very impressive write up on IPSec.

    Jon M.

    March 26, 2012 at 4:08 PM

  2. Updated with some packet captures.

    I could not get packet capture for an aggressive VPN connection because my 3550’s keep crashing when it tries to form.

    Stephen J. Occhiogrosso

    April 1, 2012 at 9:47 PM

  3. […] VPN, assuming everything is configured properly on both ends, now hopefully after going this and my post from a few weeks ago going over IKE in general you’ll have a deeper understanding of what really goes on when a […]

  4. Could you touch on Base Mode?


    June 21, 2012 at 7:01 AM

  5. first piece of write up that made me actually understand the process in IPSec, thanks a lot.


    March 6, 2013 at 12:40 PM

  6. excellent. thank you


    August 15, 2014 at 2:37 PM

  7. Wow! Just wow!!! An awesome post! This will definitely clear lot of confusion surrounding IPSec! IPSec has been one of the most frustrating topics for me to learn. So many words for same thing in different vendors! Your post really clarifies the actual steps that happen during the IPSec phases!

    Shri Ganesh

    April 11, 2015 at 8:07 AM

  8. very nice article. very informative

    rashid anwar

    September 13, 2015 at 9:43 PM

  9. […] IKE Main Mode, Aggressive Mode, & Phase 2 […]

  10. could you please send me the packet capture of this scenario as i wanted to see the hash value of the address and ports looks for the NAT-D packet…

    rohanmaini (@rohanradals)

    November 19, 2015 at 8:53 PM

  11. Great article but would like add some:
    1) PFS when configured, allow new DH sessions keys to be generated not only for IKE phase 2 but also for IPSEC transforms set specified which is used to encrypt/decrypt interesting traffic.

    Keep it up:)


    August 27, 2016 at 8:12 PM

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: