Parts of this page were adapted from http://www.cromwell-intl.com/tcpip/what-is-ipsec.html

IPsec Simplified

What is IPsec? What network security does IPsec provide?

IPsec is an extension of the IP to secure network communication through cryptography. Its primary goals are data confidentiality, data integrity, and host authentication (CIA). The combination of integrity and authentication also provides non-repudiation and detects replay attacks. Basically, it is a modification of the IP implementation within the TCP/IP protocol suite, and so it means a modification to the kernel of modern operating systems.

The following attempts to be a "just enough" explanation. If you want all the details, see the official documents:

• RFC 4301, "Security Architecture for IP"
• RFC 4302, "IP Authentication Header"
• RFC 4303, "IP Encapsulating Security Payload (ESP)"
• RFC 4306, "Internet Key Exchange (IKEv2) Protocol"

How do you pronounce its name? I've heard all of these:

• ipsec as a two-syllable word that would rhyme with tripsec
• eye pee seck
• eye pee security

Architecture

IPsec provides network communication security between hosts. User issues like authenticating a human as some user, restricting access to data by users, and so on, are outside the scope of IPsec. Those important security tasks are left to the operating systems run on the hosts.

Two hosts interested in securely communicating with each other must establish a Security Association (SA) with each other. The details can be tremendously complicated, but the SA comes down to the following. See my http://www.cromwell-intl.com/security/crypto.html for explanations of ciphers, hashes, and HMACs:

• Which of the CIA (Confidentiality, Integrity, Authentication) goals are important? Computers are fast and the usual answer is "all of them", but subsets are possible.
• Confidentiality requires a cipher and a key. If it is desired, which cipher is used (e.g., TripleDES in Cipher Block Chaining mode, AES in Cipher Block Chaining mode) and what is the key?
• Data integrity and host authentication requires a hashed message authentication code (HMAC) based on some cryptographic hash function. If it is desired, which HMAC is used (e.g., HMAC-SHA1-96) and what is the key?

The SA is negotiated by the Internet Key Exchange (IKE) protocol. There is a quick burst of IKE traffic before two hosts start communicating with IPsec. IKE simplifies to these major steps:

1. The hosts authenticate to each other using cryptography. This could be done with shared secrets:
   • Host 1 sends a challenge known to be unique. The number of seconds elapsed since 00:00:00 1 January 1970 makes for a useful challenge.
   • Host 2 encrypts the challenge using the shared secret key and returns this response.
   • Host 1 decrypts the response using the shared secret key. If the resulting cleartext is the challenge, host 2 knows the secret and must be the real host 2.
2. The problem with that approach is that it does not scale. You can set up shared secrets between a few machines. But since every machine-to-machine pair needs a key, and the number of keys grows as the square of the number of machines, that does not scale to a large organization. Set up digital certificates instead:
   • Generate a public/private key pair and digital certificate for each host. The digital certificate is an X.509 data structure including the public key wrapped inside a digital signature made by the trusted signing authority, or Certification Authority (CA).
   • Host 1 sends a challenge known to be unique. The number of seconds elapsed since 00:00:00 1 January 1970 makes for a useful challenge.
   • Host 2 encrypts the challenge using its private key and returns this response along with the digital certificate containing the public key for Host 2.
   • Host 1 verifies the digital signature on the certificate using its trusted copy of the signing authority's public key. Host 1 then extracts the public key for Host 2 from the certificate and uses that to decrypt the response. If the resulting cleartext is the challenge, the responding host knew the private key for Host 2 and must be the real Host 2.
3. After a round of that in each direction, each host is confident in the other host's identity.
4. The two hosts negotiate a mutually acceptable combination of confidentiality (or not) and integrity plus authentication (or not). This is usually the complete CIA triple, but some host pairs may be configured to only use some.

5. The two hosts negotiate a mutually acceptable set of algorithms and keys. The algorithms will generally be the strongest cipher and hash supported by both ends. The keys will be agreed upon using DiffieHellmanKeyExchange.

6. The SA information is then passed to the IPsec module of the kernel.

Once the SA has been established, IKE is used from time to time to negotiate new session keys. You want to use a key only for a limited time and amount of data. IKE can quickly re-run DiffieHellmanKeyExchange to agree on new keys. Any delay caused by periodic re-keying will tend to disappear into the background noise of routine network latency jitter.

Once the kernel has an SA for another host, it will enforce its use. Every packet sent to that host will be modified according to the SA. Every packet apparently from that host will be dropped if it cannot be handled according to the SA. That means that if one host of an IPsec pair suddenly quit using IPsec and simply sent IP datagrams, the other host would drop all those apparently bogus datagrams.

Authentication Header is an IPsec extension to IP to provide data integrity, source host authentication, and protection against replay attacks.

Encapsulating Security Payload is an IPsec extension to IP to provide data confidentiality, data integrity, source host authentication, and protection against replay attacks.

Authentication Header (AH)

Authentication Header inserts an extension to the original IP header. The AH header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes the integrity check value (ICV), basically a hashed message authentication code of the payload, the non-changing IP header fields, and appropriate key.

ORIGINAL, what would have been sent:
+------------+-----------------------------+
|            |                             |
| IP header  |  Payload                    |
|            |                             |
+------------+-----------------------------+

WITH AH:
+------------+-----+-----------------------------+
|            |     |                             |
| IP header  | AH  |  Payload                    |
|            |     |                             |
+------------+-----+-----------------------------+
   ^          ^ ^ ^
   |          | | |
   |          | | +--- "Next header" field indicates the
   |          | |     original encapsulated Protocol field.
   |          | |
   |          | +--- ICV = hash(Key, Payload, some IP header fields)
   |          |
   |          +--- SPI = indication of which SA to apply
   |
   +--- "Protocol" field changed to 51 to indicate AH.

Encapsulating Security Payload (ESP)

I have no idea why the term is Encapsulating Security Payload when Encrypting Security Payload would better emphasize the main point....

ESP inserts an extension to the original IP header, and adds an authentication block to the end of the datagram. The ESP header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes a sequence number, a number that should increase with each IPsec datagram sent. This allows a system to recognize and ignore duplicate received IPsec datagrams.

A "Next header" field is added to the end of the encrypted payload, along with padding to make the datagram an integer multiple of 32 bits. Then a final authentication data block contains the ICV, an HMAC of the payload, the non-changing IP header fields, and appropriate key.

ORIGINAL, what would have been sent:
+------------+----------------------------------+
|            |                                  |
| IP header  | Payload                          |
|            |                                  |
+------------+----------------------------------+

WITH ESP:
+------------+-----+----------------------------------+-+------+
|            |     |                                  | |      |
| IP header  | ESP | Encrypted payload                |N| Auth |
|            |     | (ciphertext of original payload) |H| Data |
+------------+-----+----------------------------------+-+------+
   ^           ^ ^                                     ^    ^
   |           | |                                     |    |
   |           | +--- Sequence number allows detection |    |
   |           |      of replay attempts               |    |
   |           |                                       |    |
   |           +--- SPI = indication of which SA       |    |
   |                      to apply                     |    |
   |                                                   |    |
   |                "Next header" field indicates the  |    |
   |                original encapsulated Protocol ----+    |
   |                                                        |
   |                ICV = hash(Key, Sequence, Payload) -----+
   |
   +--- "Protocol" field changed to 50 to indicate ESP.

Transport Mode and Tunnel Mode

Transport Mode encrypts only the payload, the IP header itself is left alone. This is used for host-to-host communications. The data contents are protected, but anyone observing network traffic can see traffic flow patterns.

ORIGINAL, what would have been sent:
+--------------+----------------------------------+
| IP header    | Payload                          |
| host1->host2 |                                  |
+--------------+----------------------------------+

PROTECTED with ESP in Transport Mode:
+--------------+-----+----------------------------------+-+------+
| IP header    | ESP | Encrypted payload                |N| Auth |
| host1->host2 |     | (ciphertext of original payload) |H| Data |
+--------------+-----+----------------------------------+-+------+

Tunnel Mode is used to encrypt the entire original IP datagram, both payload and original header, and then encapsulate that inside a new IP header. You create a Virtual Private Network (VPN) between two remote sites by doing Tunnel Mode IPsec at the gateways.

Below is a simple example. This organization has two sites, "Site 1" and "Site 2", each with its own simple LAN and IPsec gateway.

      (--------------------------------)
      (                                )
      (          The Internet          )
      (                                )
      (---+------------------------+---)
          |                        |
        +-+-+                    +-+-+
        |GW1|                    |GW2|
        +-+-+                    +-+-+
 (--------+---)                (---+--------)
 (            )                (            )
 ( Site 1 LAN )                ( Site 2 LAN )
 (            )                (            )
 (--+---------)                (---------+--)
    |                                    |
  HostA                                HostB

HostA sends a packet to HostB. The hosts see the IPsec gateways GW1 and GW2 as nothing more then routers. The packet traveling across the Site 1 LAN looks like this:

+--------------+----------------------------------+
| IP header    | Payload                          |
| HostA->HostB |                                  |
+--------------+----------------------------------+

If GW1 and GW2 do not not have an SA established, they set one up via IKE when the packet bound for the Site 2 LAN reaches GW1. Gateway GW1 then encrypts that entire datagram and wraps it inside an IPsec header for delivery across the Internet to gateway GW2. If you captured the packet as it moved across the Internet, you would see this:

+-----------+-----+-------------------------------------------------+-+----+
| IP header | ESP | /  /  /  /  / ciphertext version  /  /  /  /  / |N|Auth|
| GW1->GW2  |     |  /  /  /  /  of original datagram  /  /  /  /  /|H|Data|
+-----------+-----+-------------------------------------------------+-+----+

That datagram is received at gateway GW2 and handled as per the SA specified in the SPI field of the ESP header. It is decrypted with the appropriate cipher and key, and the resulting datagram, identical to the original, is dropped onto the Site 2 LAN.

Maybe HostA and HostB shared an SA. If that were the case, the payload of the datagrams cross the two LANs would be ciphertext. There's certainly no harm in encrypting data that happens to already be ciphertext. And this would provide defense in depth and protect against hostile insiders at both sites.

http://www.cromwell-intl.com/tcpip/what-is-ipsec.html

Many people very reasonably say that it makes no sense to arrange a method for whispering secrets to strangers. Data encryption implies data sensitivity, and so you should only send sensitive data to a host of known identity. Well, sure, but....

There is an interesting thing called opportunistic encryption. This is an attempt to establish encrypted communication with any host before falling back to plaintext communication if needed. This means you can routinely encrypt some of your network traffic without setting up any Security Associations, just by asking your operating system to attempt opportunistic encryption and then communicating with other hosts supporting the same thing.

The topic of opportunistic encryption takes many security professionals through these stages:

• Polite attempts to correct my error, surely I don't really mean what I seem to be saying.
• Derision at the foolish people who would even consider such a silly thing as whispering secrets to strangers.
• Utter horror when they are finally convinced that people really do use this for things like routine web browsing, IRC, electronic mail, peer-to-peer file sharing, and more.

Yes, this would allow the opportunistic encryption subset of Internet users to communicate with each other while frustrating attempts to routinely eavesdrop on them. See the http://en.wikipedia.org/wiki/Opportunistic_encryption for pointers to the details for doing this on Linux and routers with Openswan and other FreeS/WAN derivatives, on Windows, to protect e-mail and VoIP, and more.