Category - Computer Networks | Network Layer Security

To authenticate and/ or encrypt packets at the IP level, IPSec defines two protocols the Authentication Header (AH) Protocol and the Encapsulating Security Payload (ESP) Protocol.

Authentication Header (AH) Protocol

The Authentication Header (AH) protocol is intended to verify the legitimacy of the originating host and protect the data integrity of the IP packet's content. A message digest is generated by the protocol using a hash function and a symmetric (secret) key, and it is added to the authentication header (see MAC). In accordance with the mode (transport or tunnel), the AH is subsequently placed in the proper location. The authentication header's fields and location in transport mode are shown in the image below.

When an IP datagram contains an authentication header, the protocol field of the IP header's original value is changed to the value 51. The initial value of the protocol field (the kind of payload that the IP datagram is carrying) is stored in the following header field, which is inside the authentication header. The steps that come after adding an authentication header are as follows:

1. The authentication data field in the authentication header, which is added to the payload, is set to 0.
2. To make the total length suitable for a specific hashing technique, padding may be applied.
3. The entire packet is used for hashing. However, only those IP header values that remain constant throughout transmission are taken into account when calculating the message digest (authentication data).
4. The authentication header contains the authentication data.
5. The protocol field's value is changed to 51 before the IP header is appended.

Following is a quick summary of each field:

• Next Header - The type of payload that an IP datagram carries is specified by its 8-bit next header field (such as TCP, UDP, ICMP, or OSPF).
• Payload Length - The 8-bit field has an inaccurate name. It does not specify the size of the payload; however, it does provide the authentication header's size in 4-byte increments, excluding the first 8 bytes.
• Security Parameter Indexes - A link known as a Security Association uses a 32-bit field called the security parameter index (SPI) that serves as a virtual circuit identity for all packets delivered.
• Sequence Number - An ordered sequence of datagrams is provided by a 32-bit sequence number. A playback is prevented by the sequence numbers. In spite of packet retransmission, it should be noted that the sequence number is not repeated. After reaching 2³², a sequence number does not continue; instead, a new connection is required.
• Authentication Data - Last but not least, the IP datagram as a whole?aside from the fields that are modified during transit?is hashed to provide the authentication data field (e.g., time-to-live).

1. Data integrity and source authentication are provided by the AH protocol, but not privacy.  

Encapsulation Security Payload (ESP) Protocol

The AH protocol provides only source authentication and data integrity; secrecy is not one of them. Encapsulating Security Payload (ESP), a substitute protocol later described by IPSec, offers source authentication, integrity, and confidentiality. The ESP appends a header and trailer. The authentication data for ESP are inserted at the end of the packet, which simplifies the calculation. The placement of the ESP header and trailer is illustrated in the following figure.

The protocol field in the IP header has a value of 50 when an ESP header and trailer are present. The initial value of the protocol field (the TCP or UDP payload format that the IP datagram is using as its payload type) is stored in a field inside the ESP trailer called the next-header field. Following are the steps of the ESP procedure:

1. The payload is increased by an ESP trailer.
2. Encryption is used for both the payload and the trailer.
3. A header for ESP is added.
4. The authentication data is created using the ESP header, payload, and trailer.
5. The authentication information is included in the ESP trailer's conclusion.
6. The protocol value is changed to 50, and the IP header is added.

There are the following fields for the header and trailer:

• Security Parameter Index - The index field for the 32-bit security parameter is comparable to the one specified for the AH protocol.
• Sequence Number - The 32-bit sequence number field is comparable to the one specified for the AH protocol.
• Padding - This padding field has a configurable length (0 to 255 bytes) and contains only zeros.
• Pad Length- The number of padding bytes is specified by an 8-bit parameter called pad length. The range of the value is 0 to 255, with 255 being the rare figure.
• Next Header - The 8-bit next-header field is comparable to that specified in the AH protocol. It fulfils the same function as the IP header's pre-encapsulation protocol field.
• Authentication Data - After applying an authentication technique to specific portions of the datagram, the authentication data field is the end result. Take note of how the authentication information in AH and ESP differs. In AH, a portion of the IP header is taken into account while calculating the authentication data; this is not the case in ESP.

1. IPv4 v IPv6 - IPv4 and IPv6 are both supported by IPSec. The AH and ESP are a part of the extension header in IPv6, but not in IPv4.  

AH versus ESP

The ESP protocol was created after the AH protocol had already been implemented. With added capabilities, ESP performs the same tasks as AH (confidentiality). In reality, we do not require AH. AH will, however, continue to exist on the Internet until these items are phased out because the implementation of AH is currently present in some commercial products.

Services Provided by IPSec

For packets at the network layer, the two protocols AH and ESP can offer a number of security features. Lists of services that are offered for each protocol are shown in the table below:

1. Access Control - IPSec uses a Security Association Database (SAD) to offer access control in an indirect manner. A packet is rejected when it reaches its destination, and no Security Association has been created for it yet.
2. Data Integrity -AH and ESP both maintain the integrity of the message. The sender creates and sends a data digest for the recipient to review.
3. Entity Authentication - The sender of the data is authenticated in both AH and ESP by the Security Association, and the sender transmits the keyed-hash digest of the data.
4. Confidentiality - The message's confidentiality is provided via ESP's encryption. Though, confidentiality is not offered by AH. If confidentiality is required, ESP ought to be used rather than AH.
5. Replay Attack Protection - Both protocols employ a sliding receiver window and sequence numbers to protect against the replay attack. When the Security Association is established, a distinct sequence number is present in each IPSec header. Starting at zero, the number grows until it reaches 2³² - 1. The previous Security Association is removed, and a new one is created at the same time as the sequence number is reset to zero when it reaches its maximum value. IPSec requires the usage of a fixed-size window at the receiver to avoid processing duplicate packets. The receiver chooses the window's size, which has a default value of 64.