Windows Server 2003 : Securing Network Communication - Planning an IPSec Implementation

- Free product key for windows 10
- Free Product Key for Microsoft office 365
- Malwarebytes Premium 3.7.1 Serial Keys (LifeTime) 2019
You can store your files in encrypted form using the Encrypting File System (EFS), for example, or an individual application might be able to protect files with a password, but when you access the file over the network or send it to someone else, your computer always decrypts it first. The IP Security (IPSec) extensions are a means of securing the actual network communications themselves so that intruders cannot compromise your data by intercepting it as it travels over the network.

Evaluating Threats

When you log on to a file transfer protocol (FTP) server on your network, you have to supply a user name and a password to be granted access. The FTP client program you use probably does not display the password on the screen as you type it, but of course the password must be included in the data packets the client sends over the network to the FTP server.

Figure 1 shows a screen capture from Microsoft Network Monitor, which is displaying the contents of an FTP packet that the program captured from the network. In this packet, you can clearly see the password (which is “password”) associated with the user account that the client is supplying to the server. If you are a network administrator and you use the Administrator account to access the FTP server, someone capturing the packets in this way could learn the Administrator password and possibly wreak havoc on the network.

Figure 1. Network Monitor, displaying an FTP password

This is just an example of how easy it is for unauthorized people to capture and access your data as it is being transmitted. A user running a protocol analyzer such as Network Monitor can capture the packets containing your data files, your e-mail messages, or other confidential communications, and reconstruct the data for his or her own use.

Unauthorized personnel can use this captured data against you in many ways, including the following:

  • Compromising keys In the same way that captured packets can contain passwords, they can also contain encryption keys. An intruder capturing a key can then decrypt any data using that key. The public key infrastructure (PKI) used on networks running Microsoft Windows is not threatened by this practice because it uses separate public and private keys for encryption and decryption, and the private keys are never transmitted over the network. However, other encryption systems use a single key to encrypt and decrypt data, and if an intruder captures that key, the entire security system is compromised.

  • Spoofing Spoofing is digitally masquerading as another person or system by using captured Internet Protocol (IP) addresses and other information. By capturing network packets, an intruder can discover valid IP addresses, packet sequence numbers, and the other personal information needed to create new packets that appear to have originated from the actual user’s computer. Using this method, the intruder can send messages in the victim’s name, receive data that was meant for the victim, and even engage in financial or other transactions using the victim’s accounts. Sometimes an attacker will simultaneously initiate a denial-of-service (DoS) attack on the victim’s computer to prevent the victim from sending any further messages while the attacker assumes the victim’s identity.

    Security Alert

    Even when you use applications that encrypt your passwords for transmission, intruders can still sometimes use those passwords by simply pasting the encrypted string into a spoofed message. Even though the intruder doesn’t actually know what the password is, the authenticating system could decrypt it and accept it as genuine.

  • Modifying data When intruders capture data packets from the network, they can not only read the information inside, they can also modify it, and then send the packets to the recipient. The packets arriving at the destination, therefore, might contain information the true sender did not create, even though the packets appear to be genuine.

  • Attacking applications In addition to modifying the data in captured packets, intruders might add their own software to the packets and use the packets to introduce the software into the destination computer. Viruses, worms, and Trojan horses are just some of the dangerous types of code that can infiltrate your network in this way.

Introducing Network Security Protocols

Network security protocols are used to manage and secure authentication, authorization, confidentiality, integrity, and nonrepudiation. In a Microsoft Windows Server 2003 network, the major protocols used are Kerberos, NTLM, IPSec, and their various subprotocols. Other network communication protocols support these protocols, and other security settings support and protect the use of these security protocols. Table 1 lists the security paradigms and the protocols that support them.

Table 1. Network Security Protocols
AuthenticationTo prove you are who you say you areKerberos and NTLM (The NTLM authentication protocol is not available by default, but it can be configured.)
AuthorizationTo determine what you can do on the network after you have authenticatedKerberos and NTLM
ConfidentialityTo keep data secretEncryption components of Kerberos, NTLM, and IPSec (to secure communications other than authentication)
IntegrityTo ensure that the data received is the same data that is sentComponents of Kerberos, NTLM, and IPSec
NonrepudiationTo determine exactly who sent and received the messageKerberos and IPSec

This lesson and the next will focus specifically on IPSec. The functionality and technical details of Kerberos and NTLM are beyond the scope of this training kit.

Protecting Data with IPSec

IPSec is designed to protect data by digitally signing and encrypting it before transmission. IPSec encrypts the information in IP datagrams by encapsulating it so that even if the packets are captured, none of the data inside can be read. Using IPSec protects your network against all the threats listed in the previous section.

Because IPSec operates at the network layer as an extension to the IP protocol, it provides end-to-end encryption—meaning that the source computer encrypts the data, and it is not decrypted until it reaches its final destination. Intermediate systems, such as routers, treat the encrypted part of the packets purely as payload, so they do not have to perform any decryption; they just forward the encrypted payload as is. The routers do not have to possess the keys needed to decrypt the packets, nor do they have to support the IPSec extensions in any way.

Off the Record

By contrast, encrypting network traffic at the data-link layer would require each router that forwards packets to decrypt the incoming data and then re-encrypt it again before transmitting it. This would add a tremendous amount of processing overhead to each router and slow down the entire network.

Other protocols besides IPSec, such as Secure Sockets Layer (SSL), provide network traffic encryption but they are application-layer protocols that can encrypt only specific types of traffic. For example, SSL encrypts only communications between Web clients and servers. IPSec can encrypt any traffic that takes the form of IP datagrams, no matter what kind of information is inside them.

IPSec Functions

In addition to encrypting IP datagrams, the IPSec implementation in Windows Server 2003 provides a variety of security functions, including the following:

  • Key generation For two computers to communicate over the network using encrypted IP datagrams, both must have access to a shared encryption key. This key enables each computer to encrypt its data and the other computer to decrypt it. However, the key cannot be transmitted over the network without compromising the security of the system. Therefore, computers preparing to communicate with each other using IPSec both use a technique called the Diffie–Hellman algorithm to compute identical encryption keys. The computers publicly exchange information about the calculations that enable them to arrive at the same result, but they do not exchange the keys themselves or information that would enable a third party to calculate the key.

  • Cryptographic checksums In addition to encrypting the data transmitted over the network, IPSec uses its cryptographic keys to calculate a checksum for the data in each packet, called a hash message authentication code (HMAC), and then transmits it with the data. If anyone modifies the packet while it is in transit, the HMAC calculated by the receiving computer will be different from the one in the packet. This prevents attackers from modifying the information in a packet or adding information to it (such as a virus).

    IPSec supports two hash functions: HMAC in combination with Message Digest 5 (MD5) and HMAC in combination with Secure Hash Algorithm-1 (SHA1.) HMAC-SHA1 is the more secure function, partly because of SHA1’s longer key length. (SHA1 uses a 160-bit key as opposed to the 128-bit key used by MD5.) HMACMD5 is strong enough for a normal security environment, but HMAC-SHA1 is the better choice for a high-level security environment and it meets the United States government’s security requirements for high-level security.

  • Mutual authentication Before two computers can communicate using IPSec, they must authenticate each other to establish a trust relationship. Windows Server 2003 IPSec can use Kerberos, digital certificates, or a preshared key for authentication. Once the computers have authenticated each other, the cryptographic checksum in each packet functions as a digital signature, preventing anyone from spoofing or impersonating one of the computers.

  • Replay prevention In some cases, attackers might be able to use data from captured packets against you, even when the data in the packets is encrypted. Using traffic analysis, it is possible to determine the function of some encrypted packets. For example, the first few packets that two computers exchange during a secured transaction are likely to be authentication messages. Sometimes, by retransmitting these same packets, still in their encrypted form, attackers can use them to gain access to secured resources. IPSec prevents packet replays from being effective by assigning a sequence number to each packet. An IPSec system will not accept a packet that has an incorrect sequence number. This is also referred to as antireplay services.

  • IP packet filtering IPSec includes its own independent packet filtering mechanism that enables you to prevent denial-of-service attacks by blocking specific types of traffic through the use of IP addresses, protocols, ports, or any combination of the three.

IPSec Protocols

The IPSec standards define two protocols that provide different types of security for network communications: IP Authentication Header (AH) and IP Encapsulating Security Payload (ESP). These protocols are discussed in the following sections.

IP Authentication Header

The IP Authentication Header protocol does not encrypt the data in IP packets, but it does provide authentication, antireplay, and integrity services. You can use AH by itself or in combination with ESP. Using AH alone provides basic security services, with relatively low overhead. AH by itself does not prevent unauthorized users from reading the contents of captured data packets. However, using AH does guarantee that no one has modified the packets en route, and that the packets did actually originate at the system identified by the packet’s source IP address.

See Also

IPSec is capable of operating in two modes: transport mode and tunnel mode. These descriptions of the AH and ESP protocols refer to transport mode operations. For more information on tunnel mode, see “Transport Mode and Tunnel Mode” later in this lesson.

When a computer uses AH to protect its transmissions, the system inserts an AH header into the IP datagram, immediately after the IP header and before the datagram’s payload, as shown in Figure 2.

Figure 2. The AH header location

The contents of the AH header are shown in Figure 3.

Figure 3. The AH header format

The functions of the header fields are as follows:

  • Next Header Contains a code specifying the protocol that generated the header immediately following the AH header, using the protocol codes specified by the Internet Assigned Numbers Authority (IANA). If IPSec is using AH alone, this field contains the code for the protocol that generated the datagram’s payload, which is usually Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or Internet Control Message Protocol (ICMP).

    See Also

    The IP header has a Protocol field that contains a code identifying the protocol that generated the datagram’s payload. Normally, this code has a value of 6 for TCP, 17 for UDP, or 1 for ICMP. However, in a packet using AH, the Protocol field has a value of 51 because the AH header immediately follows the IP header. The Next Header field in the AH header contains the code representing the TCP, UDP, or ICMP that generated the payload.

  • Payload Length Specifies the length of the AH header.

  • Reserved Unused.

  • Security Parameters Index Contains a value that, in combination with the packet’s destination IP address and its security protocol (AH), defines the datagram’s security association. A security association is a list of the security measures, negotiated by the communicating computers, that the systems will use to protect the transmitted data.

  • Sequence Number Contains a value that starts at 1 in the first packet using a particular security association, and is incremented by 1 in every subsequent packet using the same security association. This field provides the IPSec antireplay service. If an IPSec system receives packets with the same sequence numbers and the same security association, it discards the duplicates.

  • Authentication Data Contains an integrity check value (ICV) that the sending computer calculates, based on selected IP header fields, the AH header, and the datagram’s IP payload. The receiving system performs the same calculation and compares its results to this value.


The ICV is the message authentication code. Its main purpose is to authenticate a message and verify its integrity.

IP Encapsulating Security Payload

The IP Encapsulating Security Payload (ESP) protocol is the one that actually encrypts the data in an IP datagram, preventing intruders from reading the information in packets they capture from the network. ESP also provides authentication, integrity, and antireplay services. Unlike AH, which inserts only a header into the IP datagram, ESP inserts a header and a trailer, which surround the datagram’s payload, as shown in Figure 4. The protocol encrypts all the data following the ESP header, up to and including the ESP trailer. Therefore, someone who captures a packet encrypted using ESP, could read the contents of the IP header, but could not read any part of the datagram’s payload, including the TCP, UDP, or ICMP header.

Figure 4. The ESP header and trailer locations

An IPSec packet can use ESP by itself or in combination with AH. When a packet uses both protocols, the ESP header follows the AH header, as shown in Figure 5. Although AH and ESP perform some of the same functions, using both protocols provides the maximum possible security for a data transmission. When ESP computes its ICV, it calculates the value only on the information between the ESP header and trailer; no IP header fields are included in an ESP ICV. Therefore, it is possible for an attacker to modify the contents of the IP header in an ESP-only packet and have those changes go undetected by the recipient. AH includes most of the IP header in its ICV calculation, so combining AH with ESP provides more protection than using ESP alone.

Figure 5. An IP datagram using AH and ESP

The contents of the ESP header are shown in Figure 6.

Figure 6. The ESP message format

The functions of the header fields are as follows:

  • Security Parameters Index Contains a value that, in combination with the packet’s destination IP address and its security protocol (AH or ESP), defines the datagram’s security association.

  • Sequence Number Contains a value that starts at 1 in the first packet using a particular security association, and is incremented by 1 in every subsequent packet using the same security association. This field provides IPSec’s antireplay service. If an IPSec system receives packets with the same sequence numbers and the same security association, it discards the duplicates.

  • Payload Data Contains the TCP, UDP, or ICMP information carried inside the original IP datagram.

  • Padding Consists of padding added to the Payload Data field to ensure the Payload Data has a boundary required by the encryption algorithm.

  • Pad Length Specifies the number of bytes of padding the system added to the Payload Data field to fill out a 32-bit word.

  • Next Header Contains a code specifying the protocol that generated the header immediately following the ESP header, using the protocol codes specified by the IANA. In virtually all cases, this field contains the code for the protocol that generated the datagram’s payload, which is usually TCP, UDP, or ICMP.

    See Also

    When an IPSec system is using AH and ESP together, the Protocol field in the IP header contains the value 51 because the AH header immediately follows the IP header. The Next Header field in the AH header has the value 50 because the ESP header immediately follows the AH header. Finally, the Next Header field in the ESP header contains the code for the protocol that generated the payload, which is usually TCP, UDP, or ICMP.

  • Authentication Data Contains an ICV based on the information after the ESP header, up to and including the ESP trailer. The receiving system uses the ICV to verify the packet’s integrity by performing the same calculation and comparing the results with this value.

Transport Mode and Tunnel Mode

IPSec can operate in two modes: transport mode and tunnel mode. To protect communications between computers on a network, you use transport mode, in which the two end systems must support IPSec but intermediate systems (such as routers) need not. All the discussion of the AH and ESP protocols so far in this lesson applies to transport mode.

Tunnel mode is designed to provide security for wide area network (WAN) connections, and particularly virtual private network (VPN) connections, which use the Internet as a communications medium. In a tunnel mode connection, the end systems do not support and implement the IPSec protocols; the routers at both ends of the WAN connection do this.

The tunnel mode communications process proceeds as follows:

  1. Computers on one of the private networks transmit their data using standard, unprotected IP datagrams.

  2. The packets reach the router that provides access to the WAN, which encapsulates them using IPSec, encrypting and hashing data as needed.

  3. The router transmits the protected packets to a second router at the other end of the WAN connection.

  4. The second router verifies the packets by calculating and comparing ICVs, and decrypts them if necessary.

  5. The second router repackages the information in the packets into standard, unprotected IP datagrams and transmits them to their destinations on the private network.

IPSec also uses a different packet structure in tunnel mode. Unlike transport mode, in which IPSec modifies the existing IP datagram by adding its own headers, tunnel mode implementations create an entirely new datagram and use it to encapsulate the existing datagram, as shown in Figure 7. The original datagram, inside the new datagram, remains unchanged. The IPSec headers are part of the outer datagram, which exists only to get the inner datagram from one router to the other.

Figure 7. An IPSec tunnel mode packet

Top 10
Free Mobile And Desktop Apps For Accessing Restricted Websites
TOYOTA CAMRY 2; 2.5 : Camry now more comely
KIA SORENTO 2.2CRDi : Fuel-sipping slugger
How To Setup, Password Protect & Encrypt Wireless Internet Connection
Emulate And Run iPad Apps On Windows, Mac OS X & Linux With iPadian
Backup & Restore Game Progress From Any Game With SaveGameProgress
Generate A Facebook Timeline Cover Using A Free App
New App for Women ‘Remix’ Offers Fashion Advice & Style Tips
SG50 Ferrari F12berlinetta : Prancing Horse for Lion City's 50th
- Messages forwarded by Outlook rule go nowhere
- Create and Deploy Windows 7 Image
- How do I check to see if my exchange 2003 is an open relay? (not using a open relay tester tool online, but on the console)
- Creating and using an unencrypted cookie in ASP.NET
- Directories
- Poor Performance on Sharepoint 2010 Server
- SBS 2008 ~ The e-mail alias already exists...
- Public to Private IP - DNS Changes
- Send Email from Winform application
- How to create a .mdb file from ms sql server database.......
programming4us programming4us