Use of IPS is becoming more widespread in SMBs and enterprises. However, not all administrators have the expertise required to effectively manage systems. For example, when an IDS generates warnings about possible intruders and attacks, the administrator may not know the real scope of these warnings.
Generally, before compromising a system, an intruder will use one of the many methods available to explore the target system. For example, an intruder could use the technique known as port scanning to find out what types of services are available in a host or subnet. This type of operation is often a clear signal of malicious intentions.
The most common methods used by attackers to explore systems and/or subnets are malformed IP packets.
Malformed IP packets are packets that do not comply with the IP standards defined in the RFC documents (Request For Comments).
The packets could also be generated by routing devices that are not correctly configured. They are usually created to use as an attack tool. By using this type of packet, they are not identified and/or blocked by an IDS or IPS or even a firewall. In some cases, they are used successfully to crash target systems.
Nowadays, TCP packets are the most widely used, as this protocol is designed for connections that use flags to specify the status of the connection. By doing this, they specify if the connection has started or ended. What’s more, they provide information about the priority of the data in the packet. Many attacks take advantage of the modifications to flags when a packet is created.
The functional behavior of TCP is defined in the corresponding RFC documents, but the lack of specifications, such as how systems and environments should respond to malformed packets, for example, a packet with modified flags, leaves intruders a lot margin to play with. Therefore, different systems respond differently to abnormal flag combinations in a packet.
- A standard packet should include at least on of the following flags:
- SYN: Starts a TCP connection.
- ACK: Used to validate and check the packet sequence numbers.
- FIN: Ends the connection in regular mode.
- RST: Ends the connection immediately.
- PSH: Informs the recipients to process the packet as soon as possible.
- URG: Specifies that the packet is urgent.
- A malformed IP packet is a packet that fulfils one of the following conditions:
- Packets that have no flag marked, known as null.
- Packets that use the flags SYN and FIN. The SYN flag is used to start a connection, whereas FIN is used to end a connection. It is absurd to carry out both actions at the same time.
- Packets with SYN FIN PSH, SYN FIN RST, SYN FIN RST PSH flags are other variants of packets that contain both the SYN and FIN flags. These packets can be used by attackers that know that intrusion detection systems can search for flags with only the SYN and FIN flag.
- Packets that only contain the FIN flag. A packet must never contain only the FIN flag, as these types of packets are often used to scan ports and find out network topology, as well as other suspicious activities.
NOTE: There are other multiple combinations of not only flags, but also of other types of packet header parameters.