Network Exploitation and Defensive Architecture
Advanced Packet Analysis
Beyond the Surface
You've seen packets fly by in Wireshark, but the real story isn't just in the traffic flow. It's hidden in the details. The Transmission Control Protocol/Internet Protocol (TCP/IP) suite is the foundation of the internet, designed for reliability, not security. This design choice means its mechanisms can be manipulated for reconnaissance and attack.
Deep packet inspection goes beyond looking at source and destination IPs. It's the art of dissecting the headers of each packet, byte by byte, to find anomalies. These deviations from standard behavior can reveal open ports, identify the target's operating system, and even bypass primitive security measures. We're moving from being a passive observer to an active analyst, learning to read the language of the network for offensive security.
Investigators use network analysis tools (such as Wireshark or tcpdump) to capture and examine data packets flowing across a network.
The Handshake as a Weapon
The standard TCP three-way handshake (SYN, SYN-ACK, ACK) is a polite conversation that establishes a stable connection. But attackers can weaponize this process. The most common technique is the SYN stealth scan, also known as a half-open scan.
Instead of completing the handshake, the attacker sends a SYN packet to a target port and waits. If a SYN-ACK packet comes back, the port is open. The attacker then sends an RST (Reset) packet instead of the final ACK, abruptly tearing down the connection before it's fully established. Many logging systems only record completed connections, so this method often flies under the radar. It’s like knocking on a door and running away when someone answers. You find out they're home without ever having a conversation.
To catch this in , you can't just look for established sessions. You need to hunt for individual packets. A filter like tcp.flags.syn == 1 and tcp.flags.ack == 0 will show you the initial SYN packets from a potential scan. If you see a SYN-ACK response followed by an RST from the original source, you've likely found a stealth scan.
Weaponizing TCP Flags
Beyond the SYN flag, other TCP flags can be manipulated for stealthy reconnaissance. Many older, stateless firewalls were configured to block incoming connections by simply dropping any packet with the SYN flag set that wasn't part of an established session. Clever attackers realized they could send packets with other flags set to see how a target responds.
These techniques are less effective against modern stateful firewalls, but they are still used to probe for misconfigurations and identify older systems. They work by exploiting a key part of the TCP specification (): a closed port must respond to any probe packet (that doesn't have the RST bit set) with an RST packet, while an open port must ignore it.
| Scan Type | Flags Set | Open Port Response | Closed Port Response |
|---|---|---|---|
| FIN Scan | FIN | No response | RST/ACK |
| Xmas Scan | FIN, PSH, URG | No response | RST/ACK |
| Null Scan | None | No response | RST/ACK |
In each case, the attacker is looking for silence. If they send a packet and get nothing back, they can infer the port is open. If they get an RST/ACK, the port is closed. The Xmas scan gets its name because with three flags turned on, the packet is said to be "lit up like a Christmas tree."
Fingerprinting an Operating System
Not all systems are built alike. While TCP/IP is a standard, different operating systems implement the protocol stack with subtle variations. Attackers exploit these differences to determine the OS of a target, a process called or TCP/IP stack fingerprinting.
This is done by sending a series of malformed or non-standard TCP packets and analyzing the responses. For example:
- Window Size: Different OSs use different initial TCP window sizes.
- TTL and DF: The initial Time-to-Live (TTL) value and whether the Don't Fragment (DF) bit is set can be strong indicators.
- Option Order: The order of TCP options in the header can vary between, say, a Windows machine and a Linux machine.
- Response to Malformed Packets: How does the OS respond to an Xmas scan or a packet with both the SYN and FIN flags set (which should never happen)? The specifics of the RST packet it sends back can give away its identity.
This is why deep packet inspection is so crucial. By capturing these responses in Wireshark and examining the headers closely, you can manually piece together the identity of a remote host. An attacker doesn't need to see the machine to know what it is; they just need to listen to how it talks on the network.
In a TCP SYN stealth scan, what packet does the attacker send after receiving a SYN-ACK from the target to avoid completing the connection?
When conducting a FIN, NULL, or Xmas scan against a system following RFC 793, what response from the target port indicates that the port is likely open?
By understanding how these protocols can be bent and broken, you can better recognize the signatures of reconnaissance and active attacks on your own networks.