TCP and UDP of highly protected servers are the cornerstones of Internet transmission. The protocols not only support services but also become a breakthrough used by attackers. In 2024, many enterprises will suffer certain economic losses due to UDP reflection attacks. There are some technical difficulties in the TCP/UDP protocol using a high-defense server. The details are as follows.

TCP/UDP protocol design differences affect attack path differentiation. The connection-oriented nature of TCP (three-way handshake, flow control, retransmission mechanism) makes it a prime target for state exhaustion attacks. SYN Flood attacks consume server memory by forging massive half-open connections, while ACK Flood attacks consume CPU resources by using invalid acknowledgement packets. Attackers can further combine the TCP window scaling option to trigger server resource allocation exceptions by manipulating window size parameters.

The connectionless and low overhead characteristics of UDP give birth to reflection amplification attacks. Attackers forge victim IP to send small requests to open UDP services (such as DNS, NTP), triggering a response of tens of times the traffic. The magnification factor of Memcached reflection attacks can reach 50,000x, and a 1Mbps request can generate 50Gbps of attack traffic. More subtle UDP fragment attacks use the IP fragment reassembly mechanism to send a large number of packets that cannot be completely reassembled, resulting in the overflow of the firewall cache.

High-defense servers face two core conflicts at the protocol layer: the balance between accurate identification and performance loss, and the conflict between protocol compliance and attack blocking. Protection schemes based on deep packet inspection (DPI) need to be resolved above the transport layer, which places great demands on processing power. Take TCP connections as an example. The defense system needs to maintain the connection status table (including parameters such as serial number, window size, and RTT). Tracking the status of one million connections consumes tens of GB of memory. The connectionless nature of UDP forces defense systems to simulate sessions through quintuples + time Windows, further increasing resource pressure.

Attackers exploit vulnerabilities in protocol standards to construct "legitimate attacks." For example, the TCP Fast Open (TFO) mechanism is designed to reduce latency, but an attacker can bypass the handshake process by sending a SYN packet carrying a malicious Cookie to transmit data directly, causing server resources to be occupied. If the defender disables TFO directly, the performance advantage is sacrificed. If retained, the Cookie verification mechanism needs to be enhanced, but strict verification will increase the handshake delay. Similar difficulties exist in the UDP scenario: The 0RTT feature of the QUIC protocol (HTTP/3 based on UDP) can be used for replay attacks, and the defense system needs to identify malicious patterns in encrypted traffic, which is far more difficult than traditional plaintext protocol analysis.

The traditional SYN Cookie scheme consumes a lot of computing resources to generate authentication credentials, and can not cope with the 2^48 scale source address forgery in IPv6 environment. The new generation of stateless SYN proxy technology reduces the computational overhead by 80% by pre-generating the response credentials through the hash chain, but needs to solve the problem of time synchronization and replay attacks.

In the face of slow attack detection, the behavior analysis model based on machine learning needs to extract features such as connection interval time and request content entropy, but the model misjudgment may lead to the normal long connection (such as video surveillance) being mistakenly killed. A CDN service provider used an "adaptive baseline" algorithm to dynamically learn customer business patterns, increasing the detection rate of slow attacks from 72% to 93%.

Finally, with respect to stack antagonism, the attacker simulates normal user behavior by customizing the TCP stack (such as adjusting the initial window size and selecting ACK policies). The defense system must be capable of protocol fingerprint identification, for example, to detect irregular TCP option ordering or timeout retransmission policies.

Upgrade the attack and defense of UDP Defense Stateless environment:

1. Reflection attack source tracing: Records the inbound routing information through traffic packet tags (such as IPFIX) and combines the threat intelligence database to quickly locate the exploited reflection source. Cloudflare's "backtracking" system can identify and block 95 percent of reflected nodes within 0.5 seconds.

2. Fragment reassembly defense: Adopts the dynamic fragment caching policy to automatically adjust the cache timeout period based on traffic characteristics. Huawei USG firewall implements nanosecond fragment hashing matching through hardware acceleration, keeping the CPU usage under fragment attacks within 10%.

3. Encryption protocol processing: For encrypted UDP traffic such as QUIC, the defense system needs to extract metadata (such as connection ID and number of packets) without decrypting, and identify anomalies through traffic behavior analysis (such as packet length distribution and timing characteristics). Google's MASQUE framework has implemented bypass detection of TLS 1.3 handshake features with a false positive rate of less than 0.1%.