Congestion Control in Network Security: A Comprehensive Review and Research Perspectives

1. Introduction

Congestion control is a central mechanism in packet-switched networks, designed to regulate traffic load and prevent collapse under heavy demand. The foundational work by [9] demonstrated that improper congestion handling could lead to severe throughput degradation, motivating the development of adaptive algorithms that adjust sending rates in response to packet loss and delay signals. The Transmission Control Protocol (TCP) (Postel, 1981) incorporated additive-increase multiplicative-decrease (AIMD) principles, establishing fairness and stability across the Internet.

Over the decades, numerous enhancements—including TCP Reno, NewReno, CUBIC [8], and BBR [4]—have refined congestion response mechanisms. Active Queue Management (AQM) schemes such as Random Early Detection (RED) [7] and more recent ECN-based techniques further improved responsiveness and queue stability. In parallel, data center transport protocols such as DCTCP [1] and TIMELY [18] addressed high-speed, low-latency environments.

However, the design philosophy of traditional congestion control assumed cooperative endpoints and benign traffic conditions. Modern networks challenge this assumption. Distributed denial-of-service (DDoS) attacks exploit congestion mechanisms to degrade service, sometimes using low-rate pulsing strategies that deliberately manipulate TCP retransmission timers [16]. IoT botnets such as Mirai have demonstrated how compromised edge devices can generate massive congestion events [2]. Furthermore, automated attack generation using reinforcement learning [26] indicates that adversaries are increasingly capable of adapting to detection systems.

At the same time, congestion signals themselves have become valuable security indicators. Entropy-based flow analysis [5], SDN-enabled anomaly detection [12], and graph-based deep learning approaches [30] leverage congestion patterns to identify malicious traffic. Consequently, congestion control now operates at the intersection of performance optimization and cyber defense.

This review addresses the following research questions:

How have classical congestion control mechanisms evolved across diverse network environments?

In what ways do attackers exploit congestion control dynamics?

How can congestion control mechanisms be enhanced to provide intrinsic security?

2. Foundations of Congestion Control Mechanisms

2.1 Classical TCP Congestion Control

The modern Internet’s stability is largely attributed to the congestion avoidance principles introduced by [9], who demonstrated that exponential backoff and adaptive window adjustment could prevent congestion collapse. TCP’s congestion control mechanism relies on four fundamental phases: slow start, congestion avoidance, fast retransmit, and fast recovery (Postel, 1981; Braden, 1989).

The additive-increase multiplicative-decrease (AIMD) algorithm ensures fairness by incrementally increasing the congestion window (cwnd) until packet loss is detected, at which point cwnd is multiplicatively reduced. This mechanism enables long-term fairness among competing flows ([13]; [25]). TCP Reno and NewReno refined recovery from multiple packet losses [7], while HighSpeed TCP addressed large bandwidth-delay product (BDP) environments [7].

Despite their robustness, loss-based algorithms exhibit significant limitations in modern heterogeneous networks. They treat packet loss as a primary congestion signal, which is problematic in wireless environments where losses frequently occur due to channel errors rather than buffer overflow (Bhandari et al., 2022). Moreover, AIMD struggles with fairness when round-trip times (RTTs) differ significantly between competing flows [8].

To address scalability challenges in high-speed networks, CUBIC was introduced as a cubic growth function independent of RTT, improving fairness in long-haul networks [8]. More recently, BBR (Bottleneck Bandwidth and Round-trip propagation time) departs from loss-based logic entirely, modeling bandwidth and RTT to operate at Kleinrock’s optimal operating point [4]. BBR’s model-based approach improves throughput and latency but introduces fairness concerns when coexisting with loss-based TCP variants.

2.2 Active Queue Management and Explicit Congestion Signaling

Router-level congestion signaling plays a critical role in stabilizing network traffic. Random Early Detection (RED) [7] was designed to prevent global synchronization by probabilistically dropping packets before queues overflow. However, RED’s sensitivity to parameter tuning limited its deployment (Ryu et al., 2010).

Explicit Congestion Notification (ECN) marked packets instead of dropping them, allowing endpoints to reduce rates without incurring retransmission penalties (Ryu et al., 2010). Data Center TCP (DCTCP) leveraged ECN more aggressively by estimating the fraction of marked packets, enabling fine-grained window adaptation and significantly reducing queue lengths [1].

These router-assisted approaches represent early attempts to embed congestion intelligence within the network fabric. However, they also introduce potential security vulnerabilities if attackers manipulate congestion signals or exploit marking thresholds.

2.3 Data Center Congestion Control

Data center networks (DCNs) differ fundamentally from wide-area networks due to low RTTs, high link capacities, and mixed workloads of short “mice” flows and long “elephant” flows (Benson et al., 2010). DCTCP uses ECN-based fine-grained control to maintain low buffer occupancy and reduce latency [1].

Alternative DCN protocols such as TIMELY rely on RTT gradients rather than packet loss [18]. Meanwhile, pFabric proposes near-optimal flow scheduling based on minimal packet priority metadata, significantly reducing flow completion time [1].

From a security perspective, DCNs are particularly sensitive to congestion-based attacks. Since cloud infrastructures consolidate multi-tenant workloads, congestion induced by malicious tenants can degrade service for co-located applications. This multi-tenancy amplifies the importance of enforcing fair congestion control and isolating abnormal traffic patterns.

3. Congestion Control Across Emerging Network Environments

3.1 Wireless and 5G Networks

Wireless communication introduces stochastic packet loss, fading channels, mobility, and high variability in capacity. TCP performance over 5G and mmWave networks is especially challenging due to rapid signal fluctuations and beamforming dynamics [17]. Loss-based algorithms frequently misinterpret wireless errors as congestion, unnecessarily reducing transmission rates (Bhandari et al., 2022).

Multipath TCP (MPTCP) has been proposed to exploit multiple radio interfaces simultaneously, enhancing resilience and throughput (Wischik et al., 2011). However, congestion control in multipath environments must ensure fairness across heterogeneous paths while preventing resource hogging.

5G networks further introduce network slicing, where distinct quality-of-service (QoS) requirements coexist. Congestion control mechanisms must adapt to ultra-reliable low-latency communication (URLLC), enhanced mobile broadband (eMBB), and massive machine-type communication (mMTC) simultaneously. This multi-service coexistence complicates congestion management and requires slice-aware control mechanisms.

3.2 Internet of Things (IoT)

IoT networks are characterized by constrained bandwidth, limited processing power, and energy restrictions. Congestion often arises in many-to-one communication patterns, where numerous sensor nodes transmit to a centralized gateway (Jan et al., 2017). Traditional TCP mechanisms are unsuitable due to overhead and energy consumption.

Congestion control in IoT frequently adopts hop-by-hop flow regulation, rate limiting, and queue-aware routing (Jan et al., 2017). However, IoT ecosystems have become significant contributors to DDoS activity. The Mirai botnet demonstrated that compromised IoT devices could launch large-scale volumetric attacks [2]. These attacks generate congestion not only at victims but also across intermediate infrastructure.

Consequently, congestion control in IoT must integrate anomaly detection and traffic authentication mechanisms to mitigate internal threats.

3.3 Software-Defined Networking (SDN)

Software-defined networking decouples the control plane from the data plane, enabling centralized traffic engineering and dynamic flow management (McKeown et al., 2008). SDN enhances visibility into network states and facilitates real-time congestion mitigation.

Quality-of-service management in SDN environments can leverage programmable routing and flow-level monitoring [12]. Moreover, programmable data planes using P4 allow fine-grained traffic classification and rate limiting at line speed (Bosshart et al., 2014).

From a security standpoint, SDN enables rapid DDoS detection by aggregating flow statistics across switches. However, SDN controllers themselves may become targets of congestion-based attacks, particularly control-plane saturation attacks.

3.4 Edge and Cloud Computing

Edge computing environments push computation closer to end users, reducing latency but introducing distributed congestion challenges. Cloud data centers hosting multi-tenant applications face unique risks where congestion may result from malicious or misconfigured virtual machines.

Traffic engineering solutions such as centralized WAN optimization (Hong et al., 2013) demonstrate how software-driven architectures can dynamically reallocate capacity. Integrating congestion-aware security policies into such architectures represents an active research frontier.

4. Congestion Control and Distributed Denial-of-Service (DDoS) Security Interplay

4.1 Congestion as an Attack Surface

Congestion control mechanisms were originally designed under cooperative assumptions, where all endpoints adhered to protocol rules. However, adversaries exploit congestion signals and feedback mechanisms to degrade network performance. The earliest demonstrations of TCP-targeted low-rate attacks showed that periodic traffic bursts could synchronize with retransmission timeout (RTO) mechanisms, drastically reducing throughput of legitimate flows while maintaining low average attack rates [16].

Unlike volumetric DDoS attacks, which rely on overwhelming bandwidth capacity, low-rate DDoS (LDoS) attacks exploit the intrinsic dynamics of congestion control. By sending periodic bursts aligned with TCP’s RTO intervals, attackers force legitimate flows into repeated slow-start phases, significantly reducing effective throughput [32]. This attack pattern avoids traditional threshold-based detection systems because aggregate traffic volume remains moderate.

Entropy-based detection models were introduced to identify abnormal traffic distribution patterns associated with such attacks [5]. These approaches evaluate statistical deviations in flow entropy across source IP addresses or ports. While effective against volumetric attacks, entropy-based models struggle with adaptive adversaries capable of distributing traffic more evenly.

4.2 Exploitation of TCP Congestion Feedback

Congestion control relies heavily on feedback signals such as acknowledgments (ACKs), packet loss events, and RTT variations. If these signals are manipulated, congestion control becomes vulnerable. Jero et al. (2018) demonstrated automated discovery of TCP congestion control attacks that exploit receiver-side misbehavior. By forging ACK patterns, a malicious receiver can artificially inflate the sender’s congestion window, violating fairness constraints and potentially triggering congestion collapse.

This class of attacks is particularly concerning because it does not require large-scale distributed infrastructure. A single compromised endpoint can manipulate control signals, undermining congestion fairness without generating excessive traffic.

Similarly, ECN-based mechanisms may be manipulated if routers misconfigure marking thresholds or if attackers artificially induce marking behavior to throttle competing flows. Thus, while congestion signals provide valuable information for control, they also represent potential attack vectors.

4.3 IoT Botnets and Congestion Amplification

The Mirai botnet illustrated how IoT devices could generate massive coordinated traffic floods targeting DNS providers and web services [2]. Beyond bandwidth exhaustion, these attacks create systemic congestion across intermediate routers and backbone networks.

IoT devices typically lack advanced congestion control mechanisms and often operate with minimal traffic shaping. When compromised, they transmit traffic at maximum capacity, disregarding fairness principles. As a result, IoT-originated DDoS attacks not only target specific victims but may destabilize upstream congestion control equilibrium.

Given the projected growth of IoT ecosystems, congestion-aware security enforcement at gateways becomes increasingly important. Rate-limiting policies and behavioral traffic profiling at edge routers can mitigate the risk of IoT-driven congestion storms.

4.4 SDN-Based DDoS Detection and Mitigation

Software-defined networking introduces centralized visibility and programmability, offering significant advantages for DDoS mitigation. Flow-level monitoring and rapid rule installation allow dynamic blocking or throttling of suspicious traffic [12].

Recent approaches integrate deep learning into SDN control planes. Wang and [30] proposed a spatiotemporal graph convolutional network for detecting low-rate DDoS attacks by modeling traffic dependencies across switches. By correlating QoS degradation metrics and flow statistics, the system identifies attack sources and installs mitigation rules in near real-time.

However, SDN introduces new vulnerabilities. Control-plane saturation attacks may overload the controller by generating excessive flow setup requests. Therefore, congestion control in SDN must protect both data-plane bandwidth and control-plane stability.

5. Artificial Intelligence and Machine Learning in Congestion Control and Security

5.1 Reinforcement Learning for Congestion Optimization

Traditional congestion control algorithms rely on deterministic control laws. However, modern networks exhibit non-stationary behavior due to mobility, heterogeneous applications, and adversarial interference. Reinforcement learning (RL) has emerged as a promising approach to dynamically adapt congestion parameters [31].

Machine-generated congestion control, as proposed in Remy [31], demonstrates that data-driven algorithms can outperform handcrafted TCP variants under specific network conditions. These approaches optimize throughput-delay trade-offs using simulated training environments.

More recent studies integrate deep reinforcement learning for adaptive window adjustment and rate selection, enabling dynamic adaptation to varying bandwidth and RTT conditions. Such approaches are particularly relevant for 5G and mobile networks where channel variability is high [17].

5.2 Machine Learning for DDoS Detection

Machine learning techniques are widely applied to detect DDoS patterns in traffic flows. Supervised classifiers, clustering models, and graph neural networks analyze flow features such as packet inter-arrival times, burst patterns, entropy, and queue occupancy.

SDN-based ML detection frameworks exploit centralized telemetry to identify anomalies [12]. More advanced models incorporate temporal dependencies to detect low-rate pulsing attacks [30].

Nevertheless, ML-based detection faces adversarial challenges. Attackers can modify traffic patterns to evade classifiers. Adversarial reinforcement learning has been used to generate adaptive DDoS traffic capable of bypassing detection thresholds [26]. This development underscores the need for robust, adversarially trained models in congestion-security systems.

5.3 Programmable Data Plane and Real-Time Enforcement

The emergence of programmable switches using P4 enables congestion monitoring and mitigation directly within the data plane (Bosshart et al., 2014). Real-time flow aggregation, heavy-hitter detection, and rate-limiting policies can be implemented at line speed without controller intervention.

Data-plane anomaly detection reduces mitigation latency and enhances resilience against large-scale attacks. However, limited memory and processing constraints in switches necessitate lightweight detection algorithms.

Integrating ML-based inference into programmable data planes remains an open research area, with challenges related to resource constraints and model complexity.

6. Secure Congestion Control Architectures

6.1 Network-Enforced Congestion Fairness

CRAFT [14] represents a foundational effort to enforce TCP fairness within the network itself. By emulating TCP state machines at routers and issuing cryptographic capabilities, CRAFT ensures that downstream traffic adheres to fair congestion principles. Such enforcement prevents malicious endpoints from bypassing congestion algorithms.

Network-enforced fairness shifts congestion control from end hosts to infrastructure, enhancing security in adversarial environments. However, scalability and deployment complexity remain significant barriers.

6.2 Aggregate-Based Congestion Mitigation

Aggregate-based congestion control mechanisms identify traffic aggregates responsible for excessive queue occupancy and apply rate-limiting strategies. Rather than dropping packets indiscriminately, these systems selectively throttle aggressive flows while allowing compliant TCP flows to adapt naturally.

This approach aligns congestion mitigation with fairness principles, converting DDoS events into manageable QoS degradation rather than catastrophic collapse.

6.3 Control-Theoretic Perspectives

Control-theoretic modeling of congestion dynamics provides analytical tools for stability analysis ([13]; [25]). Viewing congestion control as a feedback control system enables formal verification of stability under adversarial perturbations.

Future secure congestion architectures may integrate formal control guarantees with anomaly detection modules, ensuring bounded performance degradation even under attack conditions.

7. Open Research Challenges and Future Directions

Several research gaps remain:

Adversarially Robust ML Models: Developing congestion detection models resilient to adversarial traffic manipulation.

Cross-Layer Congestion-Security Integration: Coordinating transport-layer congestion signals with network-layer anomaly detection.

5G and IoT Scalability: Designing lightweight yet secure congestion control suitable for massive device ecosystems.

Programmable Hardware Constraints: Balancing detection complexity with hardware limitations.

Encrypted Traffic Visibility: Managing congestion and detecting attacks in increasingly encrypted transport protocols (e.g., QUIC).

Congestion-Aware Threat Intelligence: Integrating real-time congestion analytics into SOC workflows.

8. Conclusion

Congestion control has evolved from a performance optimization mechanism into a foundational component of network security architecture. Modern adversaries exploit congestion dynamics through low-rate pulsing attacks, TCP feedback manipulation, and IoT botnets. Meanwhile, AI-driven congestion optimization and SDN-based monitoring offer promising defensive capabilities.

Future networks must integrate performance efficiency, fairness enforcement, and adversarial resilience within unified congestion frameworks. Secure congestion control represents a multidisciplinary challenge at the intersection of networking, cybersecurity, and artificial intelligence.