A company providing AI-driven business intelligence contacted Red Button after suffering a concerning DDoS attack, which included NTP and SYN floods.
AWS provides the company’s digital infrastructure, with AWS Shield Advanced and WAF rules for cybersecurity. Certain user requests reach an AWS Application Load Balancer (ALB) associated with the AWS WAF, which then forwards the request on to the origin servers (an EKS cluster). Other requests are first funneled to AWS Global Accelerator before reaching the ALB. If they are not blocked by the ALB’s WAF, then the requests are sent to an AWS Fargate service. A third company service that Red Button looked at uses AWS CloudFront to handle content delivery. Static content requests are forwarded to an AWS S3 Bucket instance and dynamic content requests go to an ALB and then AWS Fargate origin containers.
The Solution
Red Button experts reviewed the company’s architecture and AWS configurations in the wake of the DDoS attack, noting vulnerabilities such as potential “direct-to-origin” attacks targeting ALB public IPs or AWS Fargate. As initial basic measures to mitigate risks, we recommended hardening inbound traffic rules for the AWS security group and fine-tuning PPS and RPS quotas as part of a layered security defense. Assets behind AWS Global Accelerator or AWS CloudFront were less vulnerable to network-layer DDoS attacks unless their origin IPs were exposed.
We designed a DDoS simulation involving five attack scenarios to test the effectiveness of the hardening actions. As an AWS DDoS test partner, Red Button is automatically authorized to conduct such simulations against AWS-hosted infrastructures. Our network-layer attacks specifically targeted the ALB used for the company’s log in service, while application-layer testing evaluated protections for three of company’s web services.
The Results
The simulation’s network-layer attacks – NTP, SYN, TLS Connections, and IPv4 floods – were mitigated effectively. In two cases, there were very slight and very brief increases in latency, and then AWS Shield Advanced successfully restored normal service.
However, the application-layer attacks (HTTPS/1.1 GET) against multiple endpoints impacted the company’s services with varying levels of severity. Two services initially absorbed the traffic, but one of them suffered 8 minutes of downtime and server unavailability until mitigation and rate-limiting measures restored access. When the simulated DDoS attack was relaunched at a much higher rate, AWS Shield Advanced mitigation took 10 minutes to kick in.
Recommendations
In light of the DDoS simulation, which was intended to test our initial generic guidance, we recommended that company take the following measures:
- Remediate the rate-limit failure: The rate-limit protection rules took approximately 8 minutes to mitigate the HTTPS GET flood attack, during which the login service experienced downtime. AWS should be contacted to address such delays, followed by retesting relevant DDoS attack vectors to verify any configuration changes.
- Deploy global infrastructure: The login service is resolved directly to the ALB, exposing it to network-layer attack vectors. AWS CloudFront or Global Accelerator should be deployed as redundant infrastructure to enhance protection and absorb excessive network traffic destined for the service.
- Retest network attack vectors: Some attack scenarios caused increased latency before Shield Advanced mitigation was initiated. Network-layer attack scenarios should be retested at higher rates to determine if mean time to mitigate (MTTM) remains reasonable and performance degradation is not exacerbated.
