Restoring Stability: A Datacenter’s DDoS Defense Makeover

One of the largest data centers in South America – providing webhosting, servers, server bays, and networking infrastructure – was severely compromised by a major DDoS incident. The attack repeatedly took the data center completely offline several times over the course of a few days. All the business’s customers were affected by hours of downtime, which was a clear reputational hit.

The attackers managed to overcome the company’s cloud-based cybersecurity solutions, provided by two separate vendors, as well as their on-premises network and server protection technology. The BGP-based threat (L3 or L4) mitigation solutions were just insufficient, with one of them failing to accurately identify the attack and the second relying on a scrubbing center located far outside the region where the company is located. The distance resulted in latency issues that made it hard to implement effective mitigation, but contracting additional local vendors was not a cost-effective option either. 

The Solution

The company called Red Button to help them assess and fix the security gaps. We began by reviewing their architecture, determining that their protection level was below that required for their industry. So, we configured the data center’s cloud-based protection services and upgraded the on-prem protection technology accordingly. We also took a look at the DNS, though it was not targeted in the large-scale attack, and improved its resilience with targeted technology changes. For the company’s security teams, we drafted a DDoS playbook with written procedures for handling such attacks and planned DDoS war games for training.

With those changes in place, we designed and implemented a DDoS test that included Layer 3 and Layer 4 attacks (UDP, SYN, and ACK floods) challenging the data center’s cloud protection services. We also simulated Layer 7 attacks on the DNS (Reflective, Garbage, Query, Recursive) and HTTPS vectors, as part of our preemptive approach to cybersecurity.

The Results

The multifaceted DDoS test showed that our adjustments had a positive effect, with mitigation of the simulated flood attacks on the data center’s cloud-based services.

Our suspicion that the DNS was vulnerable was proven correct when we saw that DNS Query and DNS Recursive floods were not mitigated during testing. Neither were our simulated HTTPS attacks. However, it bears noting that DNS Reflective and DNS Garbage floods were in fact mitigated by the company’s DDoS protection technology.

Recommendations

We recommended the following measures to improve the company’s DDoS mitigation outcomes across the board:

• Add DNS service redundancy: Avoid a single point of failure – especially critical for data centers – by ensuring you have two separate DNS services available and operational. They can be on-prem or cloud-based, or a combination of the two.
• Keep software up to date: Ensure on-prem solutions are regularly upgraded with the latest mitigation capabilities and higher capacities.
• Add web (HTTPS) protection: Use DNS diversion to ensure requests are routed through CDNs, DDoS protection services, WAFs, and L7 proxies. These options are effective and easy to manage at a relatively low cost.

• Retest L7 DDoS protection: Once the previous recommendations are implemented, it’s time to retest the defenses against the previously executed DDoS scenarios. In addition, it is good practice to run another DDoS war game with the data center’s security team.