steve@puluka.comThis article will walk through the what and why of DDoS attacks followed by the methods that the Arbor Networks service provider platform uses to detect and mitigate attacks. Arbor is a leader in the field generally and I have been using their solution for over 5 years finding this an effective and flexible tool for DDoS mitigation.
The threat of DDoS attacks has become more widespread in recent years. The reasons typically fall into a few categories.
- Retribution – Someone has a problem with the target organization or person and is using this to punish them
- Professional criminals
- extortion – taking down sites and demanding payment to stop
- diversion – creating mehem with the security team to cover internal threats or exfiltration
- Gamers – causing delay to competitors for advantage
Both the cost and difficulty of launching attacks has also fallen at the same time, making this option both accessible and affordable.
What is DDoS
A Distributed Denial of Service (DDoS) attack is an attempt to make an online service unavailable by overwhelming it with traffic from multiple sources.
DDoS attacks fall into two main categories trying to bring an internet based service offline:
- Bandwidth Depletion
- Resource Depletion
There is a wide variety in the details on how specific attempts at depletion work. But the fundamental objective is to generate more bandwidth or resources than the internet service can supply causing legitimate requests to be unfulfilled and bringing the service down. This chart shows some of the high level categories that are used to achieve the depletion goal.
This attack traffic is typically generated by coordinated groups of computers known as botnets. In the vast majority of cases these systems are not voluntarily participating in the DDoS activity but have been compromised and taken over as members of the participating botnet. This complicates tracing back the true source of the attack since these owners are ignorant of their participation. You may have seen articles bemoaning the low security posture of many IoT devices that are easily taken over to become botnet members.
Another major source of DDoS traffic is third party server resources published on the internet. When services like DNS, NTP or LDAP are deployed on the internet they can be used to send traffic to the DDoS target from botnet members by spoofing the targets address in the request. These are known as amplification or reflection attacks. Here good sys admins can implement policies to reduce their participation in the DDoS event.
News organizations tend to focus on the size of bandwidth depletion attacks which have grown from a 2005 peak of 8Gbps to the 2018 levels over over a Tbps. But these very large bandwidth efforts are the outliers. Most attacks are very much smaller but are still effective in downing services since most systems have significantly smaller bandwidth links and resources.
More important to note is the trend towards multi-vector DDoS attacks in this same time period. Instead of simply pushing one type of bandwidth or resource attack modern DDoS botnets are now sending multiple different vectors of attack simultaneously. This can make detection and mitigation more difficult. Further complicating the response, attackers are more often changing the specific methods during the course of the attack as well.
Detecting DDoS attacks with Arbor Networks
The first step in responding to and mitigating DDoS events is detection. Arbor Networks uses statistical modeling along with thresholds of activities to flag abnormal traffic. These anomalies generate alerts that are investigated and can then inform what types of mitigation would be engaged.
For the detection process to work, data is obviously required. For enterprise networks Arbor offers and inline appliance that can be inserted on the ISP feed to the site and see all the internet traffic in full packets. This allows the modeling and anomoly detection to occur.
On large scale service provider networks running inline for all traffic on the internet is not practical. In this environment we collect data from multiple sources to build the traffic model using snmp, sampled netflow and BGP routing tables. This data from the edge routers is then used to build the traffic models and detect anomalous
The result of the statistical sampling is an understanding of what is normal. An alert then flags traffic that is outside of normal parameters. This chart shows the generic process of anomaly detection. The horizontal axis is time the vertical axis is volume. The trigger rate indicates the first statistically significant increase of volume over the norm. The high serverity rate is statistically large deviation over the normal high. The middle line is the mean between these first two levels.
Mitigations for DDoS attacks with Arbor Networks
Now that we have an alert we will want to perform some mitigation to remove the malicious traffic while allowing the normal traffic on to the published resource. Note that when an attack is volumetric in nature, this is typically going to fill the entire resource pipe to the resource. This last link from the resource to the ISP will generally be the smallest in bandwidth capacity. Which means that any mitigation for this traffic will necessarily have to occur before entering this final pipe.
There are two methods that can be used to accomplish this: ISP “clean pipe” solutions and some type of traffic diversion.
The upstream ISP where the resource is connected could perform the mitigation. These are popularly called “clean pipe” solutions and occur on the ISP network before being delivered to the resource circuit. This method does not require any changes by the resource owner to use since the traffic is naturally following this path.
With the diversion method, the traffic destined to the published resource would be redirected to an internet scrubbing center. Then after processing the clean traffic sent down to the published resource. The return traffic is generally done via a tunnel from the published resource to the scrubbing center. The simpliest diversion option is to advertise the resouce ip address from the scrubbing center instead of the local circuit. But this can only be done if a full /24 subnet is under the control of the resource owners. This is the smallest subnet that can be controled with internet routing. If the resource does not own that much ip space, they could be diverted via DNS entries if this is a published service using names and standard ports.
Both the diversion and the return traffic method have to be setup before they can be used. So they are best arranged in advance of an actual DDoS event.
Arbor Detection & Mitigation Process
The Arbor Networks Peakflow SP system does the analysis that creates the alerts and controls the mitigation options that get deployed. These can be used for both ISP clean pipes and cloud based diversion services. The monitoring of snmp, netflow and BGP mentioned above occurs on the SP system. This is where the traffic is analyzed to determine when anomalies occur and generate alerts. As a result of collecting this traffic data the SP is also a source of reporting on traffic information.
In order for profiles of traffic to have the most accuracy, Arbor uses the concept of managed objects to define subnets or hosts on the monitored network. These create a smaller scope of traffic for analysis and make the profile that is created and monitored more accurate at detecting anomolies. Thresholds for alerts can also be set at specific levels for both packets per second or bandwidth. Reporting can also be done per object. In addition to the profile detection, Arbor also has signature and heuristic methods to see malicious activity.
Once the anomaly is detected and the alert generated, Arbor can be used to launch mitigation options against the detected traffic. There are three basic methods to apply in the order they are typically deployed:
Offramp scrubbing via Arbor TMS: Offramp scrubbing via Arbor TMS deploys a range of methods to identify and drop the malicious inbound traffic while permitting the normal traffic through the system.
BGP Flowspec filters: Arbor generates ACL filters that are advertised to the edge routers using BGP Flowspec and can drop or rate limit the traffic matching the selected criteria. These can help remove large volumes of more easily identitied malicious traffic leaving just the more difficult detection to the TMS.
Destination Blackhole: For extremely large attacks that overwhelm the incoming circuits Arbor can advertise the blackhole community to upstream peers to stop all traffic to the ip address. This essentially completes the DDoS attack taking the resource offline in order to protect other resources sharing the same network.
Offramp scrubbing via Arbor TMS
Offramp scrubbing into the Arbor TMS is the main mitigation method. As the diagram shows Arbor uses BGP to direct the traffic towards the device under attack into the TMS appliance with the scrubbed traffic then returning to continue towards the resource. All other traffic remains on the native IP path during the operations so only the attack traffic needs to be processes.
The TMS uses many counter measures to inspect and determine what traffic should be allowed and what traffic to drop or rate limit during processing. The TMS maintains a blacklist where suspect ip addresses are dynamically added and removed based on the actual activity seen in the stream. The system attempts to identify bot net members on an ongoing basis and block their traffic via these dynamic blacklists.
The system will also automatically drop protocol anomalies and malformed traffic which get generated by these malicious members. For tcp based attacks the TMS is also able to use syn authentication where the TMS will respond to the initial syn packets from bot net members and look for valid responses. If these don’t come because the bot net software is just performing a syn flood the traffic will be blocked at the TMS.
In short, there are a wide variety of specific counter measures. When an alert is generated the information seen in this traffic will be used to engage the most related counter measures for the mitigation triggered from the alert. This provides a good starting point for the mitigation efforts. Based on the effectiveness of the mitigation the counter measures can be adjusted up or down in their threshold triggers. We can also add or remove specific counter measures during the event based on their demonstrated effectiveness. The running mitigation will give feedback on exactly how much traffic is being flagged by each counter measure during the event.
Preparing via Learning Mitigations
As noted, the Arbor detection system is based on understanding what is normal for traffic and flagging anomalies that could represent DDoS events. The SP system is doing this based on sampled flow data and snmp learned traffic levels. The system also provides for deeper learning about normal operations via a process called Learning Mitigations.
With this feature we divert and offramp traffic towards a managed object when times are normal. This allows the TMS appliance to now have full packets and not just flow data during normal operations. The SP can then enrich the data profile for the managed object with this fuller data set and more accurately detect anomalies.
BGP Flowspec filters
During very large attacks where the TMS offramp has a large volumn of traffic to process it can be more effective to use BGP Flowspec filters upstream of the offramp process. Certain types of reflection attacks can be easily seen simply based on the protocol and port in use by the reflection. For these a simple ACL type filter can be applied to the inbound upstream interfaces to either rate limit or drop the traffic as soon as it hits the edge router.
BGP Flowspec is supported by all the major ISP router vendors. The Arbor SP appliance can generate the Flowspec filter and advertise this via BGP to the routers to drop or rate limit the traffic on entry to the router. There is a simple visual interface to create the filter. The data in the Arbor alert allows you to determine what parameters should be included in these filters.
Destination Blackhole
Every network has physical data stream limits and not many networks, even large ISP networks, have the capacity to even transit the ultra large DDoS attacks that are occasionally generated and make global headlines. When traffic that vastly exceeds the capacity of the upstream edge links in inbound the only choice may end up being no longer accepting that traffic.
All ISPs operate destination blackhole communities where they can simply drop traffic that is being sent to a single particular ip address on your network. To do this we advertise the affected ip address via BGP with this community and the upstream carrier will then stop sending ANY traffic to this ip address. The Arbor SP system can be preprogrammed with these communities and ready to advertise them to the edge routers on demand should the need arise.
Obviously, this takes the resource offline from this carrier and in a sense completes the DDoS attack for the attacker. But if the traffic is at a level that is overwhelming all of the links availble it may become necessary to sacrifice the resource for the sake of all other resources on the network.
To use this feature you also need to research the specific communities and availability of the blackhole feature from each of your upstream network providers. The requirements and details vary per carrier and many require pre-registration in order to use the feature or even a separate BGP session to which the blackhole advertisement is sent.
Conclusion
Keeping your resources or sites online is often critical for today’s enterprise. Knowing your risk of becoming a target for a DDoS attack and the cost to being taken offline allows you to make a business decision on how much effort to put into potential solutions.
For regional ISPs the Arbor SP with TMS offramp is a solid way to protect both your own network resources and provide the service for your downstream customers as a clean pipes solution provider.
