The rise of packet rate attacks: When core routers turn evil

Generated with AI.

This post was co-authored by Christophe Bacara.

Update: As of 04/07/2024 we are currently working with MikroTik to help resolve these issues.

Distributed Denial of Services (DDoS) attacks are a persistent threat continuing to be an effective way to impact the availability of online services. Over the past decade, several actors demonstrated how easily they could raise an army of zombie devices with their botnet using a wide range of techniques, from phishing to installing malware on a desktop host to leveraging different vulnerabilities affecting IoT devices, CCTV, or residential routers.

These botnets have mainly been used to launch DDoS attacks, leveraging tens of thousands of compromised devices globally. The attack strategy is often the same - generate as many packets as possible to maximize the bit rate or packet rate, overwhelming the target's network. This was the case with the Mirai botnet in 2016, which was the first to generate over 1Tbps of traffic. Since then, other botnets have exceeded Mirai, with traffic reaching up to 3.47Tbps in 2021. However, attacks exceeding 1Tbps were relatively rare - until recently.

Since the beginning of 2023, our teams at OVHcloud noticed a sharp increase in DDoS attacks, both in frequency and intensity. Moreover, starting from November of the same year, a significant acceleration of the trend has been observed. In the past 18 months, we went from 1+Tbps attacks being quite rare to seeing them weekly, to almost daily (averaged out over one week). The highest bit rate we observed during that period was ~2.5Tbps.

Interestingly, the recent announcement of the 911 S5 Botnet dismantling between 25 to 30 May 2024 loosely coincides with a significant decline in DDoS attacks, which started by mid-May. However, we cannot affirm with certainty these events are linked together.

While attack frequency is seemingly back to normal, we are still observing a large amount of DDoS attacks involving packet rates greater than 100 million packets per second (Mpps).

Interestingly, the recent dismantling of the 911 S5 botnet between 25 and 30 May 2024, coincided with a noticeable drop in DDoS attacks that began in mid-May. However, we cannot definitively say these events are connected. While the frequency of attacks has returned to normal, we are still seeing a significant number of DDoS attacks with packet rates exceeding 100Mpps.

What about packet rate attacks?

Usually, most DDoS attacks rely on sending a lot of garbage data to saturate the bandwidth (network-layer attacks) or by sending a lot of application requests to cause excessive CPU or memory usage (application-layer attacks). Of course, there are other methods to leverage such as packet rate attacks or packets-per-second-based attacks.

The objective of packet rate attacks is to overload the packet processing engines of networking devices close to the destination, instead of starving the available bandwidth. The general idea is to cripple the infrastructures in front of the targeted service (such as load-balancers, anti-DDoS systems, and so on), thus possibly impacting a large infrastructure as collateral damage. Simply put, instead of trying to find holes in anti-DDoS systems, just take them down!

Packet rate attacks are quite effective because dealing with a lot of small packets is usually harder than dealing with bigger but less numerous packets. This is because the computing cost is generally higher. For instance, if you're using software to process packets, each packet means one memory access at the very least (excluding possible copy, access to stored data such as connection tables, and so on), instead of simply iterating over more bytes.

If you're using hardware, while packet processing performances are not necessarily affected by the packet rate, the processing pipeline probably depends on other components, such as memory (again!), which could be significantly stressed by high packet rates. In those conditions, you may reach some limits due to the very high rate or just because you don't have enough buffers to store it all, which will probably induce latency or performance losses. We can summarize this problem into a single sentence - if your job is to deal mostly with payloads, bandwidth may be the hard limit; but if your job is to deal mostly with packet headers, packet rate is the hard limit.

That's why in most conditions, dealing with small packets is harder than dealing with big packets. In a nutshell, a 10Gbps DDoS attack with big packets (1,480 bytes) yields ~0.85Mpps. In comparison, 10Gbps with the smallest packets (84 bytes on wire for Ethernet) yields a massive ~14.88Mpps.

In the context of standard Internet MTU (1,500), you can fit 17 times more packets on the wire when generating only the smallest packets possible, compared to big packets. To give an idea of the computing capabilities needed in the context of DDoS mitigation, consider a 100Gbps link that will fit ~149Mpps line rate. This allows up to 6 nanoseconds of processing time per packet, or 18 cycles for a single compute pipeline running at 3GHz clock speed. To put it another way, even with tens of parallel pipelines, you won't have many cycles available, especially if you need to access some memory.

The rise of (big) packet rate attacks

DDoS attacks relying on high packet rates are not new and network operators all over the world have had to face such attacks at least once. As an example, the highest publicly-known packet rate attack was reported by Akamai in 2020 and reached 809Mpps. Despite this big figure, a vast majority of packet rate attacks are well below 100Mpps. This is probably because generating several small packets is harder than generating big ones - you need much more compute power (similar to processing) and it's harder to hide from network monitoring and anti-abuse systems.

Packet rate attacks started to get some serious attention at OVHcloud two years ago after we were hit - but successfully mitigated - by a gigantic UDP flood for more than six hours, reaching ~700Mpps on average for roughly four hours.

In the past 18 months, and especially in the past six months, we noticed a sharp increase in DDoS attacks leveraging packet rates greater than 100Mpps. We went from mitigating a few of them each week, to tens or even hundreds per week. Our infrastructures had to mitigate several 500+Mpps attacks at the beginning of 2024, including one peaking at 620Mpps. In April 2024, we even mitigated a record-breaking DDoS attack reaching ~840Mpps, just above the previous record reported by Akamai.

This attack was 99% of TCP ACK, originating from approximately 5,000 source IPs. Interestingly, the 1% remainder was a DNS reflection attack, leveraging ~15,000 DNS servers to amplify the traffic, which is not really efficient when trying to achieve high packet rate attacks.

While the attack was distributed worldwide, 2/3 of the total packets entered from only four Points of Presence (PoPs), all located in the US and three of them on the West Coast. This highlights the capability of the adversary to send a huge packet rate through only a few peerings, which can prove very problematic. Generally, anti-DDoS response teams assume that it's quite hard to send massive DDoS from only a few geographical locations. Based on this assumption, our infrastructures are scaled horizontally and spread worldwide, so they absorb the load more easily. However, the traffic distribution of the 840Mpps attack has strongly questioned this assumption. While we do have the local capacity to mitigate this attack, we will consider adjusting the general scaling and distribution model of our anti-DDoS infrastructures to ensure they cope with future (and probably bigger) attacks, just as we do today.

In the end, the significant rise in high packet rate attacks led us to deep dive into the topic. As a worldwide cloud provider, OVHcloud scrubs many DDoS attacks daily, which gives us an exceptional vantage point on this topic. We wanted to understand how these attacks were generated, where they came from, and possibly determine what we could do to better protect our infrastructures and customers against this kind of firepower.

Unveiling evil core routers

During our analysis, we manually examined nearly a hundred packet rate attacks ranging from 100 to 500Mpps. We noticed that a significant portion of the traffic came from a relatively small number of sources. We identified a list of well-known IP addresses capable of generating at least 1Mpps each and decided to investigate further.

We analysed the top 70 IPs issuing the highest packet rates, up to 14.8Mpps per IP. These IPs belong mostly to Autonomous Systems (ASes) in Asia but Europe, the Middle East, North America and South America are also represented. The ASes identified seem to belong mostly to ISPs or cloud connectivity providers.

To identify the types of devices involved in these DDoS attacks, we used Onyphe to check if the IPs were already known. Many of these IPs were indeed recognized as MikroTik routers, which were at least exposed to the Internet via their configuration web pages.

At this point, it remains possible this traffic is generated either by servers located behind a router configured with Network Address Translation (NAT), using spoofed IPs, or leveraging some kind of weird TCP reflection. However, we quickly dismissed these hypotheses due to the improbability of encountering such a significant number of identified MikroTik routers, given that MikroTik does not hold a proportionately large market share worldwide.

In addition, exposing an administration interface reflects poor management practices. It increases the device's attack surface and can facilitate its compromise by an attacker. Moreover, RouterOS - MikroTik's operating system - has suffered from several critical CVEs over the past years. Even if a patch has been released, these devices may have not been patched yet.

Since the HTTP interface is open on most of the devices, it is possible to use it to recover the version of RouterOS running on said devices. Half of them are running a RouterOS version before 6.49.8 - released on 23 May 2023 - and the other half is running a later version. For instance, devices running RouterOS 6.49.14 released on 4 April 2024 have been identified.

We have been surprised to discover devices with recent firmware being potentially compromised though. As far as we know, no vulnerability affecting RouterOS 6.49.14 and later versions have been publicized so far. A possible explanation would be these devices may have been patched after being compromised.

We can't say yet why these devices are involved in coordinated DDoS attacks, but one possible hypothesis could be the Bandwidth Test feature in RouterOS. It allows the administrator to test the real throughput of a router by crafting packets and performing stress tests. Coincidentally, the documentation states: "Up to RouterOS version 6.44beta39, Bandwidth Test used only single CPU core and reached its limits when the core was 100% loaded. Bandwidth Test [now] uses all available bandwidth (by default) and may impact network usability". This is quite interesting since we mostly identified RouterOS v6.44 or above among the offending IPs.

99,382 devices available on the Internet

Using Simple Network Management Protocol (SNMP) on devices exposing it, we have been able to determine what kind of devices were capable of issuing such a high packet rate. As expected these are not residential routers, but rather core network devices.

By using the Simple Network Management Protocol (SNMP) on devices that had it exposed, we were able to identify the types of devices capable of generating such high packet rates. As expected, these were not residential routers, but core network devices.

Figure 6 highlights the MikroTik Cloud Core Router (CCR) series. Indeed, SNMP returned several CCR1036-8G-2S+ and CCR1072-1G-8S+.

To have an overview of how many devices could be compromised and used in such massive packet rate DDoS attacks, we used Onyphe once again to search for CCR devices wide open on the Internet. 99,382 CCR devices were identified.

Our analysis shows that the two device models involved in the packet rate attacks, CCR1036-8G-2S+ and CCR1072-1G-8S+, account for at least 40,000 devices exposed on the Internet. The CCR1036-8G-2S+ is the most common, with 30,976 instances found, while the CCR1072-1G-8S+ ranks fourth with 9,353 instances. Although we don't yet know the specific vulnerability exploited to compromise these models, it's unclear whether other CCR models might also be affected. However, exposing the administration panel to the Internet remains a significant security risk.

Even more evil models?

Thanks to internal data sharing and discussions, we were reminded of a Layer 7 (L7) attack that occurred in November 2023. Although MikroTik routers were identified at the time, it didn't raise concerns. This attack reached 1.2 million HTTPS requests per second and involved around 3,000 source IPs. Given our recent findings, we decided to reexamine the incident.

To determine which types of routers were involved, we retrieved the 3,000 IPs linked to the attack. Previous investigations revealed that around 700 of these IPs were identified as MikroTik routers, with TCP port 8291 exposed. However, we didn't investigate the specific devices involved at that time.

As before, we conducted a quick search using Onyphe, manually at first, and found similar results: CCRs were also involved in this attack. Among the IPs identified as CCR devices, over 10% had SNMP publicly exposed. Once again, we discovered core network devices were targeted - 16% of the exposed devices were the CCR1009-7G-1C-1S+, another similar model. This model is the second most exposed on the Internet, as noted in our earlier findings.

Determining the RouterOS version running on identified devices eight months ago is probably not relevant, since we can't say which version was running on the device at the time. However, analysis shows that 22% of the devices are running a RouterOS released between 1 July 2023 and 1 July 2024. The most recent version is v6.49.15 (24/05/2024), while the oldest one is v5.20 (15/08/2012).

Unfortunately, we no longer have enough data to estimate the request rate for each device model.

As mentioned earlier, it's still unclear how these devices were compromised. Similarly, it's difficult to determine whether the high packet rate attacks and the L7 attacks are related or if the same botnet was involved in both. Nonetheless, identifying core network devices in L7 attacks is significant, as it highlights the potential threat these devices pose.

Let's do some maths

In order to hint at the possible capacity of a botnet leveraging these devices, we decided to focus on packet rate attacks that are well identified.

A quick look at the advertised capabilities of the identified devices shows they can handle up to 28Gbps for the CCR1036-8G-2S+ and 80Gbps for the CCR1072-1G-8S+. In terms of packet rate, these devices claim to handle near the theoretical packet line rate based on their bandwidth capacity. To clarify, a 1Gbps link can handle a maximum of about 1.5Mpps.

However, the number of packets per second a device can generate varies depending on whether it is crafting packets (typically handled by the CPU) rather than simply forwarding them (usually done by ASICs). This means the actual packet rate could be much lower than the advertised processing capacity. Additionally, generating traffic using the hardware of a compromised device is complex, and an attacker would likely rely on CPU capabilities or repurpose built-in features for unintended uses.

In the context of this thought exercise, we assume networking devices are capable of crafting packets at a rate equal to 10% of their maximal capacity, leading to the following assumptions:

• CCR1036-8G-2S+ should be able to generate 4Mpps each
• CCR1072-1G-8S+ should be able to generate 12Mpps each

These estimations seem mostly accurate when compared to what we actually observed in terms of packet rates depending on the identified model. Considering the available CPU on these devices, we think these estimations are quite conservative. For instance, in the case of CCR1036-8G-2S+ devices, generating more than 4Mpps with 36 cores at 1.2GHz should not be difficult.

At this point, anyone can do the maths to build a naive scale model of a botnet leveraging these devices. Considering a rate of 1% (arbitrary conservative value) of exposed devices being compromised, and focusing only on the first two models we identified as compromised:

• ~ 300x CCR1036-8G-2S+ / 4Mpps each
• ~ 90x CCR1072-1G-8S+ / 12Mpps each

Such a botnet would theoretically be able to generate 2.28B packets per second (or Gpps).

In terms of requests per second capacity for L7 attacks, we do not have enough data to form a strong enough hypothesis. We can only affirm these devices seem capable of performing L7 attacks and high packet rate attacks. Attempting to estimate possible L7 capacity is left as an exercise for the reader.

Conclusion

The evidence presented in this article points to a new trend of using compromised network core devices to launch powerful attacks. While MikroTik devices have been implicated in DDoS attacks before, there was no prior indication that botnets were specifically relying on core network devices to carry out these attacks.

While any high-end server could perfectly be capable of generating packet rates at this scale, they will probably be limited by the actual amount of available public bandwidth. Because of their location within the network, core devices are much less affected by this assertion. They are generally connected to even bigger devices using high-capacity network links. Moreover, the mitigations implemented by network administrators to identify abnormal behaviours on the network, such as servers initiating DDoS attacks, can be bypassed in this case because routers are generally not subject to these measures.

Depending on the number of compromised devices and their actual capabilities, this could be a new era for packet rate attacks. With botnets possibly capable of issuing billions of packets per second, it could seriously challenge how anti-DDoS infrastructures are built and scaled. We will definitely take this new threat into account when thinking about how we build and scale our own anti-DDoS infrastructures to make sure we remain out of any possible impact.

The security of network devices is both a pressing concern and an actual issue. Since 1 January 2024, more than 10 critical CVEs have been released affecting various network devices from multiple vendors (such as Ivanti, Cisco, Fortinet, Palo Alto, and so on). Some of them were even exploited in the wild before the public release of the CVE. However, this is the first time we face network core devices participating in coordinated DDoS attacks. This is somewhat concerning since identified devices are designed for small and medium-sized network cores, and much more powerful equipment is available today.

Closing note: We reached out to MikroTik through several communication channels to expose the situation to them. MikroTik reached out to us on 04/07/2024 and is investigating the possible causes of the issue.

We are also currently contacting various ASes to report the issue to them.

Sebastien Meriot is the Head of the Computer Security Incident Response Team at OVHcloud.

Christophe Bacara is the Technical Lead of Network Security Engineering at OVHcloud.

This post was originally published on the OVHcloud blog.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Public link

Please use the above public link if you want to share this noodl on another website.