Bandwidth Spikes

Overview

If you are getting a massive ping loss when pinging the main server IP, there is a high chance that someone is using up all the available bandwidth.

The first thing to determine is whether the transfers are primarily inbound (to the server) or outbound (from the server). The chances are that if is one or the other (and not both) that it is some kind of attack. If it is in both directions, it could be a legitimate high traffic situation but you still might need drastic action to stop all other VPS users from being affected.

Ask the datacentre

The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.

Inbound bandwidth spikes

The chances are that this is a DDOS.

It is usually possible for a datacentre to null route an IP address (i.e. make all its traffic go nowhere). However, we need to figure out exactly what IP address they need to blacklist. Again, feel free to ask the datacentre if they can figure it out, but they might ask us to.

The chances are that we can't SSH into the primary IP address as that will have its network port overloaded as well. We should try and get a console up via IPMI if possible. Failing that, we need to request a KVM (even if it costs us) and log in via the console.

Once in as root, the easiest thing to run is tcpdump which will output packet headers. For VPS nodes, the 'real' physical network interface is peth0 (eth0 will only handle data going to the dom0 where peth0 is for all VPSs including the dom0. Run tcpdump -i peth0 -n to get an overview of the packet headers. The chances are you will see something like this:

As you can see, there are many UDP packets going to IP address 109.169.23.113 from multiple IP addresses. This is pretty much a guaranteed DDOS attack and we have no option apart from asking the datacentre to null route 109.169.23.113.

Outbound bandwidth spikes

Ultimately, if there is a massive outbound spike, it's our responsibility and we will need to suspend the client's service (VPS or shared hosting).

As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing grep <IP address> /etc/xen/*. The simple solution is to pause the VPS with xm pause <VPS ID or turn off the VPS with xm shutdown <VPS ID> and destroy it (force shutdown) if it won't turn off with xm destroy <VPS ID>.

Amy has recommended the user of NetHogs to see what processes are using up all the bandwidth. It doesn't seem to work under Cloudlinux but should be useful on the dom0.

Letting affected users know

Hopefully the IP address will be just for a specific VPS or shared hosting service. In which case, please send a ticket to the user saying what happened.

If the IP address was the primary shared hosting IP address, you will need to let everyone on that particular service know. Consider using the WHMCS Mass Mail tool to email everyone active on a service (e.g. cpanel44001). You can always swap IPs around temporarily if users really require access.

Lifting the null route

Datacentres often recommend 24 hours before attempting to lift the null route. Try and wait this long if possible. When the null route is lifted, please ensure you are available to handle the likely situation that the attack is still happening and you need to request the null route again.