Bandwidth Spikes

Overview

If you are getting a massive ping loss when pinging the main server IP, there is a high chance that someone is using up all the available bandwidth.

The first thing to determine is whether the transfers are primarily inbound (to the server) or outbound (from the server). The chances are that if is one or the other (and not both) that it is some kind of attack. If it is in both directions, it could be a legitimate high traffic situation but you still might need drastic action to stop all other VPS users from being affected.

Ask the datacentre

The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.

Coventry

If there is a bandwidth spike in Coventry, we have access to the switch itself so can investigate further.

What port?

The first thing we need to do is to investigate what server the spike is coming from. If you know this, skip forwards.

If the data centre is unable to help and find an obvious IP address, we can check the monitoring service at GigaTux which polls the switch on a minute-by-minute basis. Go to Monitoring > Graphs and go through all Traffic (bits/sec), 95th percentile graphs on the right until you find one with a high spike. Note that incoming means incoming to the switch, i.e. outgoing to the Internet.

Note that the ports here, e.g. Interface GigabitEthernet0/12 are off by one. This interface, for instance, refers to Gi0/11 on the Coventry switch.

Another way to do this is to log into the switch itself via telnet, and issue the commands:

 enable
 show interfaces

Then, look at the 5 minute input rate and output rate "packets/sec" figures. There should be one that stands out.

What MAC address

The switch really only has an understanding of MAC addresses, so if you want to find this out, run something like:

 show mac address-table interface GigabitEthernet0/12

           Mac Address Table
 -------------------------------------------
 
 Vlan    Mac Address       Type        Ports
 ----    -----------       --------    -----
 1083    d050.9979.7891    DYNAMIC     Gi0/12

You can then use a site like Mac Vendors to confirm the maker of the network interface.

Block the port?

If necessary, you can block the network port alone by issuing commands like:

 sw0.rackc8#configure terminal                                  
 Enter configuration commands, one per line.  End with CNTL/Z.
 sw0.rackc8(config)#interface GigabitEthernet0/12
 sw0.rackc8(config-if)#shutdown

and enable it again by typing no shutdown

Inbound bandwidth spikes

The chances are that this is a DDOS.

It is usually possible for a datacentre to null route an IP address (i.e. make all its traffic go nowhere). However, we need to figure out exactly what IP address they need to blacklist. Again, feel free to ask the datacentre if they can figure it out, but they might ask us to.

The chances are that we can't SSH into the primary IP address as that will have its network port overloaded as well. We should try and get a console up via IPMI if possible. Failing that, we need to request a KVM (even if it costs us) and log in via the console.

Once in as root, the easiest thing to run is tcpdump which will output packet headers. For VPS nodes, the 'real' physical network interface is peth0 (eth0 will only handle data going to the dom0 where peth0 is for all VPSs including the dom0. Run tcpdump -i peth0 -n to get an overview of the packet headers. The chances are you will see something like this:

As you can see, there are many UDP packets going to IP address 109.169.23.113 from multiple IP addresses. This is pretty much a guaranteed DDOS attack and we have no option apart from asking the datacentre to null route 109.169.23.113.

Outbound bandwidth spikes

Ultimately, if there is a massive outbound spike, it's our responsibility and we will need to suspend the client's service (VPS or shared hosting).

As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing grep <IP address> /etc/xen/*. The simple solution is to pause the VPS with xm pause <VPS ID or turn off the VPS with xm shutdown <VPS ID> and destroy it (force shutdown) if it won't turn off with xm destroy <VPS ID>.

Amy has recommended the user of NetHogs to see what processes are using up all the bandwidth. It doesn't seem to work under Cloudlinux but should be useful on the dom0.

Letting affected users know

Hopefully the IP address will be just for a specific VPS or shared hosting service. In which case, please send a ticket to the user saying what happened.

If the IP address was the primary shared hosting IP address, you will need to let everyone on that particular service know. Consider using the WHMCS Mass Mail tool to email everyone active on a service (e.g. cpanel44001). You can always swap IPs around temporarily if users really require access.

Lifting the null route

Datacentres often recommend 24 hours before attempting to lift the null route. Try and wait this long if possible. When the null route is lifted, please ensure you are available to handle the likely situation that the attack is still happening and you need to request the null route again.