Bandwidth Spikes - Revision history

Cocodude at 11:17, 17 August 2017

2017-08-17T11:17:15Z

← Older revision		Revision as of 11:17, 17 August 2017
Line 75:		Line 75:

	As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to pause the VPS with ''xm pause <VPS ID'' or turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.		As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to pause the VPS with ''xm pause <VPS ID'' or turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.

			=Are they spoofing IPs?=

			If it's still hard to determine who's using all the data transfer, it could be because they're spoofing their origin IP address.

			Run a tcpdump on the physical interface to determine the type of traffic that's being used. Then you might need to go through the boring process of running tcpdump on each virtual interface to determine which VPS is actually using all the data. This can be done by seeing the list of interfaces via ''ifconfig \| grep vif''. Then, running something like ''tcpdump -i vif34.0 -n'' will show the traffic on that interface. The ''34'' in this example refers to the ''ID'' field you see in the ''xm list'' output, so you can tie it to an actual Xen instance.

	Amy has recommended the user of [http://nethogs.sourceforge.net/ NetHogs] to see what processes are using up all the bandwidth. It doesn't seem to work under Cloudlinux but should be useful on the dom0.		Amy has recommended the user of [http://nethogs.sourceforge.net/ NetHogs] to see what processes are using up all the bandwidth. It doesn't seem to work under Cloudlinux but should be useful on the dom0.

Cocodude: /* Block the port? */

2016-09-08T08:30:28Z

Block the port?

← Older revision		Revision as of 08:30, 8 September 2016
Line 53:		Line 53:

	and enable it again by typing ''no shutdown''		and enable it again by typing ''no shutdown''

			You can get a general overview of the status of ports by typing ''show interfaces status'' in the enable mode.

	=Inbound bandwidth spikes=		=Inbound bandwidth spikes=

Cocodude: /* Block the port? */

2016-09-07T13:47:59Z

Block the port?

← Older revision		Revision as of 13:47, 7 September 2016
Line 51:		Line 51:
	sw0.rackc8(config)#interface GigabitEthernet0/12		sw0.rackc8(config)#interface GigabitEthernet0/12
	sw0.rackc8(config-if)#shutdown		sw0.rackc8(config-if)#shutdown

			and enable it again by typing ''no shutdown''

	=Inbound bandwidth spikes=		=Inbound bandwidth spikes=

Cocodude at 22:07, 2 September 2016

2016-09-02T22:07:01Z

← Older revision		Revision as of 22:07, 2 September 2016
Line 8:		Line 8:

	The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.		The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.

			=Coventry=

			If there is a bandwidth spike in Coventry, we have access to the switch itself so can investigate further.

			==What port?==

			The first thing we need to do is to investigate what server the spike is coming from. If you know this, skip forwards.

			If the data centre is unable to help and find an obvious IP address, we can check the [https://monitoring.gigatux.com/ monitoring] service at GigaTux which polls the switch on a minute-by-minute basis. Go to Monitoring > Graphs and go through all ''Traffic (bits/sec), 95th percentile'' graphs on the right until you find one with a high spike. Note that incoming means incoming to the switch, i.e. outgoing to the Internet.

			Note that the ports here, e.g. ''Interface GigabitEthernet0/12'' are off by one. This interface, for instance, refers to Gi0/11 on the [[Coventry]] switch.

			Another way to do this is to log into the switch itself via telnet, and issue the commands:

			enable
			show interfaces

			Then, look at the 5 minute input rate and output rate "packets/sec" figures. There should be one that stands out.

			==What MAC address==

			The switch really only has an understanding of MAC addresses, so if you want to find this out, run something like:

			show mac address-table interface GigabitEthernet0/12

			Mac Address Table
			-------------------------------------------

			Vlan Mac Address Type Ports
			---- ----------- -------- -----
			1083 d050.9979.7891 DYNAMIC Gi0/12

			You can then use a site like [http://www.macvendors.com/ Mac Vendors] to confirm the maker of the network interface.

			==Block the port?==

			If necessary, you can block the network port alone by issuing commands like:

			sw0.rackc8#configure terminal
			Enter configuration commands, one per line. End with CNTL/Z.
			sw0.rackc8(config)#interface GigabitEthernet0/12
			sw0.rackc8(config-if)#shutdown

	=Inbound bandwidth spikes=		=Inbound bandwidth spikes=

Cocodude: /* Outbound bandwidth spikes */

2014-02-11T08:28:02Z

Outbound bandwidth spikes

← Older revision		Revision as of 08:28, 11 February 2014
Line 28:		Line 28:

	As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to pause the VPS with ''xm pause <VPS ID'' or turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.		As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to pause the VPS with ''xm pause <VPS ID'' or turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.

			Amy has recommended the user of [http://nethogs.sourceforge.net/ NetHogs] to see what processes are using up all the bandwidth. It doesn't seem to work under Cloudlinux but should be useful on the dom0.

	=Letting affected users know=		=Letting affected users know=

Cocodude: /* Outbound bandwidth spikes */

2013-05-23T08:09:47Z

Outbound bandwidth spikes

← Older revision		Revision as of 08:09, 23 May 2013
Line 27:		Line 27:
	Ultimately, if there is a massive outbound spike, it's our responsibility and we will need to suspend the client's service (VPS or shared hosting).		Ultimately, if there is a massive outbound spike, it's our responsibility and we will need to suspend the client's service (VPS or shared hosting).

	As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.		As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to pause the VPS with ''xm pause <VPS ID'' or turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.

	=Letting affected users know=		=Letting affected users know=

Cocodude: /* Inbound bandwidth spikes */

2012-03-27T09:31:22Z

Inbound bandwidth spikes

← Older revision		Revision as of 09:31, 27 March 2012
Line 19:		Line 19:
	Once in as root, the easiest thing to run is ''tcpdump'' which will output packet headers. For VPS nodes, the 'real' physical network interface is peth0 (eth0 will only handle data going to the dom0 where peth0 is for all VPSs including the dom0. Run ''tcpdump -i peth0 -n'' to get an overview of the packet headers. The chances are you will see something like this:		Once in as root, the easiest thing to run is ''tcpdump'' which will output packet headers. For VPS nodes, the 'real' physical network interface is peth0 (eth0 will only handle data going to the dom0 where peth0 is for all VPSs including the dom0. Run ''tcpdump -i peth0 -n'' to get an overview of the packet headers. The chances are you will see something like this:

			[[File:tcpdump.png]]

	As you can see, there are many UDP packets going to IP address 109.169.23.113 from multiple IP addresses. This is pretty much a guaranteed DDOS attack and we have no option apart from asking the datacentre to null route 109.169.23.113.		As you can see, there are many UDP packets going to IP address 109.169.23.113 from multiple IP addresses. This is pretty much a guaranteed DDOS attack and we have no option apart from asking the datacentre to null route 109.169.23.113.

Cocodude at 09:29, 27 March 2012

2012-03-27T09:29:58Z

← Older revision		Revision as of 09:29, 27 March 2012
Line 9:		Line 9:
	The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.		The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.

	It is usually possible for a datacentre to null route an IP address (i.e. make all its traffic go nowhere).		=Inbound bandwidth spikes=

			The chances are that this is a DDOS.

			It is usually possible for a datacentre to null route an IP address (i.e. make all its traffic go nowhere). However, we need to figure out exactly what IP address they need to blacklist. Again, feel free to ask the datacentre if they can figure it out, but they might ask us to.

			The chances are that we can't SSH into the primary IP address as that will have its network port overloaded as well. We should try and get a console up via IPMI if possible. Failing that, we need to request a KVM (even if it costs us) and log in via the console.

			Once in as root, the easiest thing to run is ''tcpdump'' which will output packet headers. For VPS nodes, the 'real' physical network interface is peth0 (eth0 will only handle data going to the dom0 where peth0 is for all VPSs including the dom0. Run ''tcpdump -i peth0 -n'' to get an overview of the packet headers. The chances are you will see something like this:



			As you can see, there are many UDP packets going to IP address 109.169.23.113 from multiple IP addresses. This is pretty much a guaranteed DDOS attack and we have no option apart from asking the datacentre to null route 109.169.23.113.

			=Outbound bandwidth spikes=

			Ultimately, if there is a massive outbound spike, it's our responsibility and we will need to suspend the client's service (VPS or shared hosting).

			As with the inbound spike, we need to get to the console and run tcpdump to determine the majority of packets. When we determine the IP address, we then need to find out what VPS is it associated with. A simple way to do this is by typing ''grep <IP address> /etc/xen/*''. The simple solution is to turn off the VPS with ''xm shutdown <VPS ID>'' and destroy it (force shutdown) if it won't turn off with ''xm destroy <VPS ID>''.

			=Letting affected users know=

			Hopefully the IP address will be just for a specific VPS or shared hosting service. In which case, please send a ticket to the user saying what happened.

			If the IP address was the primary shared hosting IP address, you will need to let everyone on that particular service know. Consider using the WHMCS Mass Mail tool to email everyone active on a service (e.g. cpanel44001). You can always swap IPs around temporarily if users really require access.

			=Lifting the null route=

			Datacentres often recommend 24 hours before attempting to lift the null route. Try and wait this long if possible. When the null route is lifted, please ensure you are available to handle the likely situation that the attack is still happening and you need to request the null route again.

Cocodude: Created page with "=Overview= If you are getting a massive ping loss when pinging the main server IP, there is a high chance that someone is using up all the available bandwidth. The first thing ..."

2012-03-27T08:57:20Z

Created page with "=Overview= If you are getting a massive ping loss when pinging the main server IP, there is a high chance that someone is using up all the available bandwidth. The first thing ..."

New page

=Overview=

If you are getting a massive ping loss when pinging the main server IP, there is a high chance that someone is using up all the available bandwidth.

The first thing to determine is whether the transfers are primarily inbound (to the server) or outbound (from the server). The chances are that if is one or the other (and not both) that it is some kind of attack. If it is in both directions, it could be a legitimate high traffic situation but you still might need drastic action to stop all other VPS users from being affected.

=Ask the datacentre=

The first thing to do is to check on a datacentre control panel to see if bandwidth graphs are available. If they are, you should be able to determine the direction of the traffic. If there are no graphs, please ring up the datacentre to see what's going on and if they can provide any advice.

It is usually possible for a datacentre to null route an IP address (i.e. make all its traffic go nowhere).