Server is not responding
Is the server really down?
Just because you can't ping it any more doesn't mean it's necessarily down. Consider checking the following:
- You have received monitoring alerts.
- Users are submitting support tickets.
- You cannot ping or SSH to the server from multiple locations.
- [IPMI Commands] are showing that the server is not responding.
- Many VPSs hosted on the server are not responding.
Sometimes, the server can be technically not down, but pretty useless. This has happened before where an infinite stream of disk error messages are displayed in the console and all disk access has failed. For all intents and purposes, the server is down in this state.
Do not be afraid to use your best judgement here. If the server is down, you will probably need to reset it.
Pre-reset Protocol
Ideally, Marc will handle these situations, but you should follow one of the below workflows depending on whether you know Marc is probably around, or unavailable (e.g. he's on holiday).
Marc is available - the vast majority of situations
- Text Marc to let him briefly know the situation.
- Call Marc if you feel this is appropriate. You must always text though.
- Answer support queries as normal - let any worried users know that you are aware of the situation.
- Wait 20 minutes for a response. If none is received, continue.
Marc is unavailable / out of the country
- Text Marc to let him briefly know the situation.
- Answer support queries as normal - let any worried users know that you are aware of the situation.
- Wait 10 minutes for a response. If none is received, continue.
Resetting the server yourself
Marc hasn't responded and the server is down. Minimising downtime is important, so make a suitable decision as to whether the server needs to be reset. Please do not feel afraid to do this.
Check whether the Server in question uses IPMI or not, and then follow one of the following guides.
After you have issued the reset commands/request, keep checking on this. If you used IPMI, the server should come up within 10 minutes, so continuously attempt to ping the server. Issue another reset command if necessary and wait again.
The server won't restart
There may be situations where you cannot restart the server yourself and you need to get the datacentre to do this for you.
- Retrieve necessary details of the server (usually a client or servernumber, e.g. IS-xxxx or cexxxx, and phone number).
- Ring the appropriate datacentre.
- State the problem and let them know that you would this done as soon as they can. Try to get an ETA if possible. Ask them to call or email you when the reset is complete.
- If it looks like the datacentre cannot do this for a while (say, 30 minutes), consider sending the Unscheduled Outage Notification (see below) now.
- Keep pinging the server to find out when it is up.
Most datacentres will do this for free. If they say it costs money, accept this as there's nothing else we can do.
The datacentre staff are unable to restart the server
We should only get to this state if there is a major hardware failure or something has corrupted the system.
It is likely that, at this point, the server has been down for a while, so consider sending out the Unscheduled Outage Notification (see below).
Ask the datacentre to attach a KVM-over-IP setup so you can see and enter commands on the physical server. See if you are able to sort out the issue yourself. Remember, even a server that has a few VPSs running is better than none at all.
There is a hardware failure
If you see that there is a hardware failure that needs sorting immediately, see what the datacentre can do. If they can sell the hardware to you, then great - accept the price and proceed.
There is no response or you do not know what is wrong
Obviously, Google is your friend if you see something that you do not recognise. Alternately, please contact everyone on the [People] list to request assistance.
If no-one on that list are able to figure this out, then ask for professional support at the datacentre. Again, accept the costs involved. Even if this takes hours for the datacentre to sort out, a few hundred pounds spent is better than the server being offline for days.
Post-reset Protocol
At this stage, the server should now be up and running.
Basic server checks
- SSH into the server.
- su to root.
- Check that Xen virtual machines are coming up by typing xm list.
- Check that the network is working by pinging the outside world and a couple of virtual machines.
- Check that disks are up by typing cat /proc/mdstat.
Unscheduled Outage Notification
Send out an Unscheduled Outage email detailing the outage once the server is back up. This should be available in your Templates IMAP folder. Send this only to users on the affected user.
Check the list of customers for each DTC node Username: gigatux, Password: o7jJtiRQ
Inform Marc
Send Marc another text to let him know that you have saved the world.