Unscheduled Outage Template
Jump to navigation
Jump to search
This is an example Unscheduled Outage email. Please change the relevant details.
Dear Customer, ---- Unscheduled Outage ---- Date: Saturday 22nd January Time: 21:15 - 22:00 (UK time) Affected: node01001 There were two instances over the weekend where node01001 did not operate as expected. On the first occasion, an error was reported on sdb (the second hard disk) and the root filesystem was remounted in read-only mode. On the second occasion, an error was reported on sdd (the fourth hard disk), and an infinite stream of error messages was output on the console. On both occasions, some functionality was still working (e.g. pings) but anything involving a disk access did not work as expected. These problems have happened before on this server, but previously they always affected sdd, the fourth hard disk. However, all disks look fine and on this occasion, sdb was affected, so we are not convinced that there is a hard disk issue. We will be doing the following in an attempt to determine the root cause of the problem: 1. Set up remote logging on the server. Logs show no entries at present because logs cannot be written to disk. This is now complete. 2. Attempt to switch the kernel to a standard CentOS dom0 kernel (as this is vastly different from the SuSE one we currently run). We will configure this such that the CentOS kernel is chosen automatically should the same issue occur again. 3. Order a new hard disk in case it turns out this is the problem. This has been ordered. The errors we see on the console when the problem occurs is an infinite stream of: sd 3:0:0:0: [sdd] Unhandled error code sd 3:0:0:0: [sdd] end_request: I/O error, dev sdd, sector <various> sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK sd 3:0:0:0: [sdd] CDB: Read(10): <10 octets> We are trying the above steps, but if you have any ideas, please do let us know. ------------------------- Apologies for the inconvenience and let us know if you have any queries. Regards, Your Name GigaTux -- GigaTux Customer Service [email protected] www.gigatux.com