Backups

Performing Backups

Backups are performed using a script at /usr/local/sbin/gigatux-backup.sh on each node. The backups are initiated by scripts in /etc/cron.X, where X is daily, weekly or monthly. The parameters used in these scripts are:

• LVM volume group
• LVM logical volume (prefixed with xen)
• Email address to send reports to
• Length of time to keep backups
• Any additional rsync options

Restoring

If you need to restore a backup, look at how backups are done on each VPS (script /usr/local/sbin/gigatux-backup.sh) and use the rdiff-backup --restore functionality to restore to an appropriate location. If a full VPS restore is required, create a new logical volume to do this (lvcreate) and change the relevant /etc/xen/xenXX file to point to this to test it out.

If no SSH key exists for the [email protected] account, you will be required to enter a password. This can be found under the Talkie entry in your KeePass file.

For example, something like the following will attempt to restore a directory from two weeks ago:

rdiff-backup -v3 --restore-as-of 2W [email protected]::/mnt/gigatux/backup-lvm-new/backup-xen-node01001-14/home/dreamful /path/to/local/directory

cPanel Backups

We do not do standard cPanel backups, but we do backup the entire VPS that each cPanel instance runs on. Thus, you will need to perform backups using the full pathname. For instance, if you wanted to restore the sites for username thisuser on cpanel01001, commands similar to the below can be used:

rdiff-backup -v3 --restore-as-of 2W [email protected]::/mnt/gigatux/backup-lvm-new/backup-xen-node01001-14/home/thisuser /path/to/local/directory
rdiff-backup -v3 --restore-as-of 2W [email protected]::/mnt/gigatux/backup-lvm-new/backup-xen-node01001-14ssd/mysql/thisuser* /path/to/local/directory

The first command restores the cPanel files and the second restores the MySQL databases for this user. Be sure to restore each set of files to the correct location.

Removing

If a customer cancels their service which has a backup associated with it, we need to delete the backup. Use a command like the below to do this:

rdiff-backup --force -v3 --remove-older-than 0D [email protected]::/mnt/gigatux/backup-lvm-new/backup-xen-node00001-49

If backups fail

If, for whatever reason the backup was not taken, there is one potential solution you could use. Install the latest (SVN) version of ext3grep and attempt to restore files using this tool. The first thing to do is to create a snapshot of the logical volume (using something like lvcreate -L 20G -s -n xenXXsnapshot /dev/lvm1/xen49) and then using a combination of the following:

 ext3grep /dev/lvm1/xenXXsnapshot --restore-file path/to/file/without/initial/slash --before epoch_time

If that fails or restores the latest file (e.g. it's a bit more complicated because the file was overwritten or truncated) then you can manually restore the file (inode) at a certain time (transaction). To do this:

 * Mount the snapshot
 * Find the inode number by doing a ls -i on the file
 * Look at the journal entries for this inode by doing using the --show-journal-inodes X argument where X is the inode number
 * Restore the inode (base this on file size and timestamps) by using the --restore-inode X@Y argument where X is the inode number and Y is the transaction number

This groups article details how one person used this technique.