Introduction
CAcert's critical servers require some protection against outage due to a single disk failure. Currently the way these servers are setup does not include provisions for RAID or mirroring at the hardware or kernel level. Therefore a simple mirroring scheme at the process level has been designed which can be retrofitted to any running server, while retaining its encryption qualities. The mirroring is not perfect, as it is driven by a script which is ran from cron at a suitable frequency: every 15 minutes is suggested, but this may be adjusted to accomodate the system's I/O load.
How to initialize the mirror disk on the webserver and signing server
Primary disk is /dev/hda, mirror disk /dev/hdc.
1. Copy the boot block from the primary disk to the mirror
Copy the boot block from the primary disk to the mirror, so we can use the mirror disk as a boot disk when necessary. There are two ways to accomplish this:
- (a) simple method, overwrites any existing partition data on mirror disk:
# dd if=/dev/hda of=/dev/hdc bs=512 count=1
- (b) careful method, preserves existing partition data on mirror disk:
# echo Copy mirror boot block including partition table data # dd if=/dev/hdc of=/tmp/ccc bs=512 count=1 # echo Use master boot block as template for new boot block # dd if=/dev/hda of=/tmp/aaa bs=512 count=1 # echo Copy partition table from mirror into master boot block template # dd if=/tmp/ccc of=/tmp/aaa skip=446 seek=446 bs=1 count=66 # echo Write patched boot block back to mirror disk # dd if=/tmp/aaa of=/dev/hdc bs=512 count=1 # echo Remove temporary files # rm -f /tmp/aaa /tmp/ccc
Method (a) is fine at this point, because in step 2 we will (re-)partition the mirror disk. Method (b) is needed when you need to do this after step 2, i.e. when the existing partitioning needs to be preserved.
2. Partition the mirror disk exactly the same as the primary disk
Partition the mirror disk exactly the same as the primary disk; i.e display primary disk partioning with:
# fdisk -l /dev/hda
then run
# fdisk -H 255 /dev/hdc
and apply the same partitioning (-H 255 seems to be necessary on Debian Etch to get the same cylinder sizes for primary disk and mirror disk ...).
3. Setup encryption for mirror disk
# dd if=/dev/urandom of=/dev/hdc5 bs=1024k count=100
(i.e. measure speed for randomizing 100 MB of the disk)
# dd if=/dev/urandom of=/dev/hdc5 bs=1024k
(this will take a very long time, you can calculate from the
size of the disk and the timing above how long; measured on
172.28.50.6 has been 1.689 MB/sec, i.e. ~53 hours for 320 GB)
(you can inspect the progress by a kill -USR1 pid-of-dd-process)
(NOTE: there should be a faster way to do this, since the Debian
installer can do it faster; need to investigate how it does it)
# cryptsetup luksFormat /dev/hdc5
(will prompt twice for passphrase, use the passphrase of hda5)
# cryptsetup luksOpen /dev/hdc5 hdc5_crypt
(will prompt again for passphrase)
# vi /etc/crypttab
add a line for hdc5_crypt similar to the line for hda5_crypt,
so the result becomes:
hda5_crypt /dev/hda5 none luks
hdc5_crypt /dev/hdc5 none luks
4. Setup LVM for mirror disk
# pvcreate /dev/mapper/hdc5_crypt # vgcreate mirror /dev/mapper/hdc5_crypt # lvs (note the LSize of the LV root in VG www | signer) # lvcreate -n root -L xxxG mirror (the number xxx should be identical to the LSize above) # pvs; vgs; lvs (check results for completeness and consistency) # mke2fs -j /dev/mapper/mirror-root # mke2fs -j /dev/hdc1
5. Adjust fstab and rsync partitions
# mkdir /mirror
# vi /etc/fstab
add two lines for the new partitions:
/dev/hdc1 /mirror/boot ext3 noauto 0 2
/dev/mapper/mirror-root /mirror ext3 noauto 0 3
# mount /mirror
# mkdir /mirror/boot
# mount /mirror/boot
# rsync --verbose --archive --hard-links --delete-after --one-file-system / /mirror
# rsync --verbose --archive --hard-links --delete-after --one-file-system /boot /mirror
# umount /mirror/boot
# umount /mirror
6. Install script for systematic rsyncing of primary disk to mirror
Install MIRROR script in /root (see Appendix for script code).
Add the following line to /etc/crontab: */15 * * * * root /root/MIRROR >>/var/log/mirror 2>&1 to enable rsyncing every 15 minutes, a log will be kept in /var/log/mirror
7. Add script to control rotation of mirror log file
Install the following file named 'mirror' in /etc/logrotate.d:
/var/log/mirror {
weekly
rotate 52
compress
delaycompress
}
Appendix: MIRROR script
# @(#)(CAcert) $Id: MIRROR,v 1.1 2009/12/29 19:51:59 root Exp $
# MIRROR - rsync / and /boot to mirror disk
LOCK=/var/lock/MIRROR.lock
# Log start of operation.
date
# Check lock or establish one (with automatic cleanup on exit)
# Note that race conditions are not an issue here, since this script
# will only be run at intervals of 15 minutes or more.
test -f ${LOCK} && exit 0
echo $$ >${LOCK}
trap "rm -f ${LOCK}" 0 1 2 3 15
# Check mirror file systems before mounting.
# Note that this also provides some protection against overlapping runs,
# since fsck will bail out for a mounted file system.
/sbin/fsck -p /dev/mapper/mirror-root || exit $?
/sbin/fsck -p /dev/hdc1 || exit $?
mount /mirror
mount /mirror/boot
# Mirror the root file system (which is encrypted) and the boot file system.
for FS in / /boot
do
echo "rsyncing ${FS} to /mirror ..."
rsync --verbose --archive --hard-links --delete-after --one-file-system \
${FS} /mirror
done
# Unmount mirror file systems to keep them offline as much as possible.
umount /mirror/boot
umount /mirror