At home I have a small server that helps co-ordinate the various devices in our house. It is small and cheap but important for us, so I want it to be robust and cheap. I originally had it running Ubuntu server with LVM and Raid mirroring, plus rsnapshot to store historical changes. This ran very well for a few years, but the worry always came when it was time for updates or upgrades. I used an LVM root snapshot before and updates but if the system broke I would need to connect a monitor, keyboard and start putting it back together. But this wasn't always possible as I would often be administering the server remotely, and was a pain if I wanted the server up and running again fast.
Now at this point my friend (who has the same server box as me) told me he was running his server OS from a USB flash drive. It was useful for him as he could backup the usb drive for redundancy and if there was a problem he could just swap in a replica drive and carry on. I liked this idea, but wasn't sure about the amount of writing a flash drive could take on a daily basis or if it failed would it be easy to detect. So I looked for an USB distro that did not write to the USB flash. Well there are a few around and I tried some, but I was drawn back to the Ubuntu server distro as it is so easy to manage and is well supported. Looking again at running Ubuntu server on a USB flash I came up with this plan ,to install Ubuntu server on a USB flash with an overlay file system to avoid writing to USB.
I am sure there are other and better ways to do this but here is my workings:
This guide starts by walking through creating a USB bootable ubuntu server installation. Where the OS runs read only from the USB flash and all writes are to a memory file system. All commands in this guide are expected to be run as root. I found it easiest to take the risk and run everything in a shell using:
sudo -s
Bootable USB platform
The first stage involves creating the bootable USB platform which will host our OS file system. For this stage I am starting out with a fresh Ubuntu 14.04 server installation running within VirtualBox. The main reason is to avoid making any mistakes no my main system as I will be in a root shell, but you should be able to apply this process to any new ubuntu installation. Install the squashfs-tools. This will be needed later to build the initrd image with squashfs support.
sudo apt-get install squashfs-tools
Install the initram customisation tools. (These are already installed with the basic Ubunutu server installation)
sudo apt-get install initramfs-tools
Set the initram tools to include the squashfs module in the initrd image, by adding squashfs to the list of modules:
edit: /etc/initramfs-tools/modules
# Examples:
#
# raid1
# sd_mod
squashfs
You can regenerate the initrd image to include the squashfs ith:
sudo update-initramfs -ck all
though the initrd will be regenerated anyway during the next stage when we install overlayroot.
Overlayroot will provide the non-USB flash writable file system, which at this stage with be a memory file system:
sudo apt-get install overlayroot
Now it is time to assemble the USB boot platform. You will need a USB drive with a ext2 formatted partition. Most usb drives have a single fat32 partition which can be reformated with:
mkfs.ext2 /dev/sd<XY>
Where <XY> is the USB device number and partition.
Now mount the usb drive:
mkdir /tmp/usb
sudo mount /dev/sd<XY> /tmp/usb
The block id is a way to identify a block device without the device point. This is useful for USB devices as the mount points can change when they are plugged in or moved between ports. To find the block id for the USB flash:
blkid
will give something like:
/dev/sdb1: UUID="511edef4-0183-40ce-90bb-037334d46ded" TYPE="ext2"
in this case for /dev/sdb1, which is the USB flash, the block id is '
511edef4-0183-40ce-90bb-037334d46ded
'.
To boot grub with a menu create this file on the mounted USB flash: /tmp/usb/boot/grub/grub.cfg
timeout=3
menuentry 'USB' {
search --no-floppy --fs-uuid --set=root <USB_ID>
linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=tmpfs
initrd /boot/initrd.img
}
where <USB_ID> is replaced with the block id. This minimal config file will then detect the usb drive based on the block id and attempt to load the kernel and initram file system.
We need a copy of a working kernel and initrd image with squashfs and overlayfs support. If you have more than one version on your system you should a version with the updated initram above. Note how the names need to match those in grub.cfg:
cp /boot/vmlinuz-* /tmp/usb/boot/vmlinuz
cp /boot/initrd.img-* /tmp/usb/boot/initrd.img
Finally install grub on the usb drive.
grub-install --root-directory=/tmp/usb /dev/sd<X>
Where <X> is the USB flash device letter.
Finally unmount the USB flash and test:
umount /tmp/usb
You can check that the USB flash starts the grub menu by booting a machine or using qemu, VirtualBox does not currently support booting from a USB device. While a USB device can be set a raw device in VirtualBox, it still being treated as a physical SATA or IDE device:
apt-get install qemu-kvm
once installed you can test the usb drive with:
kvm -m 1024 -usb /dev/sd<X>
If you can't run kvm (for example because you have VirtualBox running), you can use qemu-system-x86_64 or similar.
When this runs you should see the grub menu and be able to edit the entry by hitting 'e' before the timeout. The rest of the boot process should load the initram image and then fail to a minimal shell as there is not yet any file system present. It is worth knowing here that if you edit the grub menu entry and add 'debug' then the initram scripts will log debug information:
timeout=3
menuentry 'USB' {
search --no-floppy --fs-uuid --set=root <USB_ID>
linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=tmpfs debug
initrd /boot/initrd.img
}
then in the minimal shell you can read the debug log with:
more /dev/.initramfs/initramfs.debug
At this stage the initramfs.debug file should report:
+ REASON=ALERT! /host/boot/root.squashfs does not exist. Dropping to a shell! PS1=(initramfs) /bin.sh -i
Building the compact OS
Now we need to build the squashed file system that we will boot into. This does not need to be the same installation used to create the bootable USB flash (which is useful as you may not want to mess with mounting and formatting partitions from within a working server in case you make a typo).
First make sure the support tools are installed:
apt-get install squashfs-tools overlayroot rsync
We need a working area to rebuild the system so create the directory:
mkdir -p /remake/tmp
If you do not have much space on the system you are going to create the file system image from, you may like to mount an extra drive to work in.
This can be really useful if you are working with an image of a file system in VirtualBox and you need additional space:
This can be really useful if you are working with an image of a file system in VirtualBox and you need additional space:
mount /dev/sd<AB> /remake/tmp
Where <AB> are the drive and partition of the extra storage area.
To prepare the file system create the following file: /remake/preroot.sh
#!bin/sh
ROOT="/remake/tmp/root"
mkdir -p ${ROOT}
rsync -vax --delete \
--exclude=/tmp/* \
--exclude=/boot/* \
--exclude=/remake/tmp/* \
/ ${ROOT}
The rsync -x option here restricts the copy to one file system which is useful to avoid pulling in data from attached disks. You may need to remove this option if you want to include other drives. You may also want to review the excluded paths from the new root image, for example you may want to exclude paths like /run if you have not included the -x option.
Then make the script executable:
chmod u+x /remake/preroot.sh
We need to make some changes to the file system copy that we don't want to update in the running system, so for this we create a patching script. This is also a good place to automate any other updates you need to the new system.
create: /remake/patchroot.sh
#!/bin/sh
ROOT="/remake/tmp/root"
PATCH="/remake/patch"
cp -ra ${PATCH}/* ${ROOT}
Then make the script executable:
chmod u+x /remake/patchroot.sh
Overlayroot will create an overlay for root '/' by reading /etc/fstab, but we want overlayroot to use the new loop squashfs from the USB, so copy the system fstab and remove or comment out the root mount point:
mkdir /remake/patch/etc
cp /etc/fstab /remake/patch/etc
In my simple build I have removed all mount points so /remake/patch/etc/fstab is an empty file.
Now the system root scripts are prepared, we can start scripting the new squash file system.
create: /remake/squash.sh
#!/bin/sh
ROOT="/remake/tmp/root"
IMG="/remake/tmp/img"
mkdir ${IMG}
mksquashfs ${ROOT} ${IMG}/root.squashfs -noappend -always-use-fragments
Then make the script executable:
chmod u+x /remake/squash.sh
Now these update scripts have been created we need to run them all:
/remake/preroot.sh
/remake/patchroot.sh
/remake/squash.sh
This will produce a root.squashfs in /remake/tmp/img, so we just need to copy that to the mounted USB flash:
mkdir /tmp/usb
sudo mount /dev/s<XY> /tmp/usb
sudo cp /remake/tmp/img/root.squashfs /tmp/usb/boot/
Now you should have a bootable USB flash you can test in a machine or qemu.
When booted mount should show root as an overlayfs and two /media mounts /media/root-ro is the read only contents of the the squashfs and /media/root-rw/overlay is the in memory writable updates.
Options
So now you have a bootable system, here are some other things that you may like to try.
Remake from USB
When you have the USB installation running you may want to make some system updates, such as package installs that you want to make permanent. One way to do this is to create a new root.squashfs image after you have made any in-memory changes. In the live USB system the /remake directory and scripts should still present (unless you excluded them in the
build).
build).
To do the remake you will probably need to mount some more working space on /remake/tmp unless you have a lot of memory or are using a disk based overlay (below).
So start by mounting your working disk space:
mount /dev/sd<AB> /remake/tmp
Then just run:
/remake/preroot.sh
/remake/patchroot.sh
/remake/squash.sh
Then copy the new root.squashfs file over, after first mounting the USB flash:
mkdir /tmp/usb
mount /dev/s<XY> /tmp/usb
cp /remake/tmp/img/root.squashfs /tmp/usb/
Note: If you changed the kernel or initramfs then you will need to replace the ones on the USB drive.
When the system is powered down the USB flash may be removed and the old file(s) replaced with the new ones.
Semi writable overlay
The above grub settings use a ram based overlay filesystem. This is useful for temporary or volatile systems, but some permanence can be achieved by using another device for the writable overlay. To use a physical device as the overlay filesystem update the boot settings to add the overlay device:
linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=/dev/sd<MN>
initrd /boot/initrd.img
Where <MN> are the writable overlay device and partition number.
Boot time config
It may be too much effort to rebuild a new root.squashfs to change only a few settings. So boot time settings may be implemented for more flexability.
Create the file: /etc/initramfs-tools/scripts/init-bottom/usbinit
#!/bin/sh
PREREQ="overlayroot"
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
# Begin real processing below this line
if [ -e /host/init.sh ]
PREREQ="overlayroot"
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
# Begin real processing below this line
if [ -e /host/init.sh ]
then
/host/init.sh
fi
and make it executable:
chmod a+x /etc/initramfs-tools/scripts/init-bottom/usbinit
then rebuild the initramfs:
update-initramfs -ck all
which will produce a new /boot/initrd.img.* file. This should be copied onto the USB flash:
mount /dev/sd<XY> /remake/usb
cp /boot/initrd.img.* /remake/usb/boot/initrd.img
umount /remake/usb
Now on reboot if there is an executable script called init.sh in the USB root directory then it will be exexuted during system startup. It will be run in a busybox ash shell with access to the USB flash mounted at /host and the new overlay is mounted at /root/media/root-rw/overlay So to set the machine hostname at boot, on the USB flash:
create:hostname
usbhost
create:init.sh
#!/bin/sh
HOST=$(cat /host/hostname)
echo “Setting hostname ${HOST}”
echo ${HOST} > /root/media/root-rw/overlay/etc/hostname
and make init.sh executable:
chmod a+x init.sh
Grub install
Currently grub-install does now work with an overlay file system, but to allow grub-install to work with our USB flash we can make a bit of a hack. Follow the above step to allow an init.sh script to be executed at boot time.
Edit: init.sh
#!/bin/sh
mkdir /root/media/root-rw/usbhost
mount -o bind /host /root/media/root-rw/usbhost
mount -o bind /host/boot /root/boot
mkdir -p /root/media/root-rw/overlay/usr/sbin
cp /root/media/root-ro/usr/sbin/grub-probe /root/media/root-rw/overlay/usr/sbin/grub-probe.orig
cp /host/grub-probe /root/media/root-rw/overlay/usr/sbin/grub-probe
This will mount the USB onto /media/root-rw/usbhost in the live filesystem and the USB /boot directory to /boot in the live filesystem. It will then rename the grub-probe file (which causes the root overlayfs filesytem detection to fail) and replace it with a file from the USB flash. This script detects a probe for root '/' and redirects it to the mounted USB device.
Then on the USB flash create: grub-probe
#!/bin/sh
ARGS=$@
SUFF="media/root-rw/usbhost"
CMD="/usr/sbin/grub-probe.orig"
ROOT=$(echo $ARGS | grep -- --target=device | grep ' /$')
if [ ! -z "${ROOT}" ]
then
ARGS=${ARGS}${SUFF}
fi
${CMD} ${ARGS}
and make it executable:
chmod a+x grub-probe
Now on boot a new /boot/grub/grub.cfg can be generated in the live usb system with:
update-grub
But if update-grub was run now, the new config file would not contain our original grub entry. To fix this add our original grub.cfg to the custom grub script before running update-grub:
cat /boot/grub/grub.cfg >> /etc/grub.d/40_custom
To make this custom grub entry permanent generate a new root.squashfs:
/remake/preroot.sh
/remake/patchroot.sh
/remake/squash.sh
and replace the root.squashfs on the USB flash.
Fstab updates
One of the features of overlayroot is that it will be default replace all mount points in the fstab with the overlayfs, and disable swap partitions. This can be a problem in a server where the main storage is assigned to fixed disks. To avoid this default behaviour, update the grub.cfg. file:
The swap option allows fstab swap partitions and recurse prevents the moving of non-root mount points.
I hope this is helpful to someone.
Please let me know of any corrections or improvements.
No comments:
Post a Comment