Multi Overlay Chroot Server

Introduction

This project is an extension of my Arch Memory Safe Server From USB project. The aim of this project is to extend the memory safe server to host a range of applications in chroot environments. The purpose being to keep each service discreet to that it can be updated, replaced or moved independently of the other services and the host machine.

Mounting

As the new chroot environments will be stored on the USB partition, we need to make sure that it is mounted. To do that from the Arch usb install first mount the usb partition and chroot into it to make permanent changes (mount /dev/loop0 /mnt && arch-chroot /mnt /bin/bash). You can of course back up the root.img before making the changes (mkdir /tmp/usb && mount <usb_partition> /tmp/usb && cp /tmp/usb/boot/root.img /tmp/usb/boot/root.img.bak).

Then to mount the usb at boot time create the mount point (mkdir /mnt/usb), and add the usb block id to the fstab (blkid). /etc/fstab:

#
# /etc/fstab: static file system information
#
UUID="DCF4-6289" /mnt/usb vfat defaults 0 0

Build the overlay

Next we create a script that will build an overlay file system using images /sbin/overlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1
shift

mkdir -p /mnt/chroots/${NAME}/{loop,lowers,tmpfs,root}
mount -t tmpfs tmpfs /mnt/chroots/${NAME}/tmpfs
mkdir /mnt/chroots/${NAME}/tmpfs/{upper,work} 
mount /dev/loop0 /mnt/chroots/${NAME}/loop 

LOWER=/mnt/chroots/${NAME}/loop

while (( $# ))
do
 mkdir -p /mnt/chroots/${NAME}/lowers/$#
 mount -o loop $1 /mnt/chroots/${NAME}/lowers/$#
 LOWER="/mnt/chroots/${NAME}/lowers/$#:${LOWER}"
 shift
done

mount -t overlay overlay -o \
lowerdir=${LOWER},\
upperdir=/mnt/chroots/${NAME}/tmpfs/upper,\
workdir=/mnt/chroots/${NAME}/tmpfs/work \
/mnt/chroots/${NAME}/root

And make it executable (chmod u+x /sbin/overlayimg). Another useful script removes the overlay /sbin/uoverlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1

umount /mnt/chroots/${NAME}/root
rmdir /mnt/chroots/${NAME}/root

umount /mnt/chroots/${NAME}/loop
rmdir /mnt/chroots/${NAME}/loop

umount /mnt/chroots/${NAME}/tmpfs

rm -rf /mnt/chroots/${NAME}/tmpfs


for d in $(ls /mnt/chroots/${NAME}/lowers)
do
  umount /mnt/chroots/${NAME}/lowers/$d
  rmdir /mnt/chroots/${NAME}/lowers/$d
done

rmdir /mnt/chroots/${NAME}/lowers
rmdir /mnt/chroots/${NAME}

And again make it executable (chmod u+x /sbin/uoverlayimg).

Making layers

Making layers is pretty easy, create an empty overlay (overlayimg builder).  Then chroot into the new root (arch-chroot /mnt/chroots/builder/root/ /bin/bash), and make any changes. e.g. pacman -S lynx.

Now the upper directory will contain the file differences from the root. So we can make them into a new image. Find the size of the new image (du -hs /mnt/chroots/builder/tmpfs/upper/), and create a file (dd if=/dev/zero of=/tmp/builder.img bs=1M count=<size>). Them make the file system ( mkfs.ext4 /tmp/builder.img) and mount it (mkdir /tmp/builder && mount /tmp/builder.img /tmp/builder). Them copy the new data over and unmount (cp -ra /mnt/chroots/builder/tmpfs/upper/* /tmp/builder && umount /tmp/builder).

Once created remove the overlay (uoverlayimg builder). Now create a new overlay with the added builder image (overlayimg newroot /tmp/builder.img) and chroot into the new new root (arch-chroot /mnt/chroots/newroot/root/ /bin/bash). Where you can now use the new root features e.g. lynx.

Arch Linux - A Memory Safe Server from USB

Introduction

I have been moving to Arch Linux for my USB server installs as it is a lightweight distro and has fewer distro specific tweeks that can cause problems when building these memory safe sever installs.

These are the basic steps I have worked out to build a light memory safe USB install using Arch Linux. With the root file system stored in a loop file so that the whole file system can be backed up and archived as needed.

Install system

The quickest way to get started is following the Arch Linux beginners guide. This will create a basic install. I perform this in a virtual machine using a virtual hard drive for the installation. To keep the install small I don't include base-devel in pacstrap.

Also to make the install easier to manage I install openssh ( pacman -S openssh ) and set it to run by default (systemctrl enable sshd).

For secure access I also create a new user (useradd -m <username>) and add them to the sudo group (groupadd sudo && gpasswd -a <username> sudo). Then enable sudo for the group (visudo). To prevent root login edit /etc/ssh/sshd_config to uncomment PermitRootLogin prohibit-password. 

To use a fat root file system on the USB you will also need the dos tools (pacman -S dosfstools). Also to manage the live system add the install scripts (pacman -S arch-install-scripts)

Custom Initram image

The magic for creating a bootable image is in the initram settings. This is done inside the chroot environment. Or inside the new Arch installation if you have rebooted.

Edit /etc/mkinitcpio.conf;

Add to the modules:
MODULES="vfat overlay loop zram"

The alter the hooks, base was moved to after udev and loop was added:
HOOKS="base udev block autodetect modconf filesystems keyboard fsck loop"

Then create the two new files, that will allow a new in-memory overlay root file system to be used:

/lib/initcpio/hooks/loop:

#!/usr/bin/ash

loop_mount_handler()
{
        mkdir /loop
        default_mount_handler /loop
        mkdir -p /tmpfs
        mkdir /lower
        mount ${loopfs:+-t ${loopfs}} -o loop /loop/${loop} /lower
        MEM=$(free -m | awk '/Mem/ {print $2}')
        MEM=$((MEM/2))
        MEM=${zmem:${MEM}K}
        TMP=$(zramctl -f -s ${MEM})
        mkfs.ext4 ${TMP}
        mount ${TMP} /tmpfs
        mkdir -p /tmpfs/upper
        mkdir -p /tmpfs/work
        mount -t overlay overlayfs -o lowerdir=/lower,upperdir=/tmpfs/upper,workdir=/tmpfs/work "$1"
}

run_hook() {
        mount_handler=loop_mount_handler
}

/lib/initcpio/install/loop:

#!/bin/bash

build() {
    add_binary zramctl
    add_binary mkfs.ext4

    add_runscript

}

help() {

    cat <<HELPEOF

This hook mounts a loop filesystem based based on the command line \

value of 'loop', the loop filesystem type may also be specified with \

'loopfs'. To specify the size of the zram drive use 'zmem' else half the total memory will be used.

HELPEOF

}

Then regenerate the initram (mkinitcpio -p linux).

Configure USB

To move all this onto a USB start by installing syslinux (pacman -S syslinux). Make sure the USB has a bootable primary partition with fdisk (fdisk <usb_device>), and then format the partition with fat (mkfs.fat -F32 <usb_partition>).

Make a mount mount point (mkdir /tmp/usb) and mount the device (mount <usb_partition> /tmp/usb). On the usb make the base directories (mkdir -p /tmp/usb/boot/syslinux), and install syslinux mbr (dd conv=notrunc bs=440 count=1 if=/usr/lib/syslinux/bios/mbr.bin of=<usb_device>) and install syslinux (syslinux -d /boot/syslinux -i <usb_partition>).

Copy over the new kernel and initram (cp /boot/vmlinuz-linux /boot/initramfs-linux.img /tmp/usb/boot), and create a new syslinux config file:

/tmp/usb/boot/syslinux/syslinux.cfg:

DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz-linux
 INITRD /boot/initramfs-linux.img
 APPEND root=UUID=<UUID> loop=/boot/root.img loopfs=ext4 rwopt=rw

Where <UUID> is the block id of the usb partition which you can find with (blkid).

Build root image

Then finally you need to make the root image file (dd if=/dev/zero of=/tmp/usb/boot/root.img bs=1G count=2), and format it (mkfs.ext4 /tmp/usb/boot/root.img).

If you are running this form the chroot then exit the chroot now. If you are in the new arch installation then you will need to reboot onto the install CD and mount the file system partition to /mnt and the usb partition to /tmp/usb.

To populate the new root image we need to mount it somewhere (mkdir /tmp/root && mount /tmp/usb/boot/root.img /tmp/root). Then copy over the root file system (cp -ra /mnt* /tmp/root/). Make sure all systems are unmounted (umount /tmp/root && umount /tmp/usb && umount /mnt).

Then exit and boot to the new USB.

Alter a live image

Now the live image you boot into is running with an in memory overlay. This means that all disk writes go into memory and are lost on a reboot. This prevents excessive writes to the USB and any permanent unwanted changes to the server file system. So how do you make updates?

To alter the live file system you can mount it with (mount /dev/loop0 /mnt) then alter it using the install tools chroot (arch-chroot /mnt /bin/bash). Then when you have finished with any update exit the chroot and unmount the image (exit && umount /mnt). Then reboot for the new settings.

To alter the usb root system mount it directly (mount <usb_partition> /mnt), make any changes such as backing up the root.img file or updating the syslinux.cfg settings, then unmount (umount /mnt).

Scripts for building a Minimal Ubuntu on a flash drive

File: rooter.sh

#!/bin/bash

# This script is expected to run on a clean ubuntu install that
# has the locale settings that are required in the final system.
# For best results and to avoid risking your main system run this
# script from with a virtual machine.
# To avoid typeing add an entry to visudo:
#    %sudo   ALL=(ALL)  NOPASSWD:ALL
# assuming you are in the sudo group.


# This script takes an optional name, that will be used for the
# image name and the host name for the new system.
NAME=${1:-root}
FILE=${NAME}.img
HOST=${NAME}
echo "Install to ${FILE}"
echo "With user ${USER}"
echo "With host name ${HOST}"

# Make a loop image with a size of 1G as the deboot install is 
# quite small.
dd if=/dev/zero of=${FILE} bs=1G count=1
sudo mkfs.ext4 ${FILE}

# Mount the loop device and copy over some files from the
# source system. This again assumes that the final system will
# use the same source list as the host system.
sudo mkdir -p /mnt/installer
sudo mount ${FILE} /mnt/installer
sudo debootstrap vivid /mnt/installer
sudo cp /etc/apt/sources.list /mnt/installer/etc/apt/
sudo cp /etc/localtime /mnt/installer/etc
sudo cp /etc/default/locale /mnt/installer/etc/default/locale

# This script will be run withtin the new system chroot and 
# is used to set up the new system. Note the hard coded locale settings
# and the assumption that overlayroot will be used to make a tmpfs
# filesystem overlay.
sudo cat << EOF > /mnt/installer/tmp/script.sh
echo "${HOST}" > /etc/hostname
locale-gen en_GB.UTF-8
dpkg-reconfigure tzdata
apt-get update
apt-get upgrade -y
apt-get install -y ssh overlayroot nano
sed -i -e 's/overlayroot=""/overlayroot="tmpfs"/' /etc/overlayroot.conf 
KERNEL=$(apt-cache search linux-image-[0-9].*generic | awk '{print $1}' | tail -n 1)
apt-get install -y ${KERNEL}
adduser ${USER}
gpasswd -a ${USER} sudo
apt-get autoclean
rm -rf /tmp/*

cat << XXX >> /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
XXX

EOF

# Bind special resources into the install dir, and run the setup 
# script as chroot.
sudo mount --bind /dev /mnt/installer/dev
sudo mount --bind /dev/pts /mnt/installer/dev/pts
sudo mount -t proc proc /mnt/installer/proc
sudo mount -t sysfs sys /mnt/installer/sys
sudo chroot /mnt/installer /bin/bash /tmp/script.sh

# Now try to unmount, though this may not work so well if the 
# bound mounts are in use.
sudo umount /mnt/installer/sys
sudo umount /mnt/installer/dev/pts
sudo umount /mnt/installer/dev
sudo umount /mnt/installer/proc
sudo umount /mnt/installer

echo "Done"

File: squash.sh

#!/bin/bash

# This script squashes a root image and prepares an initrd image
# in the host system for a bootable drive.

# The script takes the name of the image to squash.
IMG=$1

echo "Compress ${IMG}"

# Make sure the squashfs tools are available.
sudo apt-get install -y squashfs-tools 
if [ -z "$(grep squashfs /etc/initramfs-tools/modules)" ]
then
 sudo sh -c 'echo "squashfs" >> /etc/initramfs-tools/modules'
fi

# Update the local inirtd image.
sudo update-initramfs -ck all

# Mount the image
mkdir -p /tmp/root
sudo mount ${IMG} /tmp/root

# Squash it.
sudo mksquashfs /tmp/root ${IMG}.squashfs -noappend -always-use-fragments
sudo umount /tmp/root
rmdir /tmp/root

File: booter.sh

#!/bin/bash

# This script makes a bootable use from a root file system image.

# This script takes the usb device, an optional image name and an 
# optional file system type
DEV=$1
IMG=${2:-root.img}
FSTYPE=${3:-ext4}

echo "USB ${DEV}"

# Make sure the device is not mounted and format the first fat partition
sudo umount ${DEV}1
sudo mkfs.fat -F32 ${DEV}1
sudo parted ${DEV} set 1 boot on

# Set up the usb with syslinux
sudo apt-get install -y syslinux
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=${DEV}
sync
sudo mkdir -p /tmp/usb
sudo mount ${DEV}1 /tmp/usb
sudo mkdir -p /tmp/usb/boot/syslinux
sudo syslinux -i -d /boot/syslinux ${DEV}1

# Make sure overlay root is installed for the local initrd image.
sudo apt-get install -y overlayroot

# Copy the local kernel and initrd image to the usb.
KERNEL=$(ls /boot/vmlinuz-* | tail -n 1)
INITRD=$(ls /boot/initrd.img-* | tail -n 1)
sudo cp -v ${KERNEL} /tmp/usb/boot/vmlinuz
sudo cp -v ${INITRD} /tmp/usb/boot/initrd.img
sudo cp -v ${IMG} /tmp/usb/boot/root.img

# Set the boot options.
UUID=$(sudo blkid ${DEV}1 | awk '{print $2}' | sed -e 's/"//g')
sudo sh -c "cat << EOF > /tmp/usb/boot/syslinux/syslinux.cfg
DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz
 INITRD /boot/initrd.img
 APPEND root=${UUID} rw loop=/boot/root.img loopfstype=${FSTYPE}
EOF"

echo "Unmounting..."
sudo umount /tmp/usb
sudo rmdir /tmp/usb
echo "Done."

Xen and Guests all on a USB Drive

How to Fit a Bare Metal Hypervisor in Your Pocket...

Why?

A while ago I set up my server to run from a USB stick here. 

It was successful and I have been running and maintaining this system for over a year. During this time I have performed a major version upgrade on the base Ubuntu system, expanded the storage by a couple of TB and converted the file system from LVM to BTRFS, all while still running the OS from a simple 8G flash drive.

But now things are starting to change and get more interesting.....

I have been really pleased with the performance and value of my HP Proliant micro server, though I have had to change the PSU unit once. I was nervous that I didn't have a spare sever if it when down. Also it would have been a challenge to find a cost effective housing for my 4 HDD's, so when the price of the Gen8 Proliant servers dropped I decided to invest in a spare box, with the intention of swapping the HDD's and the OS on the USB flash. But it didn't work out as I had expected as the new Gen8 had different network cards and needed different kernel configurations to that of my old box. After a few days I got it sorted out and running again from the USB flash on the new server, but it got me thinking about variable hardware support for my USB install. As I also like working with virtual systems I have wanted to set up a server on top of Xen for a while, and running a system in a virtual environment provides a layer or hardware abstraction...so I decided to make a USB based Xen installation that could also mange guest systems.

And in the end it all came together...

Server Install

You can do this in a number of way to do this, but I am using a temporary hard drive in the target machine to build the base system. I have tried various approaches to this such as using virtual machines, but I have found that the virtualisation layer can give inconsistent results. The base system I am using is Ubuntu 15.04.

Follow the install process. I chose:

• Install using the whole drive without LVM, which avoids creating a boot partition
• No automatic updates
• Included OpenSSH server

When the system was installed I made sure it was all up to date with:

sudo apt-get update

sudo apt-get upgrade

Then I edit /etc/fstab to remove all swap and floppy drive mount entries, this is to prevent any circular mount, or other problems being found at boot time. After that I install overlayroot:

sudo apt-get install overlayroot

Then edit the last time of /etc/overlayroot.conf to this:

overlayroot="tmpfs"

There are a few other options explained in the conf file that can be changed but this will do for now. Then reboot.

sudo reboot

Now you should be back in the server as you installed it, but this time the root file system is read only and all writes are stored in memory. To see this create a file in your home directory, reboot and it is gone.

Now I attach my usb device and configure it. First partition the usb device and make a fat 32 partition with the boot flag set:

sudo fdisk <usb dev>

sudo mkfs.fat -F32 <usb dev>1

Note: if there are any problems creating these partitions, just use gparted.

Then install syslinux and set the new mbr:

sudo apt-get install syslinux

sudo syslinux <usb dev>1
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=<usb dev>
sudo parted <usb dev> set 1 boot on

mount the USB and move the files around a bit to tidy up, also make a new disk image for the root file system:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1
sudo dd if=/dev/zero of=/mnt/usb1/root.img bs=512M count=7
sudo mkfs.ext4 /mnt/usb1/root.img
sudo mkdir /mnt/usb2

sudo mount -o loop /mnt/usb1/root.img /mnt/usb2

sudo mkdir -p /mnt/usb1/boot/syslinux

sudo mv /mnt/usb1/ld* /mnt/usb1/boot/syslinux

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

Then get the block id of the usb partition, and create the /mnt/usb/boot/syslinux/syslinux.cfg file:

export $(sudo blkid -o export <usb dev>1)

echo """
DEFAULT linux

LABEL linux

KERNEL /boot/vmlinuz

APPEND root=UUID=${UUID} loop=/root.img loopfstype=ext4
INITRD /boot/initrd.img
""" > /tmp/syslinux.cfg

sudo mv /tmp/syslinux.cfg /mnt/usb1/boot/syslinux/syslinux.cfg

Clean and Copy

In the new booted system we can clean out some files. We can do this in a chroot writeable copy of the base file system after first taking a note of the second usb partition block id:

sudo overlayroot-chroot

Make a clean script in /usr/bin/rootclean:

#!/bin/bash

apt-get autoclean

find /var/log -type f -exec rm -v {} \;

find /var/spool -type f -exec rm -v {} \;

rm -rvf $(find ${ROOT}/var/cache/* | grep -v "apt$")
rm -rf /tmp/*

Then execute it:

chmod a+x /usr/bin/rootclean
/usr/bin/rootclean
exit

Now copy the root file to the second usb partition:

sudo cp -av /media/root-ro/* /mnt/usb2/

Then unmount and reboot:

sudo umount /mnt/usb1
sudo umount /mnt/usb2
sudo reboot

You now have a bootable USB!

Add Some Xen

Boot into the usb system and then enter the overlay chroot:

sudo overlayroot-chroot

Then install the xen tools and clean:

apt-get install xen-hypervisor-amd64
apt-get install bridge-utils

Update /etc/network/interfaces to something like this:

auto lo eth0 xenbr0
iface lo inet loopback

iface xenbr0 inet dhcp
  bridge_ports eth0


iface eth0 inet manual

Add these lines to /etc/sysctl.conf:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Then exit:

rootclean
exit

Then mount the usb and copy the new kernel images:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

sudo cp /boot/xen-<version>-amd64.gz /mnt/usb1/boot/

Then copy the multi boot module from syslinux:

sudo apt-get install syslinux

sudo cp /usr/lib/syslinux/modules/bios/mboot.c32 /mnt/usb1/boot/syslinux/
sudo cp /usr/lib/syslinux/modules/bios/libcom32.c32 /mnt/usb1/boot/syslinux/

Then update the /mnt/usb1/boot/syslinux/syslinux.cfg file. I have also limited the size of dom0 which also also restrict the overlayroot tmpfs size:

DEFAULT linux

LABEL linux

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.img loopfstype=ext4 --- /boot/initrd.img

Then restart:

sudo umount /mnt/usb1

sudo reboot

On reboot the xl tools will show the dom0:

sudo xl list

From here on you can set up Xen configs in the overlay chroot or run them from partitions on the usb. There are lots of options to try out.

Squash the Root

Often I also like to have a read only compressed backup of my server operating system. One way to do this is to make an squashed image.

To start you need the squashfs tools:

sudo overlayroot-chroot
sudo apt-get install squashfs-tools

edit /etc/initramfs-tools/modules:

# raid1
# sd_mod

squashfs

 Then build a new initrd image:

sudo update-initramfs -ck all
exit

Then mount the usb and create the image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1
mksquashfs /media/root-ro /mnt/usb1/root.squashfs -noappend -always-use-fragments

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

edit /mnt/usb1/boot/syslinux/syslinux.cfg, add a new boot entry and update the default label. Depending on how you use the server you could add a time out delay so a console user could select the boot option to use. Here the UUID is the id of the partition that contains the image:

DEFAULT linux-img

...

LABEL linux-img

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.squashfs loopfstype=squashfs --- /boot/initrd.img

sudo umount /mnt/usb1
sudo reboot

Running a Guest

To set up a quick guest I am going to create a new disk image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1

Then create and mount a disk image:

sudo dd if=/dev/zero of=/mnt/usb1/xen.img bs=1G count=5

Then get the boot images from your mirror https://launchpad.net/ubuntu/+archivemirrors:

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/initrd.gz

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/vmlinuz

Then create the conf file xen.conf:

name = "guest"

kernel = "/mnt/guest/vmlinuz"

ramdisk = "/mnt/guest/initrd.gz"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Then start it with:

sudo xl create -c xen.conf

When the installation is done, update xen.conf to:

name = "guest"

#kernel = "/mnt/guest/vmlinuz"

#ramdisk = "/mnt/guest/initrd.gz"

bootloader = "/usr/lib/xen-4.5/bin/pygrub"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Now you can run and connect to the new install with:

sudo xl destroy guest

sudo xl create -c xen.conf

That's all have fun and explore...

Ubuntu server on a USB drive

At home I have a small server that helps co-ordinate the various devices in our house. It is small and cheap but important for us, so I want it to be robust and cheap. I originally had it running Ubuntu server with LVM and Raid mirroring, plus rsnapshot to store historical changes. This ran very well for a few years, but the worry always came when it was time for updates or upgrades. I used an LVM root snapshot before and updates but if the system broke I would need to connect a monitor, keyboard and start putting it back together. But this wasn't always possible as I would often be administering the server remotely, and was a pain if I wanted the server up and running again fast.

Now at this point my friend (who has the same server box as me) told me he was running his server OS from a USB flash drive. It was useful for him as he could backup the usb drive for redundancy and if there was a problem he could just swap in a replica drive and carry on. I liked this idea, but wasn't sure about the amount of writing a flash drive could take on a daily basis or if it failed would it be easy to detect. So I looked for an USB distro that did not write to the USB flash. Well there are a few around and I tried some, but I was drawn back to the Ubuntu server distro as it is so easy to manage and is well supported. Looking again at running Ubuntu server on a USB flash I came up with this plan ,to install Ubuntu server on a USB flash with an overlay file system to avoid writing to USB.

I am sure there are other and better ways to do this but here is my workings:

This guide starts by walking through creating a USB bootable ubuntu server installation. Where the OS runs read only from the USB flash and all writes are to a memory file system. All commands in this guide are expected to be run as root. I found it easiest to take the risk and run everything in a shell using:

sudo -s

Bootable USB platform

The first stage involves creating the bootable USB platform which will host our OS file system. For this stage I am starting out with a fresh Ubuntu 14.04 server installation running within VirtualBox. The main reason is to avoid making any mistakes no my main system as I will be in a root shell, but you should be able to apply this process to any new ubuntu installation. Install the squashfs-tools. This will be needed later to build the initrd image with squashfs support.

sudo apt-get install squashfs-tools

Install the initram customisation tools. (These are already installed with the basic Ubunutu server installation)

sudo apt-get install initramfs-tools

Set the initram tools to include the squashfs module in the initrd image, by adding squashfs to the list of modules:

edit: /etc/initramfs-tools/modules

# Examples:

#

# raid1

# sd_mod

squashfs

You can regenerate the initrd image to include the squashfs ith:

sudo update-initramfs -ck all

though the initrd will be regenerated anyway during the next stage when we install overlayroot.

Overlayroot will provide the non-USB flash writable file system, which at this stage with be a memory file system:

sudo apt-get install overlayroot

Now it is time to assemble the USB boot platform. You will need a USB drive with a ext2 formatted partition. Most usb drives have a single fat32 partition which can be reformated with:

mkfs.ext2 /dev/sd<XY>

Where <XY> is the USB device number and partition.

Now mount the usb drive:

mkdir /tmp/usb

sudo mount /dev/sd<XY> /tmp/usb

The  block id is a way to identify a block device without the device point. This is useful for USB devices as the mount points can change when they are plugged in or moved between ports. To find the block id for the USB flash:

blkid

will give something like:

/dev/sdb1: UUID="511edef4-0183-40ce-90bb-037334d46ded" TYPE="ext2"

in this case for /dev/sdb1, which is the USB flash, the block id is '

511edef4-0183-40ce-90bb-037334d46ded

'.

To boot grub with a menu create this file on the mounted USB flash: /tmp/usb/boot/grub/grub.cfg

timeout=3

menuentry 'USB' {

     search --no-floppy --fs-uuid --set=root <USB_ID>

linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=tmpfs

initrd /boot/initrd.img

}

where <USB_ID> is replaced with the block id. This minimal config file will then detect the usb drive based on the block id and attempt to load the kernel and initram file system.

We need a copy of a working kernel and initrd image with squashfs and overlayfs support. If you have more than one version on your system you should a version with the updated initram above. Note how the names need  to match those in grub.cfg:

cp /boot/vmlinuz-* /tmp/usb/boot/vmlinuz

cp /boot/initrd.img-* /tmp/usb/boot/initrd.img

Finally install grub on the usb drive.

grub-install --root-directory=/tmp/usb /dev/sd<X>

Where <X> is the USB flash device letter.

Finally unmount the USB flash and test:

umount /tmp/usb

You can check that the USB flash starts the grub menu by booting a machine or using qemu, VirtualBox does not currently support booting from a USB device. While a USB device can be set a raw device in VirtualBox, it still being treated as a physical SATA or IDE device:

apt-get install qemu-kvm

once installed you can test the usb drive with:

kvm -m 1024 -usb /dev/sd<X>

If you can't run kvm (for example because you have VirtualBox running), you can use qemu-system-x86_64 or similar.

When  this runs you should see the grub menu and be able to edit the entry by  hitting 'e' before the timeout. The rest of the boot process should load the initram image and then fail to a minimal shell as there is not yet any file system present. It is worth knowing here that if you edit the grub menu entry and add 'debug' then the initram scripts will log debug information:

timeout=3

menuentry 'USB' {

     search --no-floppy --fs-uuid --set=root <USB_ID>

linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=tmpfs debug

initrd /boot/initrd.img

}

then in the minimal shell you can read the debug log with:

more /dev/.initramfs/initramfs.debug

At this stage the initramfs.debug file should report:

+ REASON=ALERT!  /host/boot/root.squashfs does not exist. Dropping to a shell! PS1=(initramfs)  /bin.sh -i

Building the compact OS

Now we need to build the squashed file system that we will boot into. This does not need to be the same installation used to create the bootable USB flash (which is useful as you may not want to mess with mounting and formatting partitions from within a working server in case you make a typo).

First make sure the support tools are installed:

apt-get install squashfs-tools overlayroot rsync

We need a working area to rebuild the system so create the directory:

mkdir -p /remake/tmp

If you do not have much space on the system you are going to create the file system image from, you may like to mount an extra drive to work in.
This can be really useful if you are working with an image of a file system in VirtualBox and you need additional space:

mount /dev/sd<AB> /remake/tmp

Where <AB> are the drive and partition of the extra storage area.

To prepare the file system create the following file: /remake/preroot.sh

#!bin/sh

ROOT="/remake/tmp/root"

mkdir -p ${ROOT}

rsync -vax --delete \

--exclude=/tmp/* \

--exclude=/boot/* \

--exclude=/remake/tmp/* \

/ ${ROOT}

The rsync -x option here restricts the copy to one file system which is useful to avoid pulling in data from attached disks. You may need to remove this option if you want to include other drives. You may also want to review the excluded paths from the new root image, for example you may want to exclude paths like /run if you have not included the -x option.

Then make the script executable:

chmod u+x /remake/preroot.sh

We need to make some changes to the file system copy that we don't want to update in the running system, so for this we create a patching script. This is also a good place to automate any other updates you need to the new system.

create: /remake/patchroot.sh

#!/bin/sh

ROOT="/remake/tmp/root"

PATCH="/remake/patch"

cp -ra ${PATCH}/* ${ROOT}

Then make the script executable:

chmod u+x /remake/patchroot.sh

Overlayroot will create an overlay for root '/' by reading /etc/fstab, but we want overlayroot to use the new loop squashfs from the USB, so copy the system fstab and remove or comment out the root mount point:

mkdir /remake/patch/etc

cp /etc/fstab /remake/patch/etc

In my simple build I have removed all mount points so /remake/patch/etc/fstab is an empty file.

Now the system root scripts are prepared, we can start scripting the new squash file system.

create: /remake/squash.sh

#!/bin/sh

ROOT="/remake/tmp/root"

IMG="/remake/tmp/img"

mkdir ${IMG}

mksquashfs ${ROOT} ${IMG}/root.squashfs -noappend -always-use-fragments

Then make the script executable:

chmod u+x /remake/squash.sh

Now these update scripts have been created we need to run them all:

/remake/preroot.sh

/remake/patchroot.sh

/remake/squash.sh

This will produce a root.squashfs in /remake/tmp/img, so we just need to copy that to the mounted USB flash:

mkdir /tmp/usb

sudo mount /dev/s<XY> /tmp/usb

sudo cp /remake/tmp/img/root.squashfs /tmp/usb/boot/

Now you should have a bootable USB flash you can test in a machine or qemu.

When  booted mount should show root as an overlayfs and two /media mounts /media/root-ro is the read only contents of the the squashfs and /media/root-rw/overlay is the in memory writable updates.

Options

So now you have a bootable system, here are some other things that you may like to try.

Remake from USB

When you have the USB installation running you may want to make some system updates, such as package installs that you want to make permanent. One way to do this is to create a new root.squashfs image after you have made any in-memory changes. In the live USB system the /remake directory and scripts should still present (unless you excluded them in the
build).

To  do the remake you will probably need to mount some more working space on /remake/tmp unless you have a lot of memory or are using a disk based overlay (below).

So start by mounting your working disk space:

mount /dev/sd<AB> /remake/tmp

Then just run:

/remake/preroot.sh

/remake/patchroot.sh

/remake/squash.sh

Then copy the new root.squashfs file over, after first mounting the USB flash:

mkdir /tmp/usb

mount /dev/s<XY> /tmp/usb

cp /remake/tmp/img/root.squashfs /tmp/usb/

Note: If you changed the kernel or initramfs then you will need to replace the ones on the USB drive.

When the system is powered down the USB flash may be removed and the old file(s) replaced with the new ones.

Semi writable overlay

The  above grub settings use a ram based overlay filesystem. This is useful for temporary or volatile systems, but some permanence can be achieved by using another device for the writable overlay. To use a physical device as the overlay filesystem update the boot settings to add the overlay device:

linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=/dev/sd<MN>

initrd /boot/initrd.img

Where <MN> are the writable overlay device and partition number.

Boot time config

It  may be too much effort to rebuild a new root.squashfs to change only a few settings. So boot time settings may be implemented for more flexability.

Create the file: /etc/initramfs-tools/scripts/init-bottom/usbinit

#!/bin/sh

PREREQ="overlayroot"
prereqs()
{
 echo "$PREREQ"
}

case $1 in
prereqs)
 prereqs
   exit 0
   ;;
esac

# Begin real processing below this line

if [ -e /host/init.sh ]

then

/host/init.sh

fi

and make it executable:

chmod a+x /etc/initramfs-tools/scripts/init-bottom/usbinit

then rebuild the initramfs:

update-initramfs -ck all

which will produce a new /boot/initrd.img.* file. This should be copied onto the USB flash:

mount /dev/sd<XY> /remake/usb

cp /boot/initrd.img.* /remake/usb/boot/initrd.img

umount /remake/usb

Now  on reboot if there is an executable script called init.sh in the USB root directory then it will be exexuted during system startup. It will be run in a busybox ash shell with access to the USB flash mounted at /host and the new overlay is mounted at /root/media/root-rw/overlay So to set the machine hostname at boot, on the USB flash:

create:hostname

usbhost

create:init.sh

#!/bin/sh

HOST=$(cat /host/hostname)

echo “Setting hostname ${HOST}”

echo ${HOST} > /root/media/root-rw/overlay/etc/hostname

and make init.sh executable:

chmod a+x init.sh

Grub install

Currently  grub-install does now work with an overlay file system, but to allow grub-install to work with our USB flash we can make a bit of a hack. Follow the above step to allow an init.sh script to be executed at boot time.

Edit: init.sh

#!/bin/sh

mkdir /root/media/root-rw/usbhost

mount -o bind /host /root/media/root-rw/usbhost

mount -o bind /host/boot /root/boot

mkdir -p /root/media/root-rw/overlay/usr/sbin

cp /root/media/root-ro/usr/sbin/grub-probe /root/media/root-rw/overlay/usr/sbin/grub-probe.orig

cp /host/grub-probe /root/media/root-rw/overlay/usr/sbin/grub-probe

This  will mount the USB onto /media/root-rw/usbhost in the live filesystem and the USB /boot directory to /boot in the live filesystem. It will then rename the grub-probe file (which causes the root overlayfs filesytem detection to fail) and replace it with a file from the USB flash. This script detects a probe for root '/' and redirects it to the mounted USB device.

Then on the USB flash create: grub-probe

#!/bin/sh

ARGS=$@

SUFF="media/root-rw/usbhost"

CMD="/usr/sbin/grub-probe.orig"

ROOT=$(echo $ARGS | grep -- --target=device | grep ' /$')

if [ ! -z "${ROOT}" ]

then

  ARGS=${ARGS}${SUFF}

fi

${CMD} ${ARGS}

and make it executable:

chmod a+x grub-probe

Now on boot a new /boot/grub/grub.cfg can be generated in the live usb system with:

update-grub

But if update-grub was run now, the new config file would not contain our original grub entry. To fix this add our original grub.cfg to the custom grub script before running update-grub:

cat /boot/grub/grub.cfg >> /etc/grub.d/40_custom

To make this custom grub entry permanent generate a new root.squashfs:

/remake/preroot.sh

/remake/patchroot.sh

/remake/squash.sh

and replace the root.squashfs on the USB flash.

Fstab updates

One of the features of overlayroot is that it will be default replace all mount points in the fstab with the overlayfs, and disable swap partitions. This can be a problem in a server where the main storage is assigned to fixed disks. To avoid this default behaviour, update the grub.cfg. file:

linux /boot/vmlinuz root=UUID=<USB_ID> ro loop=/boot/root.squashfs loopfstype=squashfs overlayroot=tmpfs:swap=1,recurse=0

The swap option allows fstab swap partitions and recurse prevents the moving of non-root mount points.

I hope this is helpful to someone.

Please let me know of any corrections or improvements.