Multi Overlay Chroot Server

Introduction

This project is an extension of my Arch Memory Safe Server From USB project. The aim of this project is to extend the memory safe server to host a range of applications in chroot environments. The purpose being to keep each service discreet to that it can be updated, replaced or moved independently of the other services and the host machine.

Mounting

As the new chroot environments will be stored on the USB partition, we need to make sure that it is mounted. To do that from the Arch usb install first mount the usb partition and chroot into it to make permanent changes (mount /dev/loop0 /mnt && arch-chroot /mnt /bin/bash). You can of course back up the root.img before making the changes (mkdir /tmp/usb && mount <usb_partition> /tmp/usb && cp /tmp/usb/boot/root.img /tmp/usb/boot/root.img.bak).

Then to mount the usb at boot time create the mount point (mkdir /mnt/usb), and add the usb block id to the fstab (blkid). /etc/fstab:

#
# /etc/fstab: static file system information
#
UUID="DCF4-6289" /mnt/usb vfat defaults 0 0

Build the overlay

Next we create a script that will build an overlay file system using images /sbin/overlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1
shift

mkdir -p /mnt/chroots/${NAME}/{loop,lowers,tmpfs,root}
mount -t tmpfs tmpfs /mnt/chroots/${NAME}/tmpfs
mkdir /mnt/chroots/${NAME}/tmpfs/{upper,work} 
mount /dev/loop0 /mnt/chroots/${NAME}/loop 

LOWER=/mnt/chroots/${NAME}/loop

while (( $# ))
do
 mkdir -p /mnt/chroots/${NAME}/lowers/$#
 mount -o loop $1 /mnt/chroots/${NAME}/lowers/$#
 LOWER="/mnt/chroots/${NAME}/lowers/$#:${LOWER}"
 shift
done

mount -t overlay overlay -o \
lowerdir=${LOWER},\
upperdir=/mnt/chroots/${NAME}/tmpfs/upper,\
workdir=/mnt/chroots/${NAME}/tmpfs/work \
/mnt/chroots/${NAME}/root

And make it executable (chmod u+x /sbin/overlayimg). Another useful script removes the overlay /sbin/uoverlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1

umount /mnt/chroots/${NAME}/root
rmdir /mnt/chroots/${NAME}/root

umount /mnt/chroots/${NAME}/loop
rmdir /mnt/chroots/${NAME}/loop

umount /mnt/chroots/${NAME}/tmpfs

rm -rf /mnt/chroots/${NAME}/tmpfs


for d in $(ls /mnt/chroots/${NAME}/lowers)
do
  umount /mnt/chroots/${NAME}/lowers/$d
  rmdir /mnt/chroots/${NAME}/lowers/$d
done

rmdir /mnt/chroots/${NAME}/lowers
rmdir /mnt/chroots/${NAME}

And again make it executable (chmod u+x /sbin/uoverlayimg).

Making layers

Making layers is pretty easy, create an empty overlay (overlayimg builder).  Then chroot into the new root (arch-chroot /mnt/chroots/builder/root/ /bin/bash), and make any changes. e.g. pacman -S lynx.

Now the upper directory will contain the file differences from the root. So we can make them into a new image. Find the size of the new image (du -hs /mnt/chroots/builder/tmpfs/upper/), and create a file (dd if=/dev/zero of=/tmp/builder.img bs=1M count=<size>). Them make the file system ( mkfs.ext4 /tmp/builder.img) and mount it (mkdir /tmp/builder && mount /tmp/builder.img /tmp/builder). Them copy the new data over and unmount (cp -ra /mnt/chroots/builder/tmpfs/upper/* /tmp/builder && umount /tmp/builder).

Once created remove the overlay (uoverlayimg builder). Now create a new overlay with the added builder image (overlayimg newroot /tmp/builder.img) and chroot into the new new root (arch-chroot /mnt/chroots/newroot/root/ /bin/bash). Where you can now use the new root features e.g. lynx.

Arch Linux - A Memory Safe Server from USB

Introduction

I have been moving to Arch Linux for my USB server installs as it is a lightweight distro and has fewer distro specific tweeks that can cause problems when building these memory safe sever installs.

These are the basic steps I have worked out to build a light memory safe USB install using Arch Linux. With the root file system stored in a loop file so that the whole file system can be backed up and archived as needed.

Install system

The quickest way to get started is following the Arch Linux beginners guide. This will create a basic install. I perform this in a virtual machine using a virtual hard drive for the installation. To keep the install small I don't include base-devel in pacstrap.

Also to make the install easier to manage I install openssh ( pacman -S openssh ) and set it to run by default (systemctrl enable sshd).

For secure access I also create a new user (useradd -m <username>) and add them to the sudo group (groupadd sudo && gpasswd -a <username> sudo). Then enable sudo for the group (visudo). To prevent root login edit /etc/ssh/sshd_config to uncomment PermitRootLogin prohibit-password. 

To use a fat root file system on the USB you will also need the dos tools (pacman -S dosfstools). Also to manage the live system add the install scripts (pacman -S arch-install-scripts)

Custom Initram image

The magic for creating a bootable image is in the initram settings. This is done inside the chroot environment. Or inside the new Arch installation if you have rebooted.

Edit /etc/mkinitcpio.conf;

Add to the modules:
MODULES="vfat overlay loop zram"

The alter the hooks, base was moved to after udev and loop was added:
HOOKS="base udev block autodetect modconf filesystems keyboard fsck loop"

Then create the two new files, that will allow a new in-memory overlay root file system to be used:

/lib/initcpio/hooks/loop:

#!/usr/bin/ash

loop_mount_handler()
{
        mkdir /loop
        default_mount_handler /loop
        mkdir -p /tmpfs
        mkdir /lower
        mount ${loopfs:+-t ${loopfs}} -o loop /loop/${loop} /lower
        MEM=$(free -m | awk '/Mem/ {print $2}')
        MEM=$((MEM/2))
        MEM=${zmem:${MEM}K}
        TMP=$(zramctl -f -s ${MEM})
        mkfs.ext4 ${TMP}
        mount ${TMP} /tmpfs
        mkdir -p /tmpfs/upper
        mkdir -p /tmpfs/work
        mount -t overlay overlayfs -o lowerdir=/lower,upperdir=/tmpfs/upper,workdir=/tmpfs/work "$1"
}

run_hook() {
        mount_handler=loop_mount_handler
}

/lib/initcpio/install/loop:

#!/bin/bash

build() {
    add_binary zramctl
    add_binary mkfs.ext4

    add_runscript

}

help() {

    cat <<HELPEOF

This hook mounts a loop filesystem based based on the command line \

value of 'loop', the loop filesystem type may also be specified with \

'loopfs'. To specify the size of the zram drive use 'zmem' else half the total memory will be used.

HELPEOF

}

Then regenerate the initram (mkinitcpio -p linux).

Configure USB

To move all this onto a USB start by installing syslinux (pacman -S syslinux). Make sure the USB has a bootable primary partition with fdisk (fdisk <usb_device>), and then format the partition with fat (mkfs.fat -F32 <usb_partition>).

Make a mount mount point (mkdir /tmp/usb) and mount the device (mount <usb_partition> /tmp/usb). On the usb make the base directories (mkdir -p /tmp/usb/boot/syslinux), and install syslinux mbr (dd conv=notrunc bs=440 count=1 if=/usr/lib/syslinux/bios/mbr.bin of=<usb_device>) and install syslinux (syslinux -d /boot/syslinux -i <usb_partition>).

Copy over the new kernel and initram (cp /boot/vmlinuz-linux /boot/initramfs-linux.img /tmp/usb/boot), and create a new syslinux config file:

/tmp/usb/boot/syslinux/syslinux.cfg:

DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz-linux
 INITRD /boot/initramfs-linux.img
 APPEND root=UUID=<UUID> loop=/boot/root.img loopfs=ext4 rwopt=rw

Where <UUID> is the block id of the usb partition which you can find with (blkid).

Build root image

Then finally you need to make the root image file (dd if=/dev/zero of=/tmp/usb/boot/root.img bs=1G count=2), and format it (mkfs.ext4 /tmp/usb/boot/root.img).

If you are running this form the chroot then exit the chroot now. If you are in the new arch installation then you will need to reboot onto the install CD and mount the file system partition to /mnt and the usb partition to /tmp/usb.

To populate the new root image we need to mount it somewhere (mkdir /tmp/root && mount /tmp/usb/boot/root.img /tmp/root). Then copy over the root file system (cp -ra /mnt* /tmp/root/). Make sure all systems are unmounted (umount /tmp/root && umount /tmp/usb && umount /mnt).

Then exit and boot to the new USB.

Alter a live image

Now the live image you boot into is running with an in memory overlay. This means that all disk writes go into memory and are lost on a reboot. This prevents excessive writes to the USB and any permanent unwanted changes to the server file system. So how do you make updates?

To alter the live file system you can mount it with (mount /dev/loop0 /mnt) then alter it using the install tools chroot (arch-chroot /mnt /bin/bash). Then when you have finished with any update exit the chroot and unmount the image (exit && umount /mnt). Then reboot for the new settings.

To alter the usb root system mount it directly (mount <usb_partition> /mnt), make any changes such as backing up the root.img file or updating the syslinux.cfg settings, then unmount (umount /mnt).

Scripts for building a Minimal Ubuntu on a flash drive

File: rooter.sh

#!/bin/bash

# This script is expected to run on a clean ubuntu install that
# has the locale settings that are required in the final system.
# For best results and to avoid risking your main system run this
# script from with a virtual machine.
# To avoid typeing add an entry to visudo:
#    %sudo   ALL=(ALL)  NOPASSWD:ALL
# assuming you are in the sudo group.


# This script takes an optional name, that will be used for the
# image name and the host name for the new system.
NAME=${1:-root}
FILE=${NAME}.img
HOST=${NAME}
echo "Install to ${FILE}"
echo "With user ${USER}"
echo "With host name ${HOST}"

# Make a loop image with a size of 1G as the deboot install is 
# quite small.
dd if=/dev/zero of=${FILE} bs=1G count=1
sudo mkfs.ext4 ${FILE}

# Mount the loop device and copy over some files from the
# source system. This again assumes that the final system will
# use the same source list as the host system.
sudo mkdir -p /mnt/installer
sudo mount ${FILE} /mnt/installer
sudo debootstrap vivid /mnt/installer
sudo cp /etc/apt/sources.list /mnt/installer/etc/apt/
sudo cp /etc/localtime /mnt/installer/etc
sudo cp /etc/default/locale /mnt/installer/etc/default/locale

# This script will be run withtin the new system chroot and 
# is used to set up the new system. Note the hard coded locale settings
# and the assumption that overlayroot will be used to make a tmpfs
# filesystem overlay.
sudo cat << EOF > /mnt/installer/tmp/script.sh
echo "${HOST}" > /etc/hostname
locale-gen en_GB.UTF-8
dpkg-reconfigure tzdata
apt-get update
apt-get upgrade -y
apt-get install -y ssh overlayroot nano
sed -i -e 's/overlayroot=""/overlayroot="tmpfs"/' /etc/overlayroot.conf 
KERNEL=$(apt-cache search linux-image-[0-9].*generic | awk '{print $1}' | tail -n 1)
apt-get install -y ${KERNEL}
adduser ${USER}
gpasswd -a ${USER} sudo
apt-get autoclean
rm -rf /tmp/*

cat << XXX >> /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
XXX

EOF

# Bind special resources into the install dir, and run the setup 
# script as chroot.
sudo mount --bind /dev /mnt/installer/dev
sudo mount --bind /dev/pts /mnt/installer/dev/pts
sudo mount -t proc proc /mnt/installer/proc
sudo mount -t sysfs sys /mnt/installer/sys
sudo chroot /mnt/installer /bin/bash /tmp/script.sh

# Now try to unmount, though this may not work so well if the 
# bound mounts are in use.
sudo umount /mnt/installer/sys
sudo umount /mnt/installer/dev/pts
sudo umount /mnt/installer/dev
sudo umount /mnt/installer/proc
sudo umount /mnt/installer

echo "Done"

File: squash.sh

#!/bin/bash

# This script squashes a root image and prepares an initrd image
# in the host system for a bootable drive.

# The script takes the name of the image to squash.
IMG=$1

echo "Compress ${IMG}"

# Make sure the squashfs tools are available.
sudo apt-get install -y squashfs-tools 
if [ -z "$(grep squashfs /etc/initramfs-tools/modules)" ]
then
 sudo sh -c 'echo "squashfs" >> /etc/initramfs-tools/modules'
fi

# Update the local inirtd image.
sudo update-initramfs -ck all

# Mount the image
mkdir -p /tmp/root
sudo mount ${IMG} /tmp/root

# Squash it.
sudo mksquashfs /tmp/root ${IMG}.squashfs -noappend -always-use-fragments
sudo umount /tmp/root
rmdir /tmp/root

File: booter.sh

#!/bin/bash

# This script makes a bootable use from a root file system image.

# This script takes the usb device, an optional image name and an 
# optional file system type
DEV=$1
IMG=${2:-root.img}
FSTYPE=${3:-ext4}

echo "USB ${DEV}"

# Make sure the device is not mounted and format the first fat partition
sudo umount ${DEV}1
sudo mkfs.fat -F32 ${DEV}1
sudo parted ${DEV} set 1 boot on

# Set up the usb with syslinux
sudo apt-get install -y syslinux
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=${DEV}
sync
sudo mkdir -p /tmp/usb
sudo mount ${DEV}1 /tmp/usb
sudo mkdir -p /tmp/usb/boot/syslinux
sudo syslinux -i -d /boot/syslinux ${DEV}1

# Make sure overlay root is installed for the local initrd image.
sudo apt-get install -y overlayroot

# Copy the local kernel and initrd image to the usb.
KERNEL=$(ls /boot/vmlinuz-* | tail -n 1)
INITRD=$(ls /boot/initrd.img-* | tail -n 1)
sudo cp -v ${KERNEL} /tmp/usb/boot/vmlinuz
sudo cp -v ${INITRD} /tmp/usb/boot/initrd.img
sudo cp -v ${IMG} /tmp/usb/boot/root.img

# Set the boot options.
UUID=$(sudo blkid ${DEV}1 | awk '{print $2}' | sed -e 's/"//g')
sudo sh -c "cat << EOF > /tmp/usb/boot/syslinux/syslinux.cfg
DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz
 INITRD /boot/initrd.img
 APPEND root=${UUID} rw loop=/boot/root.img loopfstype=${FSTYPE}
EOF"

echo "Unmounting..."
sudo umount /tmp/usb
sudo rmdir /tmp/usb
echo "Done."

Xen and Guests all on a USB Drive

How to Fit a Bare Metal Hypervisor in Your Pocket...

Why?

A while ago I set up my server to run from a USB stick here. 

It was successful and I have been running and maintaining this system for over a year. During this time I have performed a major version upgrade on the base Ubuntu system, expanded the storage by a couple of TB and converted the file system from LVM to BTRFS, all while still running the OS from a simple 8G flash drive.

But now things are starting to change and get more interesting.....

I have been really pleased with the performance and value of my HP Proliant micro server, though I have had to change the PSU unit once. I was nervous that I didn't have a spare sever if it when down. Also it would have been a challenge to find a cost effective housing for my 4 HDD's, so when the price of the Gen8 Proliant servers dropped I decided to invest in a spare box, with the intention of swapping the HDD's and the OS on the USB flash. But it didn't work out as I had expected as the new Gen8 had different network cards and needed different kernel configurations to that of my old box. After a few days I got it sorted out and running again from the USB flash on the new server, but it got me thinking about variable hardware support for my USB install. As I also like working with virtual systems I have wanted to set up a server on top of Xen for a while, and running a system in a virtual environment provides a layer or hardware abstraction...so I decided to make a USB based Xen installation that could also mange guest systems.

And in the end it all came together...

Server Install

You can do this in a number of way to do this, but I am using a temporary hard drive in the target machine to build the base system. I have tried various approaches to this such as using virtual machines, but I have found that the virtualisation layer can give inconsistent results. The base system I am using is Ubuntu 15.04.

Follow the install process. I chose:

• Install using the whole drive without LVM, which avoids creating a boot partition
• No automatic updates
• Included OpenSSH server

When the system was installed I made sure it was all up to date with:

sudo apt-get update

sudo apt-get upgrade

Then I edit /etc/fstab to remove all swap and floppy drive mount entries, this is to prevent any circular mount, or other problems being found at boot time. After that I install overlayroot:

sudo apt-get install overlayroot

Then edit the last time of /etc/overlayroot.conf to this:

overlayroot="tmpfs"

There are a few other options explained in the conf file that can be changed but this will do for now. Then reboot.

sudo reboot

Now you should be back in the server as you installed it, but this time the root file system is read only and all writes are stored in memory. To see this create a file in your home directory, reboot and it is gone.

Now I attach my usb device and configure it. First partition the usb device and make a fat 32 partition with the boot flag set:

sudo fdisk <usb dev>

sudo mkfs.fat -F32 <usb dev>1

Note: if there are any problems creating these partitions, just use gparted.

Then install syslinux and set the new mbr:

sudo apt-get install syslinux

sudo syslinux <usb dev>1
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=<usb dev>
sudo parted <usb dev> set 1 boot on

mount the USB and move the files around a bit to tidy up, also make a new disk image for the root file system:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1
sudo dd if=/dev/zero of=/mnt/usb1/root.img bs=512M count=7
sudo mkfs.ext4 /mnt/usb1/root.img
sudo mkdir /mnt/usb2

sudo mount -o loop /mnt/usb1/root.img /mnt/usb2

sudo mkdir -p /mnt/usb1/boot/syslinux

sudo mv /mnt/usb1/ld* /mnt/usb1/boot/syslinux

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

Then get the block id of the usb partition, and create the /mnt/usb/boot/syslinux/syslinux.cfg file:

export $(sudo blkid -o export <usb dev>1)

echo """
DEFAULT linux

LABEL linux

KERNEL /boot/vmlinuz

APPEND root=UUID=${UUID} loop=/root.img loopfstype=ext4
INITRD /boot/initrd.img
""" > /tmp/syslinux.cfg

sudo mv /tmp/syslinux.cfg /mnt/usb1/boot/syslinux/syslinux.cfg

Clean and Copy

In the new booted system we can clean out some files. We can do this in a chroot writeable copy of the base file system after first taking a note of the second usb partition block id:

sudo overlayroot-chroot

Make a clean script in /usr/bin/rootclean:

#!/bin/bash

apt-get autoclean

find /var/log -type f -exec rm -v {} \;

find /var/spool -type f -exec rm -v {} \;

rm -rvf $(find ${ROOT}/var/cache/* | grep -v "apt$")
rm -rf /tmp/*

Then execute it:

chmod a+x /usr/bin/rootclean
/usr/bin/rootclean
exit

Now copy the root file to the second usb partition:

sudo cp -av /media/root-ro/* /mnt/usb2/

Then unmount and reboot:

sudo umount /mnt/usb1
sudo umount /mnt/usb2
sudo reboot

You now have a bootable USB!

Add Some Xen

Boot into the usb system and then enter the overlay chroot:

sudo overlayroot-chroot

Then install the xen tools and clean:

apt-get install xen-hypervisor-amd64
apt-get install bridge-utils

Update /etc/network/interfaces to something like this:

auto lo eth0 xenbr0
iface lo inet loopback

iface xenbr0 inet dhcp
  bridge_ports eth0


iface eth0 inet manual

Add these lines to /etc/sysctl.conf:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Then exit:

rootclean
exit

Then mount the usb and copy the new kernel images:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

sudo cp /boot/xen-<version>-amd64.gz /mnt/usb1/boot/

Then copy the multi boot module from syslinux:

sudo apt-get install syslinux

sudo cp /usr/lib/syslinux/modules/bios/mboot.c32 /mnt/usb1/boot/syslinux/
sudo cp /usr/lib/syslinux/modules/bios/libcom32.c32 /mnt/usb1/boot/syslinux/

Then update the /mnt/usb1/boot/syslinux/syslinux.cfg file. I have also limited the size of dom0 which also also restrict the overlayroot tmpfs size:

DEFAULT linux

LABEL linux

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.img loopfstype=ext4 --- /boot/initrd.img

Then restart:

sudo umount /mnt/usb1

sudo reboot

On reboot the xl tools will show the dom0:

sudo xl list

From here on you can set up Xen configs in the overlay chroot or run them from partitions on the usb. There are lots of options to try out.

Squash the Root

Often I also like to have a read only compressed backup of my server operating system. One way to do this is to make an squashed image.

To start you need the squashfs tools:

sudo overlayroot-chroot
sudo apt-get install squashfs-tools

edit /etc/initramfs-tools/modules:

# raid1
# sd_mod

squashfs

 Then build a new initrd image:

sudo update-initramfs -ck all
exit

Then mount the usb and create the image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1
mksquashfs /media/root-ro /mnt/usb1/root.squashfs -noappend -always-use-fragments

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

edit /mnt/usb1/boot/syslinux/syslinux.cfg, add a new boot entry and update the default label. Depending on how you use the server you could add a time out delay so a console user could select the boot option to use. Here the UUID is the id of the partition that contains the image:

DEFAULT linux-img

...

LABEL linux-img

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.squashfs loopfstype=squashfs --- /boot/initrd.img

sudo umount /mnt/usb1
sudo reboot

Running a Guest

To set up a quick guest I am going to create a new disk image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1

Then create and mount a disk image:

sudo dd if=/dev/zero of=/mnt/usb1/xen.img bs=1G count=5

Then get the boot images from your mirror https://launchpad.net/ubuntu/+archivemirrors:

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/initrd.gz

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/vmlinuz

Then create the conf file xen.conf:

name = "guest"

kernel = "/mnt/guest/vmlinuz"

ramdisk = "/mnt/guest/initrd.gz"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Then start it with:

sudo xl create -c xen.conf

When the installation is done, update xen.conf to:

name = "guest"

#kernel = "/mnt/guest/vmlinuz"

#ramdisk = "/mnt/guest/initrd.gz"

bootloader = "/usr/lib/xen-4.5/bin/pygrub"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Now you can run and connect to the new install with:

sudo xl destroy guest

sudo xl create -c xen.conf

That's all have fun and explore...