Multi Overlay Chroot Server

Introduction

This project is an extension of my Arch Memory Safe Server From USB project. The aim of this project is to extend the memory safe server to host a range of applications in chroot environments. The purpose being to keep each service discreet to that it can be updated, replaced or moved independently of the other services and the host machine.

Mounting

As the new chroot environments will be stored on the USB partition, we need to make sure that it is mounted. To do that from the Arch usb install first mount the usb partition and chroot into it to make permanent changes (mount /dev/loop0 /mnt && arch-chroot /mnt /bin/bash). You can of course back up the root.img before making the changes (mkdir /tmp/usb && mount <usb_partition> /tmp/usb && cp /tmp/usb/boot/root.img /tmp/usb/boot/root.img.bak).

Then to mount the usb at boot time create the mount point (mkdir /mnt/usb), and add the usb block id to the fstab (blkid). /etc/fstab:

#
# /etc/fstab: static file system information
#
UUID="DCF4-6289" /mnt/usb vfat defaults 0 0

Build the overlay

Next we create a script that will build an overlay file system using images /sbin/overlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1
shift

mkdir -p /mnt/chroots/${NAME}/{loop,lowers,tmpfs,root}
mount -t tmpfs tmpfs /mnt/chroots/${NAME}/tmpfs
mkdir /mnt/chroots/${NAME}/tmpfs/{upper,work} 
mount /dev/loop0 /mnt/chroots/${NAME}/loop 

LOWER=/mnt/chroots/${NAME}/loop

while (( $# ))
do
 mkdir -p /mnt/chroots/${NAME}/lowers/$#
 mount -o loop $1 /mnt/chroots/${NAME}/lowers/$#
 LOWER="/mnt/chroots/${NAME}/lowers/$#:${LOWER}"
 shift
done

mount -t overlay overlay -o \
lowerdir=${LOWER},\
upperdir=/mnt/chroots/${NAME}/tmpfs/upper,\
workdir=/mnt/chroots/${NAME}/tmpfs/work \
/mnt/chroots/${NAME}/root

And make it executable (chmod u+x /sbin/overlayimg). Another useful script removes the overlay /sbin/uoverlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1

umount /mnt/chroots/${NAME}/root
rmdir /mnt/chroots/${NAME}/root

umount /mnt/chroots/${NAME}/loop
rmdir /mnt/chroots/${NAME}/loop

umount /mnt/chroots/${NAME}/tmpfs

rm -rf /mnt/chroots/${NAME}/tmpfs


for d in $(ls /mnt/chroots/${NAME}/lowers)
do
  umount /mnt/chroots/${NAME}/lowers/$d
  rmdir /mnt/chroots/${NAME}/lowers/$d
done

rmdir /mnt/chroots/${NAME}/lowers
rmdir /mnt/chroots/${NAME}

And again make it executable (chmod u+x /sbin/uoverlayimg).

Making layers

Making layers is pretty easy, create an empty overlay (overlayimg builder).  Then chroot into the new root (arch-chroot /mnt/chroots/builder/root/ /bin/bash), and make any changes. e.g. pacman -S lynx.

Now the upper directory will contain the file differences from the root. So we can make them into a new image. Find the size of the new image (du -hs /mnt/chroots/builder/tmpfs/upper/), and create a file (dd if=/dev/zero of=/tmp/builder.img bs=1M count=<size>). Them make the file system ( mkfs.ext4 /tmp/builder.img) and mount it (mkdir /tmp/builder && mount /tmp/builder.img /tmp/builder). Them copy the new data over and unmount (cp -ra /mnt/chroots/builder/tmpfs/upper/* /tmp/builder && umount /tmp/builder).

Once created remove the overlay (uoverlayimg builder). Now create a new overlay with the added builder image (overlayimg newroot /tmp/builder.img) and chroot into the new new root (arch-chroot /mnt/chroots/newroot/root/ /bin/bash). Where you can now use the new root features e.g. lynx.