Multi Overlay Chroot Server

Introduction

This project is an extension of my Arch Memory Safe Server From USB project. The aim of this project is to extend the memory safe server to host a range of applications in chroot environments. The purpose being to keep each service discreet to that it can be updated, replaced or moved independently of the other services and the host machine.

Mounting

As the new chroot environments will be stored on the USB partition, we need to make sure that it is mounted. To do that from the Arch usb install first mount the usb partition and chroot into it to make permanent changes (mount /dev/loop0 /mnt && arch-chroot /mnt /bin/bash). You can of course back up the root.img before making the changes (mkdir /tmp/usb && mount <usb_partition> /tmp/usb && cp /tmp/usb/boot/root.img /tmp/usb/boot/root.img.bak).

Then to mount the usb at boot time create the mount point (mkdir /mnt/usb), and add the usb block id to the fstab (blkid). /etc/fstab:

#
# /etc/fstab: static file system information
#
UUID="DCF4-6289" /mnt/usb vfat defaults 0 0

Build the overlay

Next we create a script that will build an overlay file system using images /sbin/overlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1
shift

mkdir -p /mnt/chroots/${NAME}/{loop,lowers,tmpfs,root}
mount -t tmpfs tmpfs /mnt/chroots/${NAME}/tmpfs
mkdir /mnt/chroots/${NAME}/tmpfs/{upper,work} 
mount /dev/loop0 /mnt/chroots/${NAME}/loop 

LOWER=/mnt/chroots/${NAME}/loop

while (( $# ))
do
 mkdir -p /mnt/chroots/${NAME}/lowers/$#
 mount -o loop $1 /mnt/chroots/${NAME}/lowers/$#
 LOWER="/mnt/chroots/${NAME}/lowers/$#:${LOWER}"
 shift
done

mount -t overlay overlay -o \
lowerdir=${LOWER},\
upperdir=/mnt/chroots/${NAME}/tmpfs/upper,\
workdir=/mnt/chroots/${NAME}/tmpfs/work \
/mnt/chroots/${NAME}/root

And make it executable (chmod u+x /sbin/overlayimg). Another useful script removes the overlay /sbin/uoverlayimg:

#!/bin/bash

if [ -z "$1" ]
then
 echo "usage $(basename $0) "
 exit 1
fi

NAME=$1

umount /mnt/chroots/${NAME}/root
rmdir /mnt/chroots/${NAME}/root

umount /mnt/chroots/${NAME}/loop
rmdir /mnt/chroots/${NAME}/loop

umount /mnt/chroots/${NAME}/tmpfs

rm -rf /mnt/chroots/${NAME}/tmpfs


for d in $(ls /mnt/chroots/${NAME}/lowers)
do
  umount /mnt/chroots/${NAME}/lowers/$d
  rmdir /mnt/chroots/${NAME}/lowers/$d
done

rmdir /mnt/chroots/${NAME}/lowers
rmdir /mnt/chroots/${NAME}

And again make it executable (chmod u+x /sbin/uoverlayimg).

Making layers

Making layers is pretty easy, create an empty overlay (overlayimg builder).  Then chroot into the new root (arch-chroot /mnt/chroots/builder/root/ /bin/bash), and make any changes. e.g. pacman -S lynx.

Now the upper directory will contain the file differences from the root. So we can make them into a new image. Find the size of the new image (du -hs /mnt/chroots/builder/tmpfs/upper/), and create a file (dd if=/dev/zero of=/tmp/builder.img bs=1M count=<size>). Them make the file system ( mkfs.ext4 /tmp/builder.img) and mount it (mkdir /tmp/builder && mount /tmp/builder.img /tmp/builder). Them copy the new data over and unmount (cp -ra /mnt/chroots/builder/tmpfs/upper/* /tmp/builder && umount /tmp/builder).

Once created remove the overlay (uoverlayimg builder). Now create a new overlay with the added builder image (overlayimg newroot /tmp/builder.img) and chroot into the new new root (arch-chroot /mnt/chroots/newroot/root/ /bin/bash). Where you can now use the new root features e.g. lynx.

Arch Linux - A Memory Safe Server from USB

Introduction

I have been moving to Arch Linux for my USB server installs as it is a lightweight distro and has fewer distro specific tweeks that can cause problems when building these memory safe sever installs.

These are the basic steps I have worked out to build a light memory safe USB install using Arch Linux. With the root file system stored in a loop file so that the whole file system can be backed up and archived as needed.

Install system

The quickest way to get started is following the Arch Linux beginners guide. This will create a basic install. I perform this in a virtual machine using a virtual hard drive for the installation. To keep the install small I don't include base-devel in pacstrap.

Also to make the install easier to manage I install openssh ( pacman -S openssh ) and set it to run by default (systemctrl enable sshd).

For secure access I also create a new user (useradd -m <username>) and add them to the sudo group (groupadd sudo && gpasswd -a <username> sudo). Then enable sudo for the group (visudo). To prevent root login edit /etc/ssh/sshd_config to uncomment PermitRootLogin prohibit-password. 

To use a fat root file system on the USB you will also need the dos tools (pacman -S dosfstools). Also to manage the live system add the install scripts (pacman -S arch-install-scripts)

Custom Initram image

The magic for creating a bootable image is in the initram settings. This is done inside the chroot environment. Or inside the new Arch installation if you have rebooted.

Edit /etc/mkinitcpio.conf;

Add to the modules:
MODULES="vfat overlay loop zram"

The alter the hooks, base was moved to after udev and loop was added:
HOOKS="base udev block autodetect modconf filesystems keyboard fsck loop"

Then create the two new files, that will allow a new in-memory overlay root file system to be used:

/lib/initcpio/hooks/loop:

#!/usr/bin/ash

loop_mount_handler()
{
        mkdir /loop
        default_mount_handler /loop
        mkdir -p /tmpfs
        mkdir /lower
        mount ${loopfs:+-t ${loopfs}} -o loop /loop/${loop} /lower
        MEM=$(free -m | awk '/Mem/ {print $2}')
        MEM=$((MEM/2))
        MEM=${zmem:${MEM}K}
        TMP=$(zramctl -f -s ${MEM})
        mkfs.ext4 ${TMP}
        mount ${TMP} /tmpfs
        mkdir -p /tmpfs/upper
        mkdir -p /tmpfs/work
        mount -t overlay overlayfs -o lowerdir=/lower,upperdir=/tmpfs/upper,workdir=/tmpfs/work "$1"
}

run_hook() {
        mount_handler=loop_mount_handler
}

/lib/initcpio/install/loop:

#!/bin/bash

build() {
    add_binary zramctl
    add_binary mkfs.ext4

    add_runscript

}

help() {

    cat <<HELPEOF

This hook mounts a loop filesystem based based on the command line \

value of 'loop', the loop filesystem type may also be specified with \

'loopfs'. To specify the size of the zram drive use 'zmem' else half the total memory will be used.

HELPEOF

}

Then regenerate the initram (mkinitcpio -p linux).

Configure USB

To move all this onto a USB start by installing syslinux (pacman -S syslinux). Make sure the USB has a bootable primary partition with fdisk (fdisk <usb_device>), and then format the partition with fat (mkfs.fat -F32 <usb_partition>).

Make a mount mount point (mkdir /tmp/usb) and mount the device (mount <usb_partition> /tmp/usb). On the usb make the base directories (mkdir -p /tmp/usb/boot/syslinux), and install syslinux mbr (dd conv=notrunc bs=440 count=1 if=/usr/lib/syslinux/bios/mbr.bin of=<usb_device>) and install syslinux (syslinux -d /boot/syslinux -i <usb_partition>).

Copy over the new kernel and initram (cp /boot/vmlinuz-linux /boot/initramfs-linux.img /tmp/usb/boot), and create a new syslinux config file:

/tmp/usb/boot/syslinux/syslinux.cfg:

DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz-linux
 INITRD /boot/initramfs-linux.img
 APPEND root=UUID=<UUID> loop=/boot/root.img loopfs=ext4 rwopt=rw

Where <UUID> is the block id of the usb partition which you can find with (blkid).

Build root image

Then finally you need to make the root image file (dd if=/dev/zero of=/tmp/usb/boot/root.img bs=1G count=2), and format it (mkfs.ext4 /tmp/usb/boot/root.img).

If you are running this form the chroot then exit the chroot now. If you are in the new arch installation then you will need to reboot onto the install CD and mount the file system partition to /mnt and the usb partition to /tmp/usb.

To populate the new root image we need to mount it somewhere (mkdir /tmp/root && mount /tmp/usb/boot/root.img /tmp/root). Then copy over the root file system (cp -ra /mnt* /tmp/root/). Make sure all systems are unmounted (umount /tmp/root && umount /tmp/usb && umount /mnt).

Then exit and boot to the new USB.

Alter a live image

Now the live image you boot into is running with an in memory overlay. This means that all disk writes go into memory and are lost on a reboot. This prevents excessive writes to the USB and any permanent unwanted changes to the server file system. So how do you make updates?

To alter the live file system you can mount it with (mount /dev/loop0 /mnt) then alter it using the install tools chroot (arch-chroot /mnt /bin/bash). Then when you have finished with any update exit the chroot and unmount the image (exit && umount /mnt). Then reboot for the new settings.

To alter the usb root system mount it directly (mount <usb_partition> /mnt), make any changes such as backing up the root.img file or updating the syslinux.cfg settings, then unmount (umount /mnt).

Scripts for building a Minimal Ubuntu on a flash drive

File: rooter.sh

#!/bin/bash

# This script is expected to run on a clean ubuntu install that
# has the locale settings that are required in the final system.
# For best results and to avoid risking your main system run this
# script from with a virtual machine.
# To avoid typeing add an entry to visudo:
#    %sudo   ALL=(ALL)  NOPASSWD:ALL
# assuming you are in the sudo group.


# This script takes an optional name, that will be used for the
# image name and the host name for the new system.
NAME=${1:-root}
FILE=${NAME}.img
HOST=${NAME}
echo "Install to ${FILE}"
echo "With user ${USER}"
echo "With host name ${HOST}"

# Make a loop image with a size of 1G as the deboot install is 
# quite small.
dd if=/dev/zero of=${FILE} bs=1G count=1
sudo mkfs.ext4 ${FILE}

# Mount the loop device and copy over some files from the
# source system. This again assumes that the final system will
# use the same source list as the host system.
sudo mkdir -p /mnt/installer
sudo mount ${FILE} /mnt/installer
sudo debootstrap vivid /mnt/installer
sudo cp /etc/apt/sources.list /mnt/installer/etc/apt/
sudo cp /etc/localtime /mnt/installer/etc
sudo cp /etc/default/locale /mnt/installer/etc/default/locale

# This script will be run withtin the new system chroot and 
# is used to set up the new system. Note the hard coded locale settings
# and the assumption that overlayroot will be used to make a tmpfs
# filesystem overlay.
sudo cat << EOF > /mnt/installer/tmp/script.sh
echo "${HOST}" > /etc/hostname
locale-gen en_GB.UTF-8
dpkg-reconfigure tzdata
apt-get update
apt-get upgrade -y
apt-get install -y ssh overlayroot nano
sed -i -e 's/overlayroot=""/overlayroot="tmpfs"/' /etc/overlayroot.conf 
KERNEL=$(apt-cache search linux-image-[0-9].*generic | awk '{print $1}' | tail -n 1)
apt-get install -y ${KERNEL}
adduser ${USER}
gpasswd -a ${USER} sudo
apt-get autoclean
rm -rf /tmp/*

cat << XXX >> /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
XXX

EOF

# Bind special resources into the install dir, and run the setup 
# script as chroot.
sudo mount --bind /dev /mnt/installer/dev
sudo mount --bind /dev/pts /mnt/installer/dev/pts
sudo mount -t proc proc /mnt/installer/proc
sudo mount -t sysfs sys /mnt/installer/sys
sudo chroot /mnt/installer /bin/bash /tmp/script.sh

# Now try to unmount, though this may not work so well if the 
# bound mounts are in use.
sudo umount /mnt/installer/sys
sudo umount /mnt/installer/dev/pts
sudo umount /mnt/installer/dev
sudo umount /mnt/installer/proc
sudo umount /mnt/installer

echo "Done"

File: squash.sh

#!/bin/bash

# This script squashes a root image and prepares an initrd image
# in the host system for a bootable drive.

# The script takes the name of the image to squash.
IMG=$1

echo "Compress ${IMG}"

# Make sure the squashfs tools are available.
sudo apt-get install -y squashfs-tools 
if [ -z "$(grep squashfs /etc/initramfs-tools/modules)" ]
then
 sudo sh -c 'echo "squashfs" >> /etc/initramfs-tools/modules'
fi

# Update the local inirtd image.
sudo update-initramfs -ck all

# Mount the image
mkdir -p /tmp/root
sudo mount ${IMG} /tmp/root

# Squash it.
sudo mksquashfs /tmp/root ${IMG}.squashfs -noappend -always-use-fragments
sudo umount /tmp/root
rmdir /tmp/root

File: booter.sh

#!/bin/bash

# This script makes a bootable use from a root file system image.

# This script takes the usb device, an optional image name and an 
# optional file system type
DEV=$1
IMG=${2:-root.img}
FSTYPE=${3:-ext4}

echo "USB ${DEV}"

# Make sure the device is not mounted and format the first fat partition
sudo umount ${DEV}1
sudo mkfs.fat -F32 ${DEV}1
sudo parted ${DEV} set 1 boot on

# Set up the usb with syslinux
sudo apt-get install -y syslinux
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=${DEV}
sync
sudo mkdir -p /tmp/usb
sudo mount ${DEV}1 /tmp/usb
sudo mkdir -p /tmp/usb/boot/syslinux
sudo syslinux -i -d /boot/syslinux ${DEV}1

# Make sure overlay root is installed for the local initrd image.
sudo apt-get install -y overlayroot

# Copy the local kernel and initrd image to the usb.
KERNEL=$(ls /boot/vmlinuz-* | tail -n 1)
INITRD=$(ls /boot/initrd.img-* | tail -n 1)
sudo cp -v ${KERNEL} /tmp/usb/boot/vmlinuz
sudo cp -v ${INITRD} /tmp/usb/boot/initrd.img
sudo cp -v ${IMG} /tmp/usb/boot/root.img

# Set the boot options.
UUID=$(sudo blkid ${DEV}1 | awk '{print $2}' | sed -e 's/"//g')
sudo sh -c "cat << EOF > /tmp/usb/boot/syslinux/syslinux.cfg
DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz
 INITRD /boot/initrd.img
 APPEND root=${UUID} rw loop=/boot/root.img loopfstype=${FSTYPE}
EOF"

echo "Unmounting..."
sudo umount /tmp/usb
sudo rmdir /tmp/usb
echo "Done."