Arch Linux - A Memory Safe Server from USB

Introduction

I have been moving to Arch Linux for my USB server installs as it is a lightweight distro and has fewer distro specific tweeks that can cause problems when building these memory safe sever installs.

These are the basic steps I have worked out to build a light memory safe USB install using Arch Linux. With the root file system stored in a loop file so that the whole file system can be backed up and archived as needed.

Install system

The quickest way to get started is following the Arch Linux beginners guide. This will create a basic install. I perform this in a virtual machine using a virtual hard drive for the installation. To keep the install small I don't include base-devel in pacstrap.

Also to make the install easier to manage I install openssh ( pacman -S openssh ) and set it to run by default (systemctrl enable sshd).

For secure access I also create a new user (useradd -m <username>) and add them to the sudo group (groupadd sudo && gpasswd -a <username> sudo). Then enable sudo for the group (visudo). To prevent root login edit /etc/ssh/sshd_config to uncomment PermitRootLogin prohibit-password. 

To use a fat root file system on the USB you will also need the dos tools (pacman -S dosfstools). Also to manage the live system add the install scripts (pacman -S arch-install-scripts)

Custom Initram image

The magic for creating a bootable image is in the initram settings. This is done inside the chroot environment. Or inside the new Arch installation if you have rebooted.

Edit /etc/mkinitcpio.conf;

Add to the modules:
MODULES="vfat overlay loop zram"

The alter the hooks, base was moved to after udev and loop was added:
HOOKS="base udev block autodetect modconf filesystems keyboard fsck loop"

Then create the two new files, that will allow a new in-memory overlay root file system to be used:

/lib/initcpio/hooks/loop:

#!/usr/bin/ash

loop_mount_handler()
{
        mkdir /loop
        default_mount_handler /loop
        mkdir -p /tmpfs
        mkdir /lower
        mount ${loopfs:+-t ${loopfs}} -o loop /loop/${loop} /lower
        MEM=$(free -m | awk '/Mem/ {print $2}')
        MEM=$((MEM/2))
        MEM=${zmem:${MEM}K}
        TMP=$(zramctl -f -s ${MEM})
        mkfs.ext4 ${TMP}
        mount ${TMP} /tmpfs
        mkdir -p /tmpfs/upper
        mkdir -p /tmpfs/work
        mount -t overlay overlayfs -o lowerdir=/lower,upperdir=/tmpfs/upper,workdir=/tmpfs/work "$1"
}

run_hook() {
        mount_handler=loop_mount_handler
}

/lib/initcpio/install/loop:

#!/bin/bash

build() {
    add_binary zramctl
    add_binary mkfs.ext4

    add_runscript

}

help() {

    cat <<HELPEOF

This hook mounts a loop filesystem based based on the command line \

value of 'loop', the loop filesystem type may also be specified with \

'loopfs'. To specify the size of the zram drive use 'zmem' else half the total memory will be used.

HELPEOF

}

Then regenerate the initram (mkinitcpio -p linux).

Configure USB

To move all this onto a USB start by installing syslinux (pacman -S syslinux). Make sure the USB has a bootable primary partition with fdisk (fdisk <usb_device>), and then format the partition with fat (mkfs.fat -F32 <usb_partition>).

Make a mount mount point (mkdir /tmp/usb) and mount the device (mount <usb_partition> /tmp/usb). On the usb make the base directories (mkdir -p /tmp/usb/boot/syslinux), and install syslinux mbr (dd conv=notrunc bs=440 count=1 if=/usr/lib/syslinux/bios/mbr.bin of=<usb_device>) and install syslinux (syslinux -d /boot/syslinux -i <usb_partition>).

Copy over the new kernel and initram (cp /boot/vmlinuz-linux /boot/initramfs-linux.img /tmp/usb/boot), and create a new syslinux config file:

/tmp/usb/boot/syslinux/syslinux.cfg:

DEFAULT linux

LABEL linux
 KERNEL /boot/vmlinuz-linux
 INITRD /boot/initramfs-linux.img
 APPEND root=UUID=<UUID> loop=/boot/root.img loopfs=ext4 rwopt=rw

Where <UUID> is the block id of the usb partition which you can find with (blkid).

Build root image

Then finally you need to make the root image file (dd if=/dev/zero of=/tmp/usb/boot/root.img bs=1G count=2), and format it (mkfs.ext4 /tmp/usb/boot/root.img).

If you are running this form the chroot then exit the chroot now. If you are in the new arch installation then you will need to reboot onto the install CD and mount the file system partition to /mnt and the usb partition to /tmp/usb.

To populate the new root image we need to mount it somewhere (mkdir /tmp/root && mount /tmp/usb/boot/root.img /tmp/root). Then copy over the root file system (cp -ra /mnt* /tmp/root/). Make sure all systems are unmounted (umount /tmp/root && umount /tmp/usb && umount /mnt).

Then exit and boot to the new USB.

Alter a live image

Now the live image you boot into is running with an in memory overlay. This means that all disk writes go into memory and are lost on a reboot. This prevents excessive writes to the USB and any permanent unwanted changes to the server file system. So how do you make updates?

To alter the live file system you can mount it with (mount /dev/loop0 /mnt) then alter it using the install tools chroot (arch-chroot /mnt /bin/bash). Then when you have finished with any update exit the chroot and unmount the image (exit && umount /mnt). Then reboot for the new settings.

To alter the usb root system mount it directly (mount <usb_partition> /mnt), make any changes such as backing up the root.img file or updating the syslinux.cfg settings, then unmount (umount /mnt).