Xen and Guests all on a USB Drive

How to Fit a Bare Metal Hypervisor in Your Pocket...

Why?

A while ago I set up my server to run from a USB stick here. 

It was successful and I have been running and maintaining this system for over a year. During this time I have performed a major version upgrade on the base Ubuntu system, expanded the storage by a couple of TB and converted the file system from LVM to BTRFS, all while still running the OS from a simple 8G flash drive.

But now things are starting to change and get more interesting.....

I have been really pleased with the performance and value of my HP Proliant micro server, though I have had to change the PSU unit once. I was nervous that I didn't have a spare sever if it when down. Also it would have been a challenge to find a cost effective housing for my 4 HDD's, so when the price of the Gen8 Proliant servers dropped I decided to invest in a spare box, with the intention of swapping the HDD's and the OS on the USB flash. But it didn't work out as I had expected as the new Gen8 had different network cards and needed different kernel configurations to that of my old box. After a few days I got it sorted out and running again from the USB flash on the new server, but it got me thinking about variable hardware support for my USB install. As I also like working with virtual systems I have wanted to set up a server on top of Xen for a while, and running a system in a virtual environment provides a layer or hardware abstraction...so I decided to make a USB based Xen installation that could also mange guest systems.

And in the end it all came together...

Server Install

You can do this in a number of way to do this, but I am using a temporary hard drive in the target machine to build the base system. I have tried various approaches to this such as using virtual machines, but I have found that the virtualisation layer can give inconsistent results. The base system I am using is Ubuntu 15.04.

Follow the install process. I chose:

• Install using the whole drive without LVM, which avoids creating a boot partition
• No automatic updates
• Included OpenSSH server

When the system was installed I made sure it was all up to date with:

sudo apt-get update

sudo apt-get upgrade

Then I edit /etc/fstab to remove all swap and floppy drive mount entries, this is to prevent any circular mount, or other problems being found at boot time. After that I install overlayroot:

sudo apt-get install overlayroot

Then edit the last time of /etc/overlayroot.conf to this:

overlayroot="tmpfs"

There are a few other options explained in the conf file that can be changed but this will do for now. Then reboot.

sudo reboot

Now you should be back in the server as you installed it, but this time the root file system is read only and all writes are stored in memory. To see this create a file in your home directory, reboot and it is gone.

Now I attach my usb device and configure it. First partition the usb device and make a fat 32 partition with the boot flag set:

sudo fdisk <usb dev>

sudo mkfs.fat -F32 <usb dev>1

Note: if there are any problems creating these partitions, just use gparted.

Then install syslinux and set the new mbr:

sudo apt-get install syslinux

sudo syslinux <usb dev>1
sudo dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=<usb dev>
sudo parted <usb dev> set 1 boot on

mount the USB and move the files around a bit to tidy up, also make a new disk image for the root file system:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1
sudo dd if=/dev/zero of=/mnt/usb1/root.img bs=512M count=7
sudo mkfs.ext4 /mnt/usb1/root.img
sudo mkdir /mnt/usb2

sudo mount -o loop /mnt/usb1/root.img /mnt/usb2

sudo mkdir -p /mnt/usb1/boot/syslinux

sudo mv /mnt/usb1/ld* /mnt/usb1/boot/syslinux

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

Then get the block id of the usb partition, and create the /mnt/usb/boot/syslinux/syslinux.cfg file:

export $(sudo blkid -o export <usb dev>1)

echo """
DEFAULT linux

LABEL linux

KERNEL /boot/vmlinuz

APPEND root=UUID=${UUID} loop=/root.img loopfstype=ext4
INITRD /boot/initrd.img
""" > /tmp/syslinux.cfg

sudo mv /tmp/syslinux.cfg /mnt/usb1/boot/syslinux/syslinux.cfg

Clean and Copy

In the new booted system we can clean out some files. We can do this in a chroot writeable copy of the base file system after first taking a note of the second usb partition block id:

sudo overlayroot-chroot

Make a clean script in /usr/bin/rootclean:

#!/bin/bash

apt-get autoclean

find /var/log -type f -exec rm -v {} \;

find /var/spool -type f -exec rm -v {} \;

rm -rvf $(find ${ROOT}/var/cache/* | grep -v "apt$")
rm -rf /tmp/*

Then execute it:

chmod a+x /usr/bin/rootclean
/usr/bin/rootclean
exit

Now copy the root file to the second usb partition:

sudo cp -av /media/root-ro/* /mnt/usb2/

Then unmount and reboot:

sudo umount /mnt/usb1
sudo umount /mnt/usb2
sudo reboot

You now have a bootable USB!

Add Some Xen

Boot into the usb system and then enter the overlay chroot:

sudo overlayroot-chroot

Then install the xen tools and clean:

apt-get install xen-hypervisor-amd64
apt-get install bridge-utils

Update /etc/network/interfaces to something like this:

auto lo eth0 xenbr0
iface lo inet loopback

iface xenbr0 inet dhcp
  bridge_ports eth0


iface eth0 inet manual

Add these lines to /etc/sysctl.conf:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Then exit:

rootclean
exit

Then mount the usb and copy the new kernel images:

sudo mkdir /mnt/usb1
sudo mount <usb dev>1 /mnt/usb1

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

sudo cp /boot/xen-<version>-amd64.gz /mnt/usb1/boot/

Then copy the multi boot module from syslinux:

sudo apt-get install syslinux

sudo cp /usr/lib/syslinux/modules/bios/mboot.c32 /mnt/usb1/boot/syslinux/
sudo cp /usr/lib/syslinux/modules/bios/libcom32.c32 /mnt/usb1/boot/syslinux/

Then update the /mnt/usb1/boot/syslinux/syslinux.cfg file. I have also limited the size of dom0 which also also restrict the overlayroot tmpfs size:

DEFAULT linux

LABEL linux

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.img loopfstype=ext4 --- /boot/initrd.img

Then restart:

sudo umount /mnt/usb1

sudo reboot

On reboot the xl tools will show the dom0:

sudo xl list

From here on you can set up Xen configs in the overlay chroot or run them from partitions on the usb. There are lots of options to try out.

Squash the Root

Often I also like to have a read only compressed backup of my server operating system. One way to do this is to make an squashed image.

To start you need the squashfs tools:

sudo overlayroot-chroot
sudo apt-get install squashfs-tools

edit /etc/initramfs-tools/modules:

# raid1
# sd_mod

squashfs

 Then build a new initrd image:

sudo update-initramfs -ck all
exit

Then mount the usb and create the image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1
mksquashfs /media/root-ro /mnt/usb1/root.squashfs -noappend -always-use-fragments

Now copy the kernel and initrd image to the USB:

sudo cp /boot/vmlinuz-*-generic /mnt/usb1/boot/vmlinuz

sudo cp /boot/initrd.img-*-generic /mnt/usb1/boot/initrd.img

edit /mnt/usb1/boot/syslinux/syslinux.cfg, add a new boot entry and update the default label. Depending on how you use the server you could add a time out delay so a console user could select the boot option to use. Here the UUID is the id of the partition that contains the image:

DEFAULT linux-img

...

LABEL linux-img

  KERNEL mboot.c32

  APPEND /boot/xen-<version>.gz dom0_mem=max:1024M --- /boot/vmlinuz root=UUID=<UUID> loop=/root.squashfs loopfstype=squashfs --- /boot/initrd.img

sudo umount /mnt/usb1
sudo reboot

Running a Guest

To set up a quick guest I am going to create a new disk image:

sudo mkdir /mnt/usb1
sudo mount <usb device>1 /mnt/usb1

Then create and mount a disk image:

sudo dd if=/dev/zero of=/mnt/usb1/xen.img bs=1G count=5

Then get the boot images from your mirror https://launchpad.net/ubuntu/+archivemirrors:

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/initrd.gz

wget http://<mirror>/ubuntu/dists/vivid/main/installer-amd64/current/images/netboot/xen/vmlinuz

Then create the conf file xen.conf:

name = "guest"

kernel = "/mnt/guest/vmlinuz"

ramdisk = "/mnt/guest/initrd.gz"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Then start it with:

sudo xl create -c xen.conf

When the installation is done, update xen.conf to:

name = "guest"

#kernel = "/mnt/guest/vmlinuz"

#ramdisk = "/mnt/guest/initrd.gz"

bootloader = "/usr/lib/xen-4.5/bin/pygrub"

memory = 1024

vcpus = 1

vif = [ 'bridge:xenbr0' ]

disk = [ '/mnt/usb1/xen.img,raw,xvda,rw' ]

Now you can run and connect to the new install with:

sudo xl destroy guest

sudo xl create -c xen.conf

That's all have fun and explore...