home server based on Ubuntu 22.04 with ZFS

Introduction

I am running a home server since many years. Somewhere around 2018 I deciced to switch to a new server based on XUbuntu 18.04 using ZFS as filesystem. At that time I started with a combination of several howtos I found in the internet. That was a lot of work and I have no chance to list all of the sources as each of them only solved one step of the whole story.

After switching all of my desktops to XUbuntu 20.04 on ZFS I decided to do the same with my server. So I first had to check if all things I had to find out to get my requirements fullfilled still work on XUbuntu 20.04. And they did.

But time went by and meanwhile Ubuntu 22.04 is available. So again I wanted to get everything on the same basis and tried the installation on the server with this LTS version. Sadly I struggled over some things:

After reading several problem reports on the zsys daemon I decided not to install zsys - so the nice automatic rollback feature is gone ...

This site Don't use ZSYS explains why I decided not to use zsys anymore.
I don't like to have known bugs on the server which may lead to a complete removal of my user dataset ...

Additionally my boot disc changed.
Instead of using an eSATA configuration I put a NVME drive into a USB3 housing which supports the trim feature and connect this to one of the USB3 ports.

Preconditions

I don't want to use new hardware, but I have available:

I personally prefer XUbuntu, so I also need a bootable XUbuntu 22.04 USB stick.

Requirements

I have a list of things my server has to fullfill:

Known issues

Boot medium creation

I propose to use Ventoy. It's available for Linux and Windows.
This page explains the usage of ventoy under linux.
Usage under Windows is explained on the website.
The configuration of the stick content is identical.

Download ventoy

Download the tool here:
Ventoy download from Github
You'll get a file "ventoy-x.y.z-linux.tar.gz" with x, y, and z digits for the version.

Hint:
I faced an issue when trying to boot xubuntu 22.04.1 with Ventoy version 1.0.81 in UEFI mode!
Others had that issue with Ventoy version 1.0.78. I tried 1.0.79 and it works.

Optional: check the SHA256 against the value given on the website.
sha256sum ventoy-x.y.z-linux.tar.gz
and compare the value.
don't forget to use the real filename

Extract the file with the command
tar -xaf ventoy-x.y.z-linux.tar.gz
don't forget to use the real filename

Create bootable USB stick

cd into the extracted directory
cd ventoy-x.y.z
don't forget to use the real directory name

Insert an USB stick with enough space for the ubuntu iso file plus a file for persistent data.
I recommend to use a stick with at least 8GB.

run ventroy with the command
sudo sh Ventoy2Disk.sh -I /dev/XXX -g
Replace XXX by the device name of your USB stick.
The parameter -g creates a gpt partition scheme on the stick.
This stick can be booted in oldstyle BIOS mode or in UEFI mode.

Your computer will remount the stick and show one partition named
Ventoy

Copy your ubuntu iso file on this USB stick.
Now the stick is bootable, but we need the possibility to modify files on the stick persistent.

Persistent data area creation

ventoy contains a plugin for persistent data creation on bootable USB sticks.

In general that's a good idea, but I faced problems with firefox in the lifelinux when activating the persistant data storage.
Nevertheless I explain how this can be activated.

To prepare this execute the command
sudo sh CreatePersistentImg.sh -s 1024 -t ext4 -l casper-rw

This creates a file persistence.dat in your actual directory.
Copy the file on the stick.

Now we have to tell ventoy to use this persistence file for booting ubuntu.
To do so create a directory ventoy on the USB stick.

Create a file ventoy.json in this directory.
Content of this file has to be: 

{
	"persistence": [
		{
			"image": "/xubuntu-22.04.1-desktop-amd64.iso",
			"backend": "/persistence.dat",
			"autosel": 1
		}
	]
}

You can adapt the file names to your needs. It's also possible to rename persistence.dat, in that case use the new filename in ventoy.json, too.

Partition script modification

As mentioned above the default partitioning script of the ubuntu installation medium creates a swap partition of 2G. If the system shall do suspend to disk the swap partition is used to store the complete RAM content. Knowing this leads to the conclusion that 2G is - in most case - much to small.

As solution I first do a modification on the USB stick. The stick creation process explained above leads to a persistent storage in that boot system. So I boot the live stick and change the required file. Since then this modified file is used for partitioning.

After booting your future server system from the USB stick with "Try without installation" patch the file /usr/share/ubiquity/zsys-setup with root rights using this patch file: zsys-setup_22.04.patch

to apply the patch open a terminal and type 

wget https://andiweiss.github.io/projects/serverinstall/zsys-setup_22.04.patch
cd /usr/share/ubiquity
sudo patch -p1 < /home/xubuntu/zsys-setup_22.04.patch

This patch modifies the zfs initialization script in a way that the swap partition is as large as the physical ram of the machine.

After applying the patch the stick is prepared for installation. You can start the installation either with a new boot or without rebooting by using the install ubuntu icon.

If you regulary work in Linux it's easy to have a look on that file on another linux machine. After saving the file in the life system shut down the life system and connect the USB stick to your working linux system. In a default environment your system will mount the stick under /media/<user>/ventoy

create a directory for temporary mounting of the persitence file 

mkdir pers

Now mount the persistence file 

sudo mount /media/<user>/Ventoy/persistence.dat pers
In the case that you have chosen a different filename use that one.

Now you have access to your modified zsys-setup. You find it in pers/upper/usr/share/ubiquity/zsys-setup

Partitioning issue

I tried hard to get the installation doing all the things for having a separate data tank. But it always failed on grub installation. That's the reason why this howto just changes the swap partition size and does the additional pool handling after base installation but before starting the installed system the first time.

Installation process

I recommend doing the installation out of the life system, not the install xubuntu possibility.

Reason for that is to be able to add the data device before booting into the new installed system. The actual installer does not give the possibility to add an additional zfs pool with different features like mirroring or raid. The actual installer just puts everything on one hard disk / ssd.

Do the installation with only the future boot disk connected!

Do the installation in the regular way. We take care for moving the user data later.

Just keep care for one Window:

In the Window "Installation type" you have to chose "Advanced Features ...". In the now opened window select "Erase disk and use ZFS". Then select the device you want to boot from.

Now let the installer do its work. After that you'll have an ubuntu system on one hard disk using ZFS wherever it is possible. There's only one partition with another file system. The EFI partition has to be formatted as VFAT.

After the installation process has been finished the system asks for a reboot. I recommend not to do that directly but instead prepare the HDDs you want to use for the data pool.

Initialisation of the data pool

After the completion of the basic installation process I hotplug the harddisk or harddisks for the server data pool while still running the life system used for installation. After the disks have been detected I use a commandline shell to create the pool and the required data sets.

First step: import the already existing pools: 

sudo zpool import -R /target rpool
sudo zpool import -R /target bpool

Second step: create the data tank. For pool creation I use nearly the same parameters as used for rpool creation: 

sudo zpool create -f \
	-o ashift=12 \
	-o autotrim=on \
	-O compression=lz4 \
	-O acltype=posixacl \
	-O xattr=sa \
	-O relatime=on \
	-O normalization=formD \
	-O mountpoint=none \
	-O canmount=off \
	-O dnodesize=auto \
	-O sync=disabled \
	-R /tank0 \
	tank0 /dev/disk/by-id/<your_id_here>
Checking this against the creation of rpool you'll see only two differences: If the data pool shall have mirroring, raid or other fancy modes just hotplug the required disks and adapt the parameters to your needs.

Please keep in mind:
DO NOT use /dev/sd* for the pool creation!

Check the entries in /dev/disk/by-id/ for the drives you want to use and use these links instead. I personally use the wwn-* entries.

By handing over the whole device zpool will first create a gpt partition table, add two partitions - one with nearly the whole content plus a small one containing some MB additional space. This is done to be able to combine drives with small differences in the amount of sectors to one pool. The large partition is used for zfs afterwards.

Moving data

In earlier versions of this howto I moved the fresh installed data using zfs send ... | zfs receive ...
I changed this to creation of self defined datasets on the tank plus data copy with complete file system features.

Reason for this is again the buggy zsys. On Ubuntu 20.04 that system worked - as far as I know - pretty nice.

So now copying the user data is a combination of data set creation and data copy. First the required datasets have to be created and then the data has to be copied.

I want to move these directories on the tank:

Dataset creation

We start with the directory /root.

Next is the home directory of the initial user. Having in mind that there will be more than one user in the future we have to create a dataset for /home and then one dataset for each user as /home/<user>

Now create the dataset for the initial user and copy the data

Caution!
You have to keep in mind that without zsys there will be NO automatic dataset creation on creating a new user.
My recommendation is to first create the users data set including owner and mode settings and then create the user.
If the user shall be created with the standard user template all files out of /etc/skel have to be copied into the new users home directory.

Moving /srv is done in the same way than /root.

Moving the folders inside /var is a bit more complicate.
Here we first have to create a dataset /var whithout any mount feature.
Then we can crate the datasets in the same way as /home/user.

Dataset destruction

As now all future user and server data is located on the new tank the original datasets for these have to be destroyed.

Ubuntus installation process uses an id in the dataset name.
We read this id:
export MyId=`zfs list | grep ROOT/ubuntu_.*/target\$ | sed -n 's|[^_]*_\(\w*\).*|\1|g;p'`

now we destroy all datasets we copied on the tank:

Now we have to introduce the new pool to the system as bpool and rpool are.

sudo zpool set cachefile= tank0
sudo touch /target/etc/zfs/zfs-list.cache/tank0
sudo zfs set sync=standard tank0

Having this done tank0 is ready for usage, so now we export all pools in the opposite order as we imported / created them: 

sudo zpool export tank0
sudo zpool export bpool
sudo zpool export rpool

First boot of the fresh system

The system has to be rebooted. You can do that fastest with sudo reboot
You will be requested to remove the installation medium and hit enter.

After doing so your system will not boot completly.
Up to now it doesn't know that it has to import the new created data sets. But it requires these sets.

I found two different behaviors:

In both cases the system ist not yet working.

in the case that you get into the emergency mode:

Hit enter again to reach a command line.
Then type 

zpool import tank0
exit
Now your system boots first time into the fresh installation.

In the case you get the login screen:

DO NOT log in.
Instead type <ctrl>-<alt>-<F2>
Now you are in a textual login. Here you can log in.

Then type 

sudo zpool import tank0
sudo reboot
Now your system should execute the reboot and you can log in.

System configuration

The initial system installation is done now. There will be updates available - these can either be installed first or last. The result will be the same.

ssh server

Now the system has a state where I want to get remote access. Therefore my next step is the installation of the ssh server. 

sudo apt install openssh-server

In the case you want to replace an old server it may make sense to overtake the ssh keys from the old machine.
To do so copy the files /etc/ssh/*key* from the old machine to the new one.

Small configuration issues

One thing I really like is the possibility to use pageup / pagedown for searching the history.
This can be enabled in /etc/inputrc
Just search for the lines and remove the hash.

As last little config issue I disable the bluetooth service.
It usually crashes during boot and I don't have a bluetooth device connected to the server. 

sudo systemctl disable bluetooth

Configuration of suspend and hibernate

Now the basics are done. We have a system booting from ZFS and the desktop is clean.

Next step to do is the configuration of suspend to RAM and suspend to disk.

Ubuntu 22.04 has all required things installed to do suspend.
To test it use sudo systemctl suspend and the system should got in suspend to ram.

To get hibernate working we have to modify the kernel command line and modify the initramfs.

Now you can check hibernation.
execute sudo systemctl hibernate
and your system should switch off.

During next boot the kernel should report that it's recovering from your swap partition.
The screen should be locked and after logging in the systen should have the same state as before hibernation.

To have this as a sensefull feature you have to setup your BIOS in the way that after power loss the computer starts up.
In most cases the default for this setting is "keep it off".

Configuration of Wake On Lan

As already explained in the prerequesites the Atheros AR8171 in principle supports Wake On Lan, but the related part of the driver has been taken out of the kernel.
This driver is build as module. Its name is alx.

First thing to do is to install ethtool, git and dkms: 

sudo apt install ethtool git dkms

Now we need to know which interface we want to configure: 

ip link
We identify the interface, in my case this is enp2s0

Next step is to check the supported wake on lan modes and the actually configured ones: 

sudo ethtool enp2s0 | grep Wake-on
On my server hardware this report is empty.
The original kernel driver doesn't support Wake On Lan.

This is sad, because this means: No Wake On Lan supported ...

But: one can still use an old version of a driver to analyse the code and take the parts required for wol and patch them into the current driver.
I did some analysis on the kernel driver source and found out that there is no big change from kernel 5.15 to kernel 6.03. So I created a dkms package to compile an alx driver with wol feature on each new installed kernel.

This dkms package can be found here:
dkms package for alx with wake on lan
Best way to use it is to clone it and rund the installation script: 

git clone https://github.com/AndiWeiss/alx-wol.git
cd alx-wol
sudo ./install_alx-wol.sh

This script does the complete installation of the dkms and also replaces the original module of the running system by the one supporting wol. To check that use the command 

sudo ethtool enp2s0 | grep Wake-on
again.

The new result is: 

Supports Wake-on: pg
Wake-on: pg

So now Wake On Lan is activated on

I don't like phy activity waking up my system, so I have to use ethtool to change that behavior. Changing it in the commandline will be lost after reboot, therefore I add a task in systemd.

To create that service create the file /etc/systemd/system/wol.service with root rights. This file has to contain: 

[Unit]
Description=Configure Wake-up on LAN

[Service]
Type=oneshot
ExecStart=/sbin/ethtool -s enp2s0 wol g

[Install]
WantedBy=basic.target
Then enable the service with 
sudo systemctl enable wol.service

After a reboot ethtool reports 

Supports Wake-on: pg
Wake-on: g
This is what we want to get.

Testing the feature can be done with another linux system. I recommend to use wakeonlan which has to be installed with 

sudo apt install wakeonlan
on another linux machine in the same network.

The previous used command ip link does not only print the interface name. It also prints the mac address. The mac address is the part behind link/ether, containing six hexadecimal values with 2 digits each, the values separated by colons.

To try if wake on lan is functional put your server to sleep e.g. with sudo systemctl suspend.
Then, on the other machine, use 

wakeonlan 11:22:33:44:55:66
to wake it up again. Take care to use the mac address of your server.
Another thing you need to know is that the magic packet used for waking up a system is not routed over wifi, so you have to use a computer connected to the lan.

exporting nfs shares

Now we have a system providing the basic system features. Now the server featurs have to be activated. I prefer haveing nfs shares, so I don't take cifs shared into account here.

First thing to prepare nfs shares is the installtion of the nfs server: 

sudo apt install nfs-kernel-server

I decided to use the zfs mechanisms for exporting nfs shares. Therefore exporting the filesystems is done with zfs properties, not in /etc/exports.

I'm going to show the export with my folder containing the mp3 collection. This is located in /srv/music

First I have to create that dataset: 

sudo zfs create tank0/srv/music
By creating this dataset there's automatically a directory /srv/music available. Take care to set the access (user, group, read, write, execute, ...) according your needs.

Now we simply tell zfs to export this filesystem: 

sudo zfs set sharenfs=on tank0/srv/music
After doing so this share can be mounted on another linux machine: 
sudo mount server:/srv/music music

Automatic shutdown

Now the basic system is complete. User can be added and their home directory will be created on the tank device. Suspend to ram and disk is working. Wake on LAN is also full functional. So the next thing is to shutdown the system automatically.

Actually I see two things which have to keep the system alive:

But before shutting down the system it shall be checked if there are updates. If yes these updates shall be installed. And if - after this update - the system requires a reboot this shall be done, too.

Detection of nfs mounts

To detect how many nfs mounts are actually in use I use the program ss. This is included in the ubuntu package iproute2, which is not installed as default.
So we have to install it: 

sudo apt install iproute2

To get a list of active nfs connections I use 

ss -n --inet
I use the -n to get numeric output. Otherwise there has to be a functional nameserver in your network. Additionally we need the ip address of the client lateron.

This command does not only print the nfs connections, it prints all active tcp and udp connections. So we have to filer for nfs.

Taking a look into /etc/services we find the ports used for nfs. This is 2049. With this I use 

ss -n --inet | grep ':2049\W'
to get only the lines with active nfs connections.

During my tries I found an issue for generic usage:
If the system using the share doesn't unmount the share it can happen that ss detects the link active for a long time. So we also have to check if the system with the actual link is still alive.

For the correct handling of this we need a script. Here I don't quote the script, you'll find it where the cron job is explained. I'll just explain how that works.

As long as the system is alive we expect that it can be ping'ed. This depends if there is a firewall installed which blocks the ping, on my system this is not the case. So I can use ping.

To be able to ping the system we need its ip address. To get this we use the output of the found line of ss and kill everything but the ip address: 

ss -n --inet | grep ':2049\W' | sed -n 's|.*]:2049[^[]*\[\([^]]*\).*$|\1|g;p'

Detection of active users

Getting the information if users are logged in is easy: 

users
prints all users who are logged in.

Checking for updates

Easiest thing is not checking for updates but just doing the updates. To do so we tell the system to

Reboot if required

The update process creates a file /var/run/reboot-required if a reboot is required.
If this file is detected the server will - instead of go to suspend - do a reboot.

Wake up for maintenance

In my case the server is often not used for a long time. To avoid having a too long tim without updates the system shall wake up at least once a week and do maintenance.

To get that there is a tool called rtcwakeup which uses the rtc alarm to start the system at a defined time. My decission is to start the system at the same day in one week at 4 o'clock in the morning.

This is done in the script before shutting down the system with the command 

rtcwake -m no -t `date --date='7 days 04:00' +%s`
This sets the alarm in the rtc which then will start up the system.

Putting it together

Now we know how to get the information if we have to shut down. These things have to be used for putting it all together. We need to know if a user is actually logged in and if a share is actually in use.
If this is not the case for a certain time the system shall shut down.

As this is time based we create a cron job for that. In the cron job, once per minute, we collect the information. Then we store the information when we first have no keep alive status anymore. If this doesn't change for the defined time we do a s2both and the system gets suspended.

I provide a script for the automatic shutdown:
autosuspend.sh
Save this script with root rights as /bin/autosuspend.sh and don't forget to sudo chmod +x /bin/autosuspend.sh to make it executable. 

wget https://AndiWeiss.github.io/projects/serverinstall/autosuspend.sh
chmod a+x autosuspend.sh
sudo chown root:root autosuspend.sh
sudo mv autosuspend.sh /bin
After that we create a cronjob calling this script every minute. Therefore open your /etc/crontab with root rights and add the line 
* * * * * root /bin/autosuspend.sh
With this the script is executed each minute and then checks if the system shall stay active or not.

To check the autosuspend activities a log file /var/log/autosuspend.log is created.

Crash recovery

The best server is senseless if there is no possibility to do crash recovery.

I see the following things which have to be easy and fast recoverable:

Recovery of a crashed boot system

I decided to define the way of setting up a fresh system in the case that the boot system crashes. I completely separated the server data and user data from the boot system so I only need to be sure that there is an easy and fast possibility to get a clean system working with the still existing tank0.

This method is rather simple:

Recovery of a crashed tank0 hard disc

up to now not collected. Follow the standard zfs recovery documentation.