ZFS file system on Raspberry Pi

FISH

I do a good bit of hardware integration with the web, with manufacturing equipment, with embedded systems and with big data set, or that can sustain multiple failures. Not necessarily all at once, but typically, people expect FISH from me :)

FISH is Fully Integrated Software and Hardware (btw, as a side note, the internal project at Sun to create appliances based on ZFS was known as FISHWorks). The Raspberry Pi is a cool piece of hardware, but I typically need stuff that is only (or mostly) found on Solaris and derived OSes, such as ZFS. I've been using ZFS for many years now, since the first public release on Solaris Nevada. ZFS scales and give you data integrity. And it can run on the largest systems known to man.

It scales

For example, I'm listening right now to ZFS Day's live video stream and hearing a talk about ZFS on the Sequoia supercomputer, which is the fastest supercomputer out there. They are using it as a native port, not using FUSE.

What is ZFS? 

Wikipedia: "ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes, support for high storage capacities, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. ZFS is implemented as open-source software, licensed under the Common Development and Distribution License (CDDL)."

From Supercomputers to $35 computers

So, ZFS scales at the highest level obviously. Well, it also scales down: I've been using a bit ZFS on the Raspberry Pi using FUSE, until I can get a Solaris derived OS ( such as illumos, smartos, openindiana, opensolaris etc) on the Raspberry Pi. That way, at least I have ZFS. Still missing zones, smf and dtrace, but it is a start.

Now just a reminder, the Pi only has 256MB total ram, and a BCM arm processor. So first thing first, we need to give as much ram to the OS as possible, and reduce the video buffer size:

I'm using a 240MB split on that Raspberry Pi since it is running only in text mode at the console, and I remote to it using ssh -X.

If you use the composite out you might want to use the 224MB split and definitely 192 or 128 using HDMI, but then at that point, you are chocking ZFS. That's 128 for OS and ZFS and whatever apps you are running...

Fully loaded

Altough Raspbian comes with a good amount of stuff preloaded, it was not intended to be used with FUSE out of the box, and ZFS was probably never on the radar screen of anybody. So let's start with adding the FUSE stuff and the libraries and tools we will need to build ZFS. This is the shortlist:

fdion@raspberrypi ~/zfs $ sudo apt-get install fuse-utils libfuse-dev libfuse2
fdion@raspberrypi ~/zfs $ sudo apt-get install libaio-dev libattr1-dev attr
fdion@raspberrypi ~/zfs $ sudo apt-get install git scons

If you build it...

So we have the prerequisites. Let's get the code, compile it and install the tools:

fdion@raspberrypi ~ $ mkdir zfs
fdion@raspberrypi ~ $ cd zfs
fdion@raspberrypi ~/zfs $ git clone https://bitbucket.org/cli/zfs-fuse-arm.git
fdion@raspberrypi ~/zfs $ cd zfs-fuse-arm/
fdion@raspberrypi ~/zfs/zfs-fuse-arm $ cd src
fdion@raspberrypi ~/zfs/zfs-fuse-arm/src $ scons
[a lot of stuff will scroll by]
fdion@raspberrypi ~/zfs/zfs-fuse-arm/src $ sudo scons install
[again, more stuff will scroll by]

Wow, it compiled (scons). And installed (sudo scons install). It's a good thing we are using the zfs-fuse-arm version, because the mainline wont go very far on the compile.

A demonstration, if you please? 

Well of course! Let's start the zfs-fuse daemon and create two virtual disks. I'm creating two 100M disks here using dd/ (this is on a slow SD card, rated 10MB/s). You could also use an actual /dev (like a pair of USB keys):

fdion@raspberrypi ~/zfs/zfs-fuse-arm/src/zfs-fuse $ sudo sh run.sh &

fdion@raspberrypi ~/zfs/zfs-fuse-arm/src/zfs-fuse $ cd
fdion@raspberrypi ~ $ cd zfs
fdion@raspberrypi ~/zfs $ mkdir test
fdion@raspberrypi ~/zfs $ cd test
fdion@raspberrypi ~/zfs/test $ dd if=/dev/zero of=fakedisk1 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 10.2747 s, 10.2 MB/s
fdion@raspberrypi ~/zfs/test $ dd if=/dev/zero of=fakedisk2 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 10.7517 s, 9.8 MB/s

Up to now we haven't done anything with ZFS per say. And basically to mirror two drives in ZFS and create a new storage out of that, all we have to do:

fdion@raspberrypi ~/zfs/test $ sudo zpool create mymirror mirror /home/fdion/zfs/test/fakedisk1 /home/fdion/zfs/test/fakedisk2

Now let's create a filesystem on that new zpool device, and mount it to a local folder in my home directory, change permissions so I can write to it and finally copy some files from /etc to my new filesystem:

fdion@raspberrypi ~/zfs/test $ cd
fdion@raspberrypi ~ $ mkdir myfilesystem
fdion@raspberrypi ~ $ sudo zfs create mymirror/myfilesystem -o mountpoint=/home/fdion/myfilesystem
fdion@raspberrypi ~ $ sudo chown fdion:pi myfilesystem/
fdion@raspberrypi ~/myfilesystem $ cp /etc/*.conf .
cp: cannot open `/etc/fuse.conf' for reading: Permission denied
fdion@raspberrypi ~/myfilesystem $ ls
adduser.conf          gssapi_mech.conf  libaudit.conf   pnm2ppa.conf
asound.conf           hdparm.conf       logrotate.conf  resolv.conf
ca-certificates.conf  host.conf         mke2fs.conf     rsyslog.conf
colord.conf           idmapd.conf       mtools.conf     sensors3.conf
debconf.conf          insserv.conf      nsswitch.conf   sysctl.conf
deluser.conf          ld.so.conf        ntp.conf        ts.conf
gai.conf              libao.conf        pam.conf        ucf.conf
fdion@raspberrypi ~/myfilesystem $ sudo zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
mymirror                191K  63.3M    22K  /mymirror
mymirror/myfilesystem  89.5K  63.3M  89.5K  /home/fdion/myfilesystem
fdion@raspberrypi ~/myfilesystem $ sudo zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
mymirror  95.5M   196K  95.3M     0%  1.00x  ONLINE  -
fdion@raspberrypi ~/myfilesystem $ 

How cool is that? I now have a mirrored backup of my .conf files. Well, not quite. We are using fake disks, so if the SD card dies I loose all.

So next time we'll demo with actual USB drives.