ZFS is an advanced, open-source, filesystem available on FreeBSD, illumos, Solaris and Linux. ZFS is quite different from traditional filesystems: the best way to understand it is with hands-on experience. This series of tutorials give you that experience, covering pool and filesystem management, fault tolerance, quotas, compression, snapshots, clones, and caching. We'll also put ZFS to work with jails on FreeBSD and containers on Linux.
To follow this tutorial you need an OS with good ZFS support, such as FreeBSD 9+, illumos, or Ubuntu 16.04 LTS. If you're using another Linux distro then you should install zfsonlinux v0.6.4+. Solaris users will find that most of what is said here is relevant, but keep in mind that the Solaris implementation is a little different.
If you're running Ubuntu 16.04 LTS (Xenial Xerus) your kernel has ZFS support, but you still need to install the command-line tools:
$ sudo apt-get install zfsutils-linux
NB. FreeBSD/PC-BSD 10.1 users on SSD should make sure they're on 10.1-RELEASE-p11 or greater to avoid a kernel panic. See FreeBSD-EN-15:07.zfs for details.
You need root privileges to create or manage ZFS pools. If a command is shown with the # prompt then it needs to be run as root or with sudo. NB. On Ubuntu all ZFS commands need root privileges.
In this tutorial we use files to represent two disks:
$ mkdir /tmp/zfstut $ dd bs=1m count=256 if=/dev/zero of=/tmp/zfstut/disk1 $ dd bs=1m count=256 if=/dev/zero of=/tmp/zfstut/disk2 $ ls -lh /tmp/zfstut total 1 -rw-r--r-- 1 flux wheel 256M Jan 23 17:54 disk1 -rw-r--r-- 1 flux wheel 256M Jan 23 17:54 disk2
Before you can manipulate filesystems you need to create a pool (you can learn about ZFS pools in part 1). When you create a pool an initial ZFS filesystem is created and mounted for you.
You use the zpool command to work with pools and the zfs command to work with filesystems.
Make sure you've created your disk files in
/tmp/zfstut (see above). Then create a pool and view information on it and its initial filesystem:
# zpool create salmon mirror /tmp/zfstut/disk1 /tmp/zfstut/disk2 $ zpool list salmon NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT salmon 240M 80.5K 240M - 1% 0% 1.00x ONLINE - $ zfs list salmon NAME USED AVAIL REFER MOUNTPOINT salmon 68.5K 208M 19K /salmon
We can see our filesystem is mounted on
/salmon and is 240M in size.
We can create an almost unlimited number (264) of new filesystems within our pool. Let's add filesystems for three users:
# zfs create salmon/kent # zfs create salmon/dennisr # zfs create salmon/billj $ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 134K 208M 19K /salmon salmon/billj 19K 208M 19K /salmon/billj salmon/dennisr 19K 208M 19K /salmon/dennisr salmon/kent 19K 208M 19K /salmon/kent
zfs list with the recursive (
-r) option to restrict the listing to those filesystems that are part of salmon.
Note how all four filesystems share the same pool space and all report 208M available. We’ll see how to set quotas and reserve space later in this tutorial. We can create arbitrary levels of filesystems. For example, you could create whole tree of filesystems inside Bill's filesystem:
We can also see our filesystems using df. Be aware that sizes and free space reported by df can be misleading due to the shared nature of ZFS. Always use the zfs command if you want to ensure sizes are reported correctly.
$ df -h | grep salmon salmon 208M 19K 208M 0% /salmon salmon/kent 208M 19K 208M 0% /salmon/kent salmon/dennisr 208M 19K 208M 0% /salmon/dennisr salmon/billj 208M 19K 208M 0% /salmon/billj
You can remove filesystems with zfs destroy. Bill has stopped working on salmon, so let's remove his filesystem:
# zfs destroy salmon/billj $ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 117K 208M 19K /salmon salmon/dennisr 19K 208M 19K /salmon/dennisr salmon/kent 19K 208M 19K /salmon/kent
It’s convenient that ZFS automatically mounts your filesystem under the pool name, but this isn't always what you want. Thankfully it's very easy to change the mount point. For example, if we want to mount salmon under
# zfs set mountpoint=/projects/salmon salmon $ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 130K 208M 19K /projects/salmon salmon/dennisr 19K 208M 19K /projects/salmon/dennisr salmon/kent 19K 208M 19K /projects/salmon/kent
Mount points of filesystems are not limited to those of the pool as a whole, for example:
# zfs set mountpoint=/fishing salmon/kent $ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 142K 208M 19K /projects/salmon salmon/dennisr 19K 208M 19K /projects/salmon/dennisr salmon/kent 19K 208M 19K /fishing $ df -h /fishing Filesystem Size Used Avail Capacity Mounted on salmon/kent 208M 19K 208M 0% /fishing
To mount and unmount ZFS filesystems you use
zfs mount and
zfs unmount (
zfs umount works too). ZFS filesystems are entirely managed by ZFS and don’t appear in
/etc/fstab. In a subsequent tutorial we will look at using legacy mount points to manage filesystems the traditional way.
Try the following mount commands:
# zfs unmount salmon/kent $ mount | grep salmon salmon on /projects/salmon (zfs, local, nfsv4acls) salmon/dennisr on /projects/salmon/dennisr (zfs, local, nfsv4acls) # zfs mount salmon/kent $ mount | grep salmon salmon on /projects/salmon (zfs, local, nfsv4acls) salmon/dennisr on /projects/salmon/dennisr (zfs, local, nfsv4acls) salmon/kent on /fishing (zfs, local, nfsv4acls)
ZFS allows you to control many aspects of a filesystem using properties. Filesystem properties work in the same way as the mount point (which is itself a property). To get and set properties we use
zfs get and
$ zfs get mountpoint salmon/kent NAME PROPERTY VALUE SOURCE salmon/kent available 208M -
We can also get all properties on a filesystem:
$ zfs get all salmon/kent NAME PROPERTY VALUE SOURCE salmon/kent type filesystem - salmon/kent creation Sat Feb 13 20:39 2016 - salmon/kent used 19K - salmon/kent available 208M - salmon/kent referenced 19K - salmon/kent compressratio 1.00x - salmon/kent mounted yes - salmon/kent quota none default salmon/kent reservation none default salmon/kent recordsize 128K default salmon/kent mountpoint /fishing local salmon/kent sharenfs off default …
The source value shows where a property gets its value from. The first set of properties have a source of
-: these properties provide information on your filesystem and are read only. There are three others sources for a property, all of which can be modified with
- default - the default ZFS value for this property
- local - the property is set directly on this filesystem
- inherited - the property is inherited from a parent filesystem
salmon/kent mountpoint property is shown as from a local source; this is because we set a specific mount point for this filesystem above.
There are several useful options you can use with
zfs get to fine tune the information it returns.
- -r recursively gets the property for all child filesystems.
- -p reports precise values in bytes (e.g. 9437184 rather than 9M).
- -H omits header fields, making the output easier for scripts to parse.
- -o field1, field2… specify a list of fields you wish to get (avoids having to use awk or cut).
For example, to get the exact used and available values for salmon without including a header:
$ zfs get -p -H -o name,property,value used,available salmon salmon used 147456 salmon available 217956352
To retrieve the mount point for salmon and its child filesystems:
$ zfs get -r -H -o name,value mountpoint salmon salmon /projects/salmon salmon/dennisr /projects/salmon/dennisr salmon/kent /fishing
I'm going to look at three properties in the remainder of this tutorial: quota, reservation, and compression. Other properties will be covered in later tutorials or you can read about them in the zfs man page.
Quotas & Reservations
All the filesystems in a pool share the same disk space. This maximises flexibility and lets ZFS make best use of the resources, however it does allow one filesystem to (mis)use all the space. Quotas and reservations allow you to manage space utilisation within a pool. A quota sets a limit on the space a filesystem can use within a pool. A reservation sets aside part of the pool for the exclusive use of a filesystem.
To see how this works, let’s consider our existing salmon pool:
$ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 142K 208M 19K /projects/salmon salmon/dennisr 19K 208M 19K /projects/salmon/dennisr salmon/kent 19K 208M 19K /fishing
For example, let's say we want to set a quota of 50 megabytes on Dennis and Ken to ensure there's space for later users of the salmon pool:
# zfs set quota=50m salmon/dennisr # zfs set quota=50m salmon/kent $ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 144K 208M 19K /projects/salmon salmon/dennisr 19K 50.0M 19K /projects/salmon/dennisr salmon/kent 19K 50.0M 19K /fishing
Note how the available space had updated for the filesystems we set a quota on. We can also use the
zfs get command to retrieve the quota property:
$ zfs get -r quota salmon NAME PROPERTY VALUE SOURCE salmon quota none default salmon/dennisr quota 50M local salmon/kent quota 50M local
As an example of reservations let's add a new filesystem for Jeff and reserve 40 megabytes of space for it. This ensures that however full the disk gets this filesystem always has access to at least 40 MiB.
# zfs create salmon/jeffb # zfs set reservation=40M salmon/jeffb $ zfs get -r reservation salmon NAME PROPERTY VALUE SOURCE salmon reservation none default salmon/dennisr reservation none default salmon/jeffb reservation 40M local salmon/kent reservation none default
If we review our list of filesystems we can see the overall effect of the quotas and reservation:
$ zfs list -r salmon NAME USED AVAIL REFER MOUNTPOINT salmon 40.1M 168M 19K /projects/salmon salmon/dennisr 19K 50.0M 19K /projects/salmon/dennisr salmon/jeffb 19K 208M 19K /projects/salmon/jeffb salmon/kent 19K 50.0M 19K /fishing
As expected the space available to Dennis and Ken is limited to 50 MiB, but there appears to be no change to Jeff. However, if we look at the used space for salmon as a whole we can see this has risen to 40 MiB. This space isn't actually used, but because it has been reserved for Jeff it isn't available to the rest of the pool. Reservations can lead you to over-estimate the space used in your pool; be careful when looking at 'AVAIL' values when using reservations.
ZFS has built-in support for lz4 compression; not only does this save disk space, but it usually improves performance.
Let's enable compression on Jeff's filesystem:
# zfs set compression=lz4 salmon/jeffb $ zfs get -r compression salmon NAME PROPERTY VALUE SOURCE salmon compression off default salmon/dennisr compression off default salmon/jeffb compression lz4 local salmon/kent compression off default
Now let's write some data to Jeff and Dennis's filesystems to test it. In this case we use the OS word list, but you can use your own file if you prefer:
$ cp /usr/share/dict/words /projects/salmon/dennisr $ cp /usr/share/dict/words /projects/salmon/jeffb # zfs list -r salmon/dennisr salmon/jeffb NAME USED AVAIL REFER MOUNTPOINT salmon/dennisr 2.52M 47.5M 2.52M /projects/salmon/dennisr salmon/jeffb 1.35M 204M 1.35M /projects/salmon/jeffb
You can see the dictionary takes up about half the space on the compressed filesystem. We can use options with
zfs list to see details of the compression in a table:
$ zfs list -r -o name,used,compressratio,compression salmon NAME USED RATIO COMPRESS salmon 44.0M 1.30x off salmon/dennisr 2.52M 1.00x off salmon/jeffb 1.35M 1.87x lz4 salmon/kent 19K 1.00x off
Compression can be enabled or disabled at any time, but it only affects blocks written after the setting is changed: existing data blocks remain as they are. This means it's generally desirable to enable compression as soon as the filesystem is created.
If your OS is installed on ZFS you can check out the compression settings on your system filesystems. In this case we can see that FreeBSD is using lz4 on all filesystems:
$ zfs list -r -o name,used,compressratio,compression zroot NAME USED RATIO COMPRESS zroot 554M 1.98x lz4 zroot/ROOT 501M 2.07x lz4 zroot/ROOT/default 500M 2.07x lz4 zroot/tmp 51.5M 1.20x lz4 …
By default I recommend you enable compression on all your ZFS filesystems: even the performance penalty for incompressible data is minor. If you're interested in learning more about ZFS compression performance you should read the illumos wiki.
Finally you should clean up the salmon pool and remove the disk files:
# zpool destroy salmon $ rm -r /tmp/zfstut
That's it for part 2. I hope you now feel confident working with ZFS filesystems and are ready to experiment with filesystem properties. In the next part (being written as of April 2016) we will look at filesystem snapshots, clones, and send/receive. ♆