ZFS Tutorial Part 2


ZFS is an advanced, open-source, filesystem available on FreeBSD, illumos, Solaris and Linux. ZFS is quite different from traditional filesystems: the best way to understand it is with hands-on experience. This series of tutorials give you that experience, covering pool and filesystem management, fault tolerance, quotas, compression, snapshots, clones, and caching. We'll also put ZFS to work with jails on FreeBSD and containers on Linux.

In part 2 we look at ZFS filesystems. This builds on part 1, which covered ZFS pools. I highly recommend you read part 1 if you haven't already.


To follow this tutorial you need an OS with good ZFS support, such as FreeBSD 9+, illumos, or Ubuntu 16.04 LTS. If you're using another Linux distro then you should install zfsonlinux v0.6.4+. Solaris users will find that most of what is said here is relevant, but keep in mind that the Solaris implementation is a little different.

If you're running Ubuntu 16.04 LTS (Xenial Xerus) your kernel has ZFS support, but you still need to install the command-line tools:

$ sudo apt-get install zfsutils-linux

NB. FreeBSD/PC-BSD 10.1 users on SSD should make sure they're on 10.1-RELEASE-p11 or greater to avoid a kernel panic. See FreeBSD-EN-15:07.zfs for details.


You need root privileges to create or manage ZFS pools. If a command is shown with the # prompt then it needs to be run as root or with sudo. NB. On Ubuntu all ZFS commands need root privileges.

Disk Files

In this tutorial we use files to represent two disks:

$ mkdir /tmp/zfstut
$ dd bs=1m count=256 if=/dev/zero of=/tmp/zfstut/disk1
$ dd bs=1m count=256 if=/dev/zero of=/tmp/zfstut/disk2

$ ls -lh /tmp/zfstut
total 1
-rw-r--r--  1 flux  wheel   256M Jan 23 17:54 disk1
-rw-r--r--  1 flux  wheel   256M Jan 23 17:54 disk2

Filesystem Basics

Before you can manipulate filesystems you need to create a pool (you can learn about ZFS pools in part 1). When you create a pool an initial ZFS filesystem is created and mounted for you.

You use the zpool command to work with pools and the zfs command to work with filesystems.

Make sure you've created your disk files in /tmp/zfstut (see above). Then create a pool and view information on it and its initial filesystem:

# zpool create salmon mirror /tmp/zfstut/disk1 /tmp/zfstut/disk2

$ zpool list salmon
salmon   240M  80.5K   240M         -     1%     0%  1.00x  ONLINE  -

$ zfs list salmon
salmon  68.5K   208M    19K  /salmon

We can see our filesystem is mounted on /salmon and is 240M in size.

We can create an almost unlimited number (264) of new filesystems within our pool. Let's add filesystems for three users:

# zfs create salmon/kent
# zfs create salmon/dennisr
# zfs create salmon/billj

$ zfs list -r salmon
salmon           134K   208M    19K  /salmon
salmon/billj      19K   208M    19K  /salmon/billj
salmon/dennisr    19K   208M    19K  /salmon/dennisr
salmon/kent       19K   208M    19K  /salmon/kent

We use zfs list with the recursive (-r) option to restrict the listing to those filesystems that are part of salmon.

Note how all four filesystems share the same pool space and all report 208M available. We’ll see how to set quotas and reserve space later in this tutorial. We can create arbitrary levels of filesystems. For example, you could create whole tree of filesystems inside Bill's filesystem: /salmon/billj/src, /salmon/billj/src/c, /salmon/billj/src/x11 etc.

We can also see our filesystems using df. Be aware that sizes and free space reported by df can be misleading due to the shared nature of ZFS. Always use the zfs command if you want to ensure sizes are reported correctly.

$ df -h | grep salmon
salmon                208M     19K    208M     0%    /salmon
salmon/kent           208M     19K    208M     0%    /salmon/kent
salmon/dennisr        208M     19K    208M     0%    /salmon/dennisr
salmon/billj          208M     19K    208M     0%    /salmon/billj

You can remove filesystems with zfs destroy. Bill has stopped working on salmon, so let's remove his filesystem:

# zfs destroy salmon/billj

$ zfs list -r salmon
salmon           117K   208M    19K  /salmon
salmon/dennisr    19K   208M    19K  /salmon/dennisr
salmon/kent       19K   208M    19K  /salmon/kent

Mount Points

It’s convenient that ZFS automatically mounts your filesystem under the pool name, but this isn't always what you want. Thankfully it's very easy to change the mount point. For example, if we want to mount salmon under /projects directory:

# zfs set mountpoint=/projects/salmon salmon

$ zfs list -r salmon
salmon           130K   208M    19K  /projects/salmon
salmon/dennisr    19K   208M    19K  /projects/salmon/dennisr
salmon/kent       19K   208M    19K  /projects/salmon/kent

Mount points of filesystems are not limited to those of the pool as a whole, for example:

# zfs set mountpoint=/fishing salmon/kent

$ zfs list -r salmon
salmon           142K   208M    19K  /projects/salmon
salmon/dennisr    19K   208M    19K  /projects/salmon/dennisr
salmon/kent       19K   208M    19K  /fishing

$ df -h /fishing
Filesystem     Size    Used   Avail Capacity  Mounted on
salmon/kent    208M     19K    208M     0%    /fishing

To mount and unmount ZFS filesystems you use zfs mount and zfs unmount (zfs umount works too). ZFS filesystems are entirely managed by ZFS and don’t appear in /etc/fstab. In a subsequent tutorial we will look at using legacy mount points to manage filesystems the traditional way.

Try the following mount commands:

# zfs unmount salmon/kent

$ mount | grep salmon
salmon on /projects/salmon (zfs, local, nfsv4acls)
salmon/dennisr on /projects/salmon/dennisr (zfs, local, nfsv4acls)

# zfs mount salmon/kent

$ mount | grep salmon
salmon on /projects/salmon (zfs, local, nfsv4acls)
salmon/dennisr on /projects/salmon/dennisr (zfs, local, nfsv4acls)
salmon/kent on /fishing (zfs, local, nfsv4acls)

Filesystem Properties

ZFS allows you to control many aspects of a filesystem using properties. Filesystem properties work in the same way as the mount point (which is itself a property). To get and set properties we use zfs get and zfs set.

$ zfs get mountpoint salmon/kent
salmon/kent  available   208M        -

We can also get all properties on a filesystem:

$ zfs get all salmon/kent
NAME         PROPERTY              VALUE                  SOURCE
salmon/kent  type                  filesystem             -
salmon/kent  creation              Sat Feb 13 20:39 2016  -
salmon/kent  used                  19K                    -
salmon/kent  available             208M                   -
salmon/kent  referenced            19K                    -
salmon/kent  compressratio         1.00x                  -
salmon/kent  mounted               yes                    -
salmon/kent  quota                 none                   default
salmon/kent  reservation           none                   default
salmon/kent  recordsize            128K                   default
salmon/kent  mountpoint            /fishing               local
salmon/kent  sharenfs              off                    default

The source value shows where a property gets its value from. The first set of properties have a source of -: these properties provide information on your filesystem and are read only. There are three others sources for a property, all of which can be modified with zfs set:

  • default - the default ZFS value for this property
  • local - the property is set directly on this filesystem
  • inherited - the property is inherited from a parent filesystem

The salmon/kent mountpoint property is shown as from a local source; this is because we set a specific mount point for this filesystem above.

Get Options

There are several useful options you can use with zfs get to fine tune the information it returns.

  • -r recursively gets the property for all child filesystems.
  • -p reports precise values in bytes (e.g. 9437184 rather than 9M).
  • -H omits header fields, making the output easier for scripts to parse.
  • -o field1, field2… specify a list of fields you wish to get (avoids having to use awk or cut).

For example, to get the exact used and available values for salmon without including a header:

$ zfs get -p -H -o name,property,value used,available salmon
salmon used  147456
salmon available 217956352

To retrieve the mount point for salmon and its child filesystems:

$ zfs get -r -H -o name,value mountpoint salmon
salmon /projects/salmon
salmon/dennisr /projects/salmon/dennisr
salmon/kent /fishing

I'm going to look at three properties in the remainder of this tutorial: quota, reservation, and compression. Other properties will be covered in later tutorials or you can read about them in the zfs man page.

Quotas & Reservations

All the filesystems in a pool share the same disk space. This maximises flexibility and lets ZFS make best use of the resources, however it does allow one filesystem to (mis)use all the space. Quotas and reservations allow you to manage space utilisation within a pool. A quota sets a limit on the space a filesystem can use within a pool. A reservation sets aside part of the pool for the exclusive use of a filesystem.

To see how this works, let’s consider our existing salmon pool:

$ zfs list -r salmon
salmon           142K   208M    19K  /projects/salmon
salmon/dennisr    19K   208M    19K  /projects/salmon/dennisr
salmon/kent       19K   208M    19K  /fishing

For example, let's say we want to set a quota of 50 megabytes on Dennis and Ken to ensure there's space for later users of the salmon pool:

# zfs set quota=50m salmon/dennisr
# zfs set quota=50m salmon/kent

$ zfs list -r salmon
salmon           144K   208M    19K  /projects/salmon
salmon/dennisr    19K  50.0M    19K  /projects/salmon/dennisr
salmon/kent       19K  50.0M    19K  /fishing

Note how the available space had updated for the filesystems we set a quota on. We can also use the zfs get command to retrieve the quota property:

$ zfs get -r quota salmon
salmon          quota     none   default
salmon/dennisr  quota     50M    local
salmon/kent     quota     50M    local

As an example of reservations let's add a new filesystem for Jeff and reserve 40 megabytes of space for it. This ensures that however full the disk gets this filesystem always has access to at least 40 MiB.

# zfs create salmon/jeffb
# zfs set reservation=40M salmon/jeffb

$ zfs get -r reservation salmon
salmon          reservation  none    default
salmon/dennisr  reservation  none    default
salmon/jeffb    reservation  40M     local
salmon/kent     reservation  none    default

If we review our list of filesystems we can see the overall effect of the quotas and reservation:

$ zfs list -r salmon
salmon          40.1M   168M    19K  /projects/salmon
salmon/dennisr    19K  50.0M    19K  /projects/salmon/dennisr
salmon/jeffb      19K   208M    19K  /projects/salmon/jeffb
salmon/kent       19K  50.0M    19K  /fishing

As expected the space available to Dennis and Ken is limited to 50 MiB, but there appears to be no change to Jeff. However, if we look at the used space for salmon as a whole we can see this has risen to 40 MiB. This space isn't actually used, but because it has been reserved for Jeff it isn't available to the rest of the pool. Reservations can lead you to over-estimate the space used in your pool; be careful when looking at 'AVAIL' values when using reservations.


ZFS has built-in support for lz4 compression; not only does this save disk space, but it usually improves performance.

Let's enable compression on Jeff's filesystem:

# zfs set compression=lz4 salmon/jeffb
$ zfs get -r compression salmon
salmon          compression  off       default
salmon/dennisr  compression  off       default
salmon/jeffb    compression  lz4       local
salmon/kent     compression  off       default

Now let's write some data to Jeff and Dennis's filesystems to test it. In this case we use the OS word list, but you can use your own file if you prefer:

$ cp /usr/share/dict/words /projects/salmon/dennisr
$ cp /usr/share/dict/words /projects/salmon/jeffb

# zfs list -r salmon/dennisr salmon/jeffb
salmon/dennisr  2.52M  47.5M  2.52M  /projects/salmon/dennisr
salmon/jeffb    1.35M   204M  1.35M  /projects/salmon/jeffb

You can see the dictionary takes up about half the space on the compressed filesystem. We can use options with zfs list to see details of the compression in a table:

$ zfs list -r -o name,used,compressratio,compression salmon
salmon          44.0M  1.30x       off
salmon/dennisr  2.52M  1.00x       off
salmon/jeffb    1.35M  1.87x       lz4
salmon/kent       19K  1.00x       off

Compression can be enabled or disabled at any time, but it only affects blocks written after the setting is changed: existing data blocks remain as they are. This means it's generally desirable to enable compression as soon as the filesystem is created.

If your OS is installed on ZFS you can check out the compression settings on your system filesystems. In this case we can see that FreeBSD is using lz4 on all filesystems:

$ zfs list -r -o name,used,compressratio,compression zroot
NAME                 USED  RATIO  COMPRESS
zroot                554M  1.98x       lz4
zroot/ROOT           501M  2.07x       lz4
zroot/ROOT/default   500M  2.07x       lz4
zroot/tmp           51.5M  1.20x       lz4

By default I recommend you enable compression on all your ZFS filesystems: even the performance penalty for incompressible data is minor. If you're interested in learning more about ZFS compression performance you should read the illumos wiki.

Finally you should clean up the salmon pool and remove the disk files:

# zpool destroy salmon
$ rm -r /tmp/zfstut


That's it for part 2. I hope you now feel confident working with ZFS filesystems and are ready to experiment with filesystem properties. In the next part (being written as of April 2016) we will look at filesystem snapshots, clones, and send/receive. ♆