I spent a couple of years running ZFS on one of my disks, after reading all the hype. I think ZFS might be a little bit overhyped indeed.
I put ZFS on a disk used for various workloads. It had small files, large files, bulk i/o, persistent background i/o, interactive i/o. A little bit of everything.
There was a single physical disk under ZFS, no raid.
FreeBSD went through a buffer cache unification sometime back in 4.x or 5.0 era. My understanding of it is there are fewer copies of data read from disk between the disk itself and the program actually operating on it. This means better performance and better memory utilization.
ZFS throws all of this work out of the water because it has its own ARC cache. And because ZFS does not apparently use any of freebsd's other caches, the arc cache needs to be quite large indeed. On a ZFS-only system maybe this is not so bad, but on a mixed system this is somewhat problematic as the ram is fixed in arc cache and, for example, I cannot temporarily repurpose it to build one of those huge packages like web browsers.
Writes block entire system
On UFS, writes start going out to disk almost immediately, and as they are happening the system remains usable and pretty responsive, just slower. On ZFS, writes are nearly instantaneous as long as they are going into the write cache, but once the cache fills up the entire system seems to pause while the cache is being flushed.
This is extremely noticeable during interactive work if there is a bulk transfer happening in the background. The terminal simply hangs if it has anything at all to do with the ZFS disk.
Another case where this sort of behavior is unacceptable is for recording Internet radio. At 128 kbps (that's bits) the amount of disk i/o bandwidth required for persisting the stream is negligible. Yet ZFS pauses the entire world to such an extent that my recorded streams were skipping if I was recording them to ZFS partitions.
This bug report for FreeNAS is even more extreme than the behavior I was seeing, but the concept appears to be the same.
When multiple files are being written to a ZFS partition concurrently, they appear to be severely fragmented. I can tell because reading these files back takes much longer than normal.
UFS seems to be much more suitable to concurrent write workloads.
This article reflects someone else's experience with fragmentation on ZFS.
I could not tell that ZFS was faster than UFS in any way. Perhaps with a raid the situation would have been different. My preferred approach to backups is to copy important files to a separate disk, ideally with different access patterns. For cases where it is feasible (git repositories), I back up off-site to my servers.
I could not see any benefits of ZFS for my use case of a single disk.
df output looks prettier with ZFS, but
du accomplishes nearly the same
thing, just takes longer.
Constant system freezes while ZFS writes data to disk, on the other hand, are simply unacceptable.
As the FreeNAS bug report suggests, ZFS performance can be improved by tuning it. But why bother when UFS does everything right out of the box? I am moving back to UFS, unfortunately I needed to source another disk because I could not scrape together enough free space on the remaining disks to get all of the data off of the ZFS disk to repartition it.