Testing BTRFS compression

[bjlli]

Hi,

I was trying to use fio to test BTRFS compression on my SoM (i.MX6ULL/eMMC), but the compression doesn’t seem to work for write or randwrite. The generated file is not compressed even using 0x00 pattern, i.e the results are the same whether the compression is enabled or not.

I’m using fio version 3.30 (a3e48f4), and I’m not sure if this is a limitation or something I’m doing wrong.

If anyone has experience testing BTRFS compression with fio, I’d appreciate any guidance.

[sitsofe]

Hello @bjlli:
You didn't include the fio command line you are running (as mentioned in https://github.com/axboe/fio/blob/fio-3.41/REPORTING-BUGS ) and we don't know anything about your BtrFS environment or how you determined that compression didn't take place. As such it's hard to say what has happened...

One possible reason why you didn't see any compression is because you requested direct I/O via --direct=1 and you have checksumming turned off at the inode level. If so, then (by virtue of what BtrFS does) checksumming and compression are bypassed. See https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#compatibility for details.

[bjlli]

Hi, sorry for not following the guidelines earlier. As I mentioned, I’m not even sure this is actually a bug, probably some wrong usage.
Just to add a bit more context about what I tested:

• Kernel: 6.1.42
• Filesystem mounted with compression enabled

fio command I used:

fio --name=test --rw=write --size=500M --direct=0 --numjobs=1 --bsrange=64k-128k --group_reporting --zero_buffers

I’ll retest with the last release and check if the behavior changes.

Thanks for the help.

[sitsofe]

Hi @bjlli,

(As you might have guessed I'm not a BtrFS expert) While preparing to try and recreate your issue locally I found this in the BtrFS documentation Compression: Exceptions section:

Exceptions

Any file that has been touched by the fallocate system call will always be excepted from compression even if force-compress mount option is used.

The reason for this is that a successful fallocate call must guarantee that future writes to the allocated range will not fail because of lack of space. This is difficult to guarantee in a COW filesystem. To reduce the chances of it happening, btrfs preallocates space and disables compression for the file.

[...]

By default on Linux fio defaults its fallocate option to native which means if the file to be used doesn't exist at the required size it will be pre-allocated using fallocate before I/O is done to it. I believe it's this interaction which is leading to the result you are seeing. You can tell fio not to use fallocate but you'll have to ensure the file to be used isn't a leftover from a previous run that created it with fallocate.

Let us know if this explains what you've seen.

[sitsofe]

Some quick local tests on BtrFS and their results are below (@bjlli you didn't mention how you determined whether compression took place so I'm going to use compsize to determine this):

fallocate -l 500MiB /tmp/btrfs0
sudo losetup /dev/loop0 /tmp/btrfs0
sudo mkfs.btrfs /dev/loop0
sudo mkdir /mnt/btrfs0
sudo mount -o compress=zstd /dev/loop0 /mnt/btrfs0/
sudo chmod a+rw /mnt/btrfs0/
cd /mnt/btrfs0
rm -f fio.*
fio --minimal --filename=fio.fallocate.default                       --size=10m --bs=1m --rw=write --end_fsync=1 --zero_buffers --name=go > /dev/null
fio --minimal --filename=fio.fallocate.native --fallocate=native     --size=10m --bs=1m --rw=write --end_fsync=1 --zero_buffers --name=go > /dev/null
fio --minimal --filename=fio.fallocate.none --fallocate=none         --size=10m --bs=1m --rw=write --end_fsync=1 --zero_buffers --name=go > /dev/null
fio --minimal --filename=fio.fallocate.posix --fallocate=posix       --size=10m --bs=1m --rw=write --end_fsync=1 --zero_buffers --name=go > /dev/null
fio --minimal --filename=fio.fallocate.truncate --fallocate=truncate --size=10m --bs=1m --rw=write --end_fsync=1 --zero_buffers --name=go > /dev/null

for f in fio.*; do echo $f; compsize $f; done
fio.fallocate.default
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       10M          10M          10M       
none       100%       10M          10M          10M       
fio.fallocate.native
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       10M          10M          10M       
none       100%       10M          10M          10M       
fio.fallocate.none
Processed 1 file, 80 regular extents (80 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL        3%      320K          10M          10M       
zstd         3%      320K          10M          10M       
fio.fallocate.posix
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       10M          10M          10M       
none       100%       10M          10M          10M       
fio.fallocate.truncate
Processed 1 file, 80 regular extents (80 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL        3%      320K          10M          10M       
zstd         3%      320K          10M          10M       

Additionally we can see fallocate triggering preallocation if we are able to check the file before writes are flushed to it:

$ rm -f fio.tmp; fio --minimal --filename=fio.tmp --size=10m --bs=4k --number_ios=1 --zero_buffers --name=go --rw=write > /dev/null; compsize fio.tmp
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       10M          10M          10M       
prealloc   100%       10M          10M          10M 

(notice how it says prealloc rather than zstd or none).

If you choose to turn fallocate off (or choose truncation) be aware that this may also impact your runtime because you will be growing the file while writing to it which can have consequences (e.g. it may be more fragmented by the time you finish). Of course it's hard to reason about pre-allocation on a filesystem that is Copy On Write...

[bjlli]

Hi @sitsofe, thank you so much!

The compression worked when I turned off the fallocate (I used --fallocate=none). I tested with the older version of fio 3.30, but I'll also generate a Linux image to test with a newer version :)