BLAH-BLAH > IT > General > NILFS2 filesystem performance
First published on: 05.Sep.2011

NILFS2 filesystem performance


Introduction

I tried a couple of times to install my Linux distribution (I use Gentoo) on a USB-stick so that I wouldn't have to waste a HDD just for that, but even by expecting a low write-speed due to the usual problems that flash-memory has in that area, I was always disappointed to see that the timings were in the end actually even worse than what I thought. I always ended up using a normal HDD.

During the last days the desire to have the distribution on a USB-stick emerged again as I do not want to waste one of the 4 HDD-slots of my NAS just for the OS and I therefore looked again at possible candidate filesystems that might give me a good performance when used on flash-storage.

I excluded right at the beginning many of the classic journaling filesystems (ext3, reiserfs, jfs, xfs) as, as I said, when I did my attempts in the past using them the results were quite disappointing. Additionally I always felt unsure because of the issue that they continously modify their journal, accentuating therefore the problem of those flash-cells reaching their physical limit of write-cycles much quicker than the rest.

Then I excluded ext4 because ......... I somehow never had a good feeling about it being supported in the future or having some interesting or new features.

Btrfs isn't actually bad, especially if mounted with the ''ssd_spread'' but I excluded it as well as I had already 2 total failures (once in a VM and once on an OS running on a USB-stick). Currently the filesystem-check of btrfs isn't very useful (in the VM it didn't find the errors, on the USB-stick it found them but didn't correct them).

A big mistake I did in the past was not to look at ext2.
It's very old, and well, I wanted to use something new! And I probably got influenced by its bad performance when I tested it using normal HDDs.
But by testing it now I saw that its performance on flash-memory isn't actually that bad!

Finally, I saw that NILFS2, with its continuous log approach, was a good candidate.
I therefore ended up benchmarking NILFS2, EXT2 and EXT3 (as a representative of journaling filesystems).
Addition 28.Jul.2010: I added as well benchmarks of ext4 and btrfs as I was asked how performance with these 2 other fs was.

Update 02.Oct.2010:
Everything still running fine - no corruptions until now (perhaps I shouldn't say this ;oP ).
Things were running so fine with the OS on the USB-stick (e.g. even small things like ssh-logins that take 0.5 seconds to bring back the console compared to the 1 second when using 2.5 5400rpm HDD) that I decided to use NILFS2 as well for the new SSD I bought for the notebook.
On the notebook I did everything the same way (there I am now using the kernel 2.6.35-r5 of Gentoo) with the exception of adding in /etc/fstab the "discard" option for nilfs2 to use the TRIM function of the SSD.

Update 18.Dec.2010:
Everything still running fine on both USB-stick and SSD.
I forgot to mention that when running the tests on the USB-stick I was using the "CFQ" scheduler and that now on the SSD I am using "No-op" (thanks for the question, Gordan).
Using the SSD (Corsair Nova V128 SSD MLC, 128GB) in combination with Nilfs2 + No-op I reach ~120MB/s write and ~230MB/s read for single big files - time for "sync" after writing is 0.

Update 07.Nov.2011:
I reran a similar test, but this time using a SSD instead of the USB-stick - here are the results.


 

Tests

I decided not to use a test-suite - they perform a huge amount of tests and in the end I usually don't know which ones represent my usual workload.

Therefore I just ran the following manual ones:

  • write a 4GB file - general idea about big files
    time dd if=/dev/zero of=2delete bs=64k count=65536 && time sync
  • read the 4GB file - general idea about big file
    time cat 2delete > /dev/null
  • write 1019 directories and 4499 very small files - usual Gentoo workload when using "emerge"
    time cp -vR /usr/portage/dev-perl/ /mnt/memstick/ && time sync
  • overwrite the 1019 directories and 4499 very small files - usual Gentoo workload when using "emerge"
    time cp -vR /usr/portage/dev-perl/ /mnt/memstick/ && time sync

 

I assumed that read-times are like reading the big 4GB-file as there are no problems with seek-times on flash-memory.

After formatting the partition (default settings - the block size was always of 4KB) I mounted them disabling the tracking of the access time, as I do in the "real world" with all my partitions on all my PCs:

mount -v -o noatime /dev/sdb3 /mnt/memstick/

Exception: with Btrfs I added as well the option "ssd" as I wasn't sure if the USB-stick was being automatically recognized as a SSD/flash.

I initially used 2 different USB-sticks: a "Trascend JF V30 16GB" and a "extremememory Supersonic 150/50x.
The "extremememory Supersonic 150/50x" was so bad just after the first test that I decided to ignore it (funny, being branded with such a name...). That's a pity because another "extremememory" USB-stick I bought at the beginning of 2009 was very fast, but it is too small (4GB) to hold my distribution (~2GB) plus a swap-partition (1GB) plus some additional space to grow and/or temporary files.

The tests were performed on my notebook: Kernel 2.6.32-gentoo-r3 SMP PREEMPT x86_64 Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz
The btrfs-utils I used were version 0.19.


No other activities/tasks running while performing the tests.


 

Benchmark results

Options/Filesystem ext2 ext3 ext4 btrfs nilfs
Format options default
default
default
default
default
Mount options noatime
noatime
noatime
noatime, ssd
noatime

 

Tests/Filesystem ext2 ext3 ext4 btrfs nilfs
Write 4GB file 893 817 552 553 808
Read 4GB file 237 237 222 208 239
Write small files 7 18 12 10 4
Overwrite small files 6 4 5 19 4

Execution time in seconds:

NILFS - 4GB write

NILFS - 4GB read

NILFS - Write 1019 dirs and 4499 small files

NILFS - Overwrite 1019 dirs and 4499 small files

 

Looking at the results:

As I mentioned previously I anyway excluded BtrFS because of its instability, but anyway as I can see it has big problems when overwriting small files. Therefore, no Btrfs.

ext3
, which I would anyway exclude because of the problems with the journal & flash-memory, has problems when writing small files, which is what the Gentoo distribution does extensively. So, no ext3.

ext4
is strong when writing and reading big files, at least on flash-memory. It's for sure a good candidate to be used as storage e.g. for media files. But on the other side it is 3 times slower compared to nilfs2 when writing the directories and small files, so not good for e.g. taking backups of root-partitions or holding root partitions.

ext2
is quite good too, but nilfs2 needed about 1 minute less when writing the big file and was almost double as fast when writing for the first time the small files.

In the end I chose nilfs2 to hold my root partition on my USB-stick.

I am using it since almost 2 months now and I didn't have any problems (yet).
Sometimes when the free space gets too low the garbage collector wakes up and starts hunting old checkpoints that can be deleted to free up space (this until the desired limit has been reached - check the config file options) and there it becomes slow, but as this happens rarely (using ~2.5GB out of the 16GB available) it doesn't bother me.


 

Problems

I used NILFS2 for my root filesystem and I came up to some interesting things:

  • No clue why, but the root-fs always mounted read-only during the boot sequence.
    • One thing I did was to change grub's config to have "rw" for the root filesystem:
      kernel /boot/kernel-2.6.32-gentoo-r7 rw root=/dev/sda3 doslowusb rootdelay=15 console=tty1
    • Then I set that as well in '''/etc/fstab''':
      /dev/sda3 / nilfs2 noatime,'''rw''' 0 0
    • After that one more thing I had to do was to change the code of '''/etc/init.d/checkroot'''. For reasons I did not understand it always remounted the root filesystem in read-only mode. I added 2 lines and commented the original test:
      ================

      depend() {

      before *

      }

      start() {

      local retval=0

      if [[ ! -f /fastboot && -z ${CDBOOT} ]] \

      && ! is_net_fs / && ! is_union_fs / ; then


      MYCUSTOM=0

      if [[ -z ${MYCUSTOM} ]] ; then

      #               if touch -c / >& /dev/null ; then


      ebegin "Remounting root filesystem read-only"

      mount -n -o remount,ro /     

      eend $?

      ......

      ================

      Quite ugly, but it works Smile
  • The behaviour of NILFS2 is set by the setup in the file /etc/nilfs_cleanerd.conf.
    You'll HAVE to review the settings in there even just to be aware of how to behave with the fs (e.g. the default setting is not to deallocate space that was used by files that were deleted during the last 3600 seconds => if you do a lot of writing you should lower this setting).
  • The garbage-collector did not work for me out-of-the-box when using NILFS2 as root fs. I had to implement a small script that runs "killall nilfs_cleanerd" and right after that "nilfs_cleanerd /dev/sda3" when the OS boots (in the last runlevel).
  • NILFS2 doesn't have a filesystem-check utility (fsck) because (from what I understood) if the FS is corrupted it will just enable the youngest checkpoint which is marked as not corrupted. To avoid the message during boot about the fsck-utility not being found just change in '''/etc/fstab''' the settings not to perform a filesystem check - in my case:
    /dev/sda3   /   nilfs2   noatime,rw   0   0