NILFS2 filesystem performance: (18.Jun.2010)
3 simple tests to compare NILFS2 against EXT2 and EXT3
Intro
I tried a couple of times to install my Linux distribution (I use Gentoo) on a USB-stick so that I wouldn't have to waste a HDD just for that, but even by expecting a low write-speed due to the usual problems that flash-memory has in that area, I was always disappointed to see that the timings were in the end actually even worse than what I thought. I always ended up using a normal HDD.
During the last days the desire to have the distribution on a USB-stick emerged again as I do not want to waste one of the 4 HDD-slots of my NAS just for the OS and I therefore looked again at possible candidate filesystems that might give me a good performance when used on flash-storage.
I excluded right at the beginning many of the classic journaling filesystems (ext3, reiserfs, jfs, xfs) as, as I said, when I did my attempts in the past using them the results were quite disappointing. Additionally I always felt unsure because of the issue that they continously modify their journal, accentuating therefore the problem of those flash-cells reaching their physical limit of write-cycles much quicker than the rest.
Then I excluded ext4 because ......... I somehow never had a good feeling about it being supported in the future or having some interesting or new features.
Btrfs isn't actually bad, especially if mounted with the ssd_spread but I excluded it as well as I had already 2 total failures (once in a VM and once on an OS running on a USB-stick). Currently the filesystem-check of btrfs isn't very useful (in the VM it didn't find the errors, on the USB-stick it found them but didn't correct them).
A big mistake I did in the past was not to look at ext2.
It's very old, and well, I wanted to use something new! And I probably got influenced by its bad performance when I tested it using normal HDDs.
But by testing it now I saw that its performance on flash-memory isn't actually that bad!
Finally, I saw that NILFS2, with its continuous log approach, was a good candidate.
I therefore ended up benchmarking NILFS2, EXT2 and EXT3 (as a representative of journaling filesystems).
Addition 28.Jul.2010: I added as well benchmarks of ext4 and btrfs as I was asked how performance with these 2 other fs was.
Tests
I decided not to use a test-suite - they perform a huge amount of tests and in the end I usually don't know which ones represent my usual workload.
Therefore I just ran the following manual ones:
- write a 4GB file - general idea about big files
time dd if=/dev/zero of=2delete bs=64k count=65536 && time sync
- read the 4GB file - general idea about big files
time cat 2delete > /dev/null
- write 1019 directories and 4499 very small files - usual Gentoo workload when using "emerge"
time cp -vR /usr/portage/dev-perl/ /mnt/memstick/ && time sync
- overwrite the 1019 directories and 4499 very small files - usual Gentoo workload when using "emerge"
time cp -vR /usr/portage/dev-perl/ /mnt/memstick/ && time sync
I assumed that read-times are like reading the big 4GB-file as there are no problems with seek-times on flash-memory.
After formatting the partition (default settings - the block size was always of 4KB) I mounted them disabling the tracking of the access time, as I do in the "real world" with all my partitions on all my PCs:
mount -v -o noatime /dev/sdb3 /mnt/memstick/
Exception: with Btrfs I added as well the option "ssd" as I wasn't sure if the USB-stick was being automatically recognized as a SSD/flash.
I initially used 2 different USB-sticks: a "Trascend JF V30 16GB" and a "extremememory Supersonic 150/50x".
The "extremememory Supersonic 150/50x" was so bad just after the first test that I decided to ignore it (funny, being branded with such a name...). That's a pity because another "extremememory" USB-stick I bought at the beginning of 2009 was very fast, but it is too small (4GB) to hold my distribution (~2GB) plus a swap-partition (1GB) plus some additional space to grow and/or temporary files.
The tests were performed on my notebook:
The btrfs-utils I used were version 0.19.
No other activities/tasks running while performing the tests.
Results
| Options/Filesystem | ext2 | ext3 | ext4 | btrfs | nilfs |
|---|---|---|---|---|---|
| Format options | default | default | default | default | default |
| Mount options | noatime | noatime | noatime | noatime,ssd | noatime |
| Tests/Filesystem | ext2 | ext3 | ext4 | btrfs | nilfs |
|---|---|---|---|---|---|
| Write 4GB file | 893 | 817 | 552 | 553 | 808 |
| Read 4GB file | 237 | 237 | 222 | 208 | 239 |
| Write small files | 7 | 18 | 12 | 10 | 4 |
| Overwrite small files | 6 | 4 | 5 | 19 | 4 |
On the Y-axis there is the execution time in seconds:
Looking at the results:
- As I mentioned previously I anyway excluded BtrFS because of its instability, but anyway as I can see it has big problems when overwriting small files. Therefore, no Btrfs.
- ext3, which I would anyway exclude because of the problems with the journal & flash-memory, has problems when writing small files, which is what the Gentoo distribution does extensively. So, no ext3.
- ext4 is strong when writing and reading big files, at least on flash-memory. It's for sure a good candidate to be used as storage e.g. for media files. But on the other side it is 3 times slower compared to nilfs2 when writing the directories and small files, so not good for e.g. taking backups of root-partitions or holding root partitions.
- ext2 is quite good too, but nilfs2 needed about 1 minute less when writing the big file and was almost double as fast when writing for the first time the small files.
In the end I chose nilfs2 to hold my root partition on my USB-stick.
I am using it since almost 2 months now and I didn't have any problem (yet).
Sometimes when the free space gets too low the garbage collector wakes up and starts hunting old checkpoints that can be deleted to free up space (this until the desired limit has been reached - check the config file options) and there it becomes slow, but as this happens rarely (using ~2.5GB out of the 16GB available) it doesn't bother me.
Problems
I used NILFS2 for my root filesystem and I came up to some interesting things:
- No clue why, but the root-fs always mounted read-only.
- One thing I did was to change grub's config to have "rw" for the root filesystem:
kernel /boot/kernel-2.6.32-gentoo-r7 rw root=/dev/sda3 doslowusb rootdelay=15 console=tty1- Then I set that as well in /etc/fstab:
/dev/sda3 / nilfs2 noatime,rw 0 0- After that one more thing I had to do was to change the code of /etc/init.d/checkroot. For reasons I did not understand it always remounted the root filesystem in read-only mode. I added 2 lines and commented the original test:
================ depend() { before * } start() { local retval=0 if [[ ! -f /fastboot && -z ${CDBOOT} ]] \ && ! is_net_fs / && ! is_union_fs / ; then MYCUSTOM=0 if [[ -z ${MYCUSTOM} ]] ; then # if touch -c / >& /dev/null ; then ebegin "Remounting root filesystem read-only" mount -n -o remount,ro / eend $? ...... ================Quite ugly, but it works :o)
- The behaviour of NILFS2 is set by the setup in the file /etc/nilfs_cleanerd.conf.
- The garbage-collector did not work for me out-of-the-box when using NILFS2 as root fs. I had to implement a small script that runs "killall nilfs_cleanerd" and "nilfs_cleanerd /dev/sda3" when the OS boots (in the last runlevel).
- NILFS2 doesn't have a filesystem-check utility (fsck) because (from what I understood) if the FS is corrupted it will just enable the youngest checkpoint which is marked as not corrupted. To avoid the message during boot about the fsck-utility not being found just change in /etc/fstab the settings not to perform a filesystem check - in my case:
/dev/sda3 / nilfs2 noatime,rw 0 0