Title seems a bit off, per caveat below...why not use the original title, "WinFsp Performance Testing" (?)
"The comparison to NTFS is very important to establish a baseline. It is also very misleading because NTFS is a disk file system and MEMFS (either the WinFsp or Dokany variants) is an in memory file system. The tests will show that MEMFS is faster than NTFS. This should not be taken to mean that we are trying to make the (obvious) claim that an in memory file system is faster than a disk file system, but to show that the approach of writing a file system in user mode is a valid proposition and can be efficient."
I am the author of WinFsp and I believe you are missing the point or you did not actually read the article. Early on I say that:
<<
This performance testing shows that WinFsp has excellent performance in all tested scenarios. It outperforms NTFS in most scenarios (an unfair comparison as NTFS is a disk file system and WinFsp is tested with an in-memory file system). It also outperforms Dokany in all scenarios, often by an order of magnitude.
>>
My intent was to measure performance of WinFsp against NTFS in order to establish a baseline and to show that building file systems in user mode is not as inefficient as people think.
You should also note that a hacker news submission is only 80 characters and it is hard to clearly communicate one's intentions fully.
I will change the title to say "Faster than Dokany" without mentioning NTFS.
The deception is not intentional. For me it was important to verify that my project is not slower than NTFS when using my test file system, which is an in-memory file system. You may notice that Dokany is actually slower or on par with NTFS in many scenarios even with an in memory file system.
Trying NTFS on a RAM disk is a good idea. I simply did not think of it.
There is no on disk implementation of WinFsp that I can share at this moment.
Your comments make a fair point. NTFS is in fact effectively an in-memory filesystem when you have caching, and not going out to disk to satisfy various requests.
It's a valid observation that we can have a filesystem that trombones out to user space to (say) fetch a block of data, and (contrary to what some might expect) it doesn't perform worse than an in-kernel one which gets it from a cache.
> NTFS is in fact effectively an in-memory filesystem when you have caching, and not going out to disk to satisfy various requests.
It is interesting to see that NTFS is doing extremely well in a few different tests that are entirely satisfied from the cache: the list directory operation and the cached read/write operations. OTOH it does not do as well in tests that could potentially be satisfied from cache (like file open tests).
In any case I intend to retry the tests using NTFS on a RAM disk when I get some time.
You say in your bench that Dokan support does not support memory mapped files but it does as I can see with fsx (memory mapped read/write enabled) running on their CI https://ci.appveyor.com/project/Maxhy/dokany/build/1.0.1-48/...
It is your implementation that doesn't support it.
And last point...running it on VirtualBox O_o ? Even running it 5x time and take the best time. You cannot be sure that virtualbox has not corrupt your test. For me a benchmark on a VM is not reliable at any point.
My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.
The WinFsp/MEMFS had indeed a problem which I fixed, but it was not WinFsp related (it was MEMFS related - aka my test user mode file system). I attempted to apply the same fix to Dokany/MEMFS, but I could not because Dokany does not pass an Offset parameter to its FindFiles operation.
> My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.
No. WinFsp is faster, because it is faster. Feel free to send me any pull requests that you want for Dokany/MEMFS and I will incorporate them into the project and re-run the tests.
> >Mapping = CreateFileMappingW(Handle, 0, PAGE_READWRITE, 0, FileSize, 0);
>
> So it is more a "case" not handled by dokan rather that all memory mapped not handled ? Is this made in purpose just to remove dokan from the bench ?
I do not understand this.
Are you familiar with how memory mapping works on Windows? CreateFileMappingW is it. If that API fails you do not have memory mapping.
Furthermore I challenge you to show me how Dokany/MEMFS makes this API fail. If you find such an issue I promise to fix it and re-run the tests.
>The lock exists on WinFsp/MEMFS. It is just taken in a different place:
It changes nothing on what I said. You added it on dokan side without reason and not on winfsp memfs.
I added it on Dokan side, because I needed to protect the Dokany/MEMFS data structures against corruption. On the WinFsp side this lock is taken automatically as a convenience (although one can remove it if the file system can cope by using a custom "operation guard" (FspFileSystemSetOperationGuard) -- it is not needed for WinFsp operation).
The exact same lock is taken on both sides. There is no unfair advantage to WinFsp. If you design a file system that takes no locks and works on both WinFsp and Dokany, I will be happy to run the tests against that file system.
> A pull request will not change that a real application on a non-virtual machine will not see the difference.
All the "real application" has to do is to open a file and issues some I/O. Just because Dokany does not cache I/O it will be 9x-10x slower than WinFsp.
> The exact same lock is taken on both sides. There is no unfair advantage to WinFsp. If you design a file system that takes no locks and works on both WinFsp and Dokany, I will be happy to run the tests against that file system.
Alternatively you can show me that there is a lock already taken within Dokany for this purpose (protection of the data structures of the user mode file system) and I will be happy to remove the lock and rerun the tests.
I do not see a big difference to the fsbench call. It may be that fsbench attempts to make a mapping that is 4MB long. Now you can fix that problem in Dokany ;)
OK since it is the author of Winfsp that has made memfs of dokan and winfsp and also the tools used to bench/compare them, I wanted to be sure to have the same result on my side using this tool.
For now, I have been able to make it work on NTFS but when using on winfsp/memfs-dokany it just crash ><
I feel very weird about this article and that I am losing my time here.
MEMFS limits how many files you can create (up to 1024 by default). Try running it with the parameter -n set to 100000, which will allow you to create up to 100000 files.
26 comments
[ 2.9 ms ] story [ 67.1 ms ] thread"The comparison to NTFS is very important to establish a baseline. It is also very misleading because NTFS is a disk file system and MEMFS (either the WinFsp or Dokany variants) is an in memory file system. The tests will show that MEMFS is faster than NTFS. This should not be taken to mean that we are trying to make the (obvious) claim that an in memory file system is faster than a disk file system, but to show that the approach of writing a file system in user mode is a valid proposition and can be efficient."
Please also note that I want to convey that my solution is faster than Dokany, which is a competitor solution.
https://github.com/billziss-gh/sshfs-win
Why did they not use a RAM Drive for the NTFS test like one of these https://en.wikipedia.org/wiki/List_of_RAM_drive_software#Non...
Are there any on-disk filesystem implementations based on WinFsp so a comparison can be made?
I count this as intentionally misleading, since the submitter is also the Project author and performed the benchmark and should know better.
<< This performance testing shows that WinFsp has excellent performance in all tested scenarios. It outperforms NTFS in most scenarios (an unfair comparison as NTFS is a disk file system and WinFsp is tested with an in-memory file system). It also outperforms Dokany in all scenarios, often by an order of magnitude. >>
My intent was to measure performance of WinFsp against NTFS in order to establish a baseline and to show that building file systems in user mode is not as inefficient as people think.
You should also note that a hacker news submission is only 80 characters and it is hard to clearly communicate one's intentions fully.
It's a cool project, and this deception is entirely unnecessary.
Why not claim "FASTER than Dokany".
An interesting comparison would be ntfs-3g on WinFst vs. native NTFS either both on RAM drive or Disk.
You also did not answer any of the questions in my comment.
The deception is not intentional. For me it was important to verify that my project is not slower than NTFS when using my test file system, which is an in-memory file system. You may notice that Dokany is actually slower or on par with NTFS in many scenarios even with an in memory file system.
Trying NTFS on a RAM disk is a good idea. I simply did not think of it.
There is no on disk implementation of WinFsp that I can share at this moment.
Please let me know if you have other questions.
It's a valid observation that we can have a filesystem that trombones out to user space to (say) fetch a block of data, and (contrary to what some might expect) it doesn't perform worse than an in-kernel one which gets it from a cache.
It is interesting to see that NTFS is doing extremely well in a few different tests that are entirely satisfied from the cache: the list directory operation and the cached read/write operations. OTOH it does not do as well in tests that could potentially be satisfied from cache (like file open tests).
In any case I intend to retry the tests using NTFS on a RAM disk when I get some time.
You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1... After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a... but not on dokan ?
You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620...
And last point...running it on VirtualBox O_o ? Even running it 5x time and take the best time. You cannot be sure that virtualbox has not corrupt your test. For me a benchmark on a VM is not reliable at any point.
My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.
It was me that ported FSX to windows so I am quite familiar with this test.
Here is the code that fails in fsbench:
I would be happy if you explained to me how Dokany/MEMFS fails this code.> You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1....
The lock exists on WinFsp/MEMFS. It is just taken in a different place:
> After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a.... but not on dokan ?The WinFsp/MEMFS had indeed a problem which I fixed, but it was not WinFsp related (it was MEMFS related - aka my test user mode file system). I attempted to apply the same fix to Dokany/MEMFS, but I could not because Dokany does not pass an Offset parameter to its FindFiles operation.
> You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620....
Actually that is incorrect.
> My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.No. WinFsp is faster, because it is faster. Feel free to send me any pull requests that you want for Dokany/MEMFS and I will incorporate them into the project and re-run the tests.
So it is more a "case" not handled by dokan rather that all memory mapped not handled ? Is this made in purpose just to remove dokan from the bench ?
>The lock exists on WinFsp/MEMFS. It is just taken in a different place:
It changes nothing on what I said. You added it on dokan side without reason and not on winfsp memfs.
>send me any pull requests that you want for Dokany/MEMFS
A pull request will not change that a real application on a non-virtual machine will not see the difference.
I do not understand this.
Are you familiar with how memory mapping works on Windows? CreateFileMappingW is it. If that API fails you do not have memory mapping.
Furthermore I challenge you to show me how Dokany/MEMFS makes this API fail. If you find such an issue I promise to fix it and re-run the tests.
>The lock exists on WinFsp/MEMFS. It is just taken in a different place: It changes nothing on what I said. You added it on dokan side without reason and not on winfsp memfs.
I added it on Dokan side, because I needed to protect the Dokany/MEMFS data structures against corruption. On the WinFsp side this lock is taken automatically as a convenience (although one can remove it if the file system can cope by using a custom "operation guard" (FspFileSystemSetOperationGuard) -- it is not needed for WinFsp operation).
The exact same lock is taken on both sides. There is no unfair advantage to WinFsp. If you design a file system that takes no locks and works on both WinFsp and Dokany, I will be happy to run the tests against that file system.
> A pull request will not change that a real application on a non-virtual machine will not see the difference.
All the "real application" has to do is to open a file and issues some I/O. Just because Dokany does not cache I/O it will be 9x-10x slower than WinFsp.
Alternatively you can show me that there is a lock already taken within Dokany for this purpose (protection of the data structures of the user mode file system) and I will be happy to remove the lock and rerun the tests.
https://github.com/billziss-gh/secfs.test/blob/60f9f44b87731...
I do not see a big difference to the fsbench call. It may be that fsbench attempts to make a mapping that is 4MB long. Now you can fix that problem in Dokany ;)
Examples command lines:
WinFsp: memfs -t 0 -n 100000 -m Z: # no caching memfs -t 1 -n 100000 -m Z: # enables metadata caching memfs -t -1 -n 100000 -m Z: # enables data and metadata caching
Dokany: memfs-dokany -n 100000 -m Z:
Please read about the WinFsp Testing Strategy here:
https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing