You haven't discovered anything strange, nor have you discovered anything unexpected.
The Isilon OneFS architecture has tradeoffs, as does every other platform. One of the tradeoffs of a scale-out architecture is that creates and deletes will be slow. Other operations will be fast. The tradeoff you've made in your test is that you've thrown most of the redundancy out the window and provided no way for your environment to scale. What happens when you have a lot of clients hitting your client? Will it scale linearly? Until what point?
In our environment, what matters the most is wall-clock time of the entire application spread across hundreds of servers. This is a mixture of a large number of metadata operations and many reads and writes. The Isilon wins. I've measured 1,000,000 NFS operations per second load-balanced across 28 S200 nodes. I hit 200-300K NFS operations per second just about all day, every day. This is getting mission critical production workloads done. I regularly provide 80Gbps out of my Isilon cluster to my clients and it rarely hits below 20Gbps (24x7).
Taking a single tarball and extracting it and saying you've "won" the benchmark is sadly misguided. If that's the operation you do every day, and that's *all* you do, then yes, the Isilon will probably lose and you've bought the wrong platform. On the other hand, if you actually want to do something with that tarball, and provide redundancy if your single Linux system fails, and scale as you grow, then you may have a competition on your hands.
In my Isilon environment, I can lose 2 ENTIRE Isilon nodes - and all of their combined 48 spindles and SSDs - and not lose access to data. The users may see a small pause as the IP addresses fail over, but the data is still there, along with access to that data. In your case, you can have a single raid controller or chunk of memory go bad the wrong way and potentially lose the data. If you need to upgrade the kernel, you'll lose access to the data. If you need to restart your NFS server for any reason, you'll lose access to the data. I can lose multiple drives and not lose access to the data, nor run out of redundancy while those drives are being repaired. Imagine your single server having a drive fail. Are you 100% sure you can rebuild a failed drive in a timely manner before the 2nd drive fails? Are you willing to bet your job on it? Does your little Linux server provide snapshots for data recovery? Off-node replication at the block level? Without significantly impacting performance?
We had a DDN storage array here for our workloads a couple of years ago. It's designed for high performance computing environments in supercomputer centers. The Isilon kicked its butt out the door.
> When stated that there is no difference in the packetsize i would deepdive into the trace and take a look on Fails or something. There has to be an explanation for these.
Indeed. Using my git repository, I took packet captures when cloning to a Linux NFS server and to an Isilon NFS server. The number of NFS packets was 30404 and 30406, respectively. Both NFS servers were pretty close to "out of the box" in terms of configurations.
what you describe would expain higher latency / servertime needed to execute the commands provided by the client. And of course you are right at this point.
But for me there is no explanation about the
Since he has a double amount of requests / answers
part.
When stated that there is no difference in the packetsize i would deepdive into the trace and take a look on Fails or something. There has to be an explanation for these.
But I do not have a lot of experience with NFS so im out
Much simpler: Try unpacking the test tar file locally on the Linux server and on the Isilon.
The difference comes mostly from the file system rather than from the NFS protocol or other network parameters...
Linux filesystems create files seemingly at blazing speed BUT they lie to the user/applications -- "thanks" to write caching in RAM(!) Linux reports successful file creations and data writes back to the applications LONG BEFORE the files will be safely written to disk...
Isilon caches metadata and data writes in non-volatile memory (NVRAM) on multiple nodes, well protected and still pretty fast, but can't beat local RAM.
> Isilon caches metadata and data writes in non-volatile memory (NVRAM) on multiple nodes, well protected and still pretty fast, but can't beat local RAM.
While this is true, OneFS will still use its write cache too, depending on the filesystem configuration (the default is to use it). So writes are actually coalesced in volatile RAM before they are flushed to the journal, which is non-volatile and considered stable storage as far as OneFS is concerned.
This is also true when NFS is being used. Client writes by default will use the OneFS write cache, unless disabled on the Isilon, via a 'sync' mount option, or via an O_DIRECT or O_SYNC open flag in the client application.
kipcranford
125 Posts
3032
0
Posted April 13th, 2015 08:00
> Mount Exports are same on both exports from the client side.
Can you please post the mount string/options here?
> Disabled Sync writes on Isilon.
Please describe how you did this in OneFS.
> Copy From Client to Isilon NFS Server - 1m55s
Please post here the way you did the copy (e.g. "cp", "dd", etc), as well as the size of the file you used.
> TcpDumps show that when doing the same action via Isilon it requires Double amounts of Request/Ansers
Maybe a block size difference. Are the request sizes the same between the two copies?