Tsky

updated

12 years ago

T

Tsky

1 Rookie

•

32 Posts

0

5518

April 13th, 2015 08:00

What is the Diffrence between Native NFS v.s NFS on OneFS ?

Hi All,

I've started to investigate an issue that is very strange , and until now no one had an solution for this.

Maybe some of the NFS Gurus can help here.

Client : CentOS 6.6 , 64Bit (Everything works NFSv3)

NFS Native Server Export : Export comes from a Linux Server with 2 x local drives SAS15K on it .

NFS OneFS Server Export : Export comes from a OneFS isilon ,S-200 with data on SSD.


Mount Exports are same on both exports from the client side.

Disabled Sync writes on Isilon.


The Scenario:

  • Copy From Client to Native NFS Server - 0m35s
  • Copy From Client to Isilon NFS Server - 1m55s

Is there any explanation of Why NFS Protocol will act this way on Isilon?

Remarks:  TcpDumps show that when doing the same action via Isilon it requires Double amounts of Request/Ansers 
                   Between Client <-> Isilon

No diffrence via MTU/Not diffrence of Network .... Same scenario diffrent target




  • kipcranford

    125 Posts

    3032

    0

    Posted April 13th, 2015 08:00

    > Mount Exports are same on both exports from the client side.


    Can you please post the mount string/options here?


    > Disabled Sync writes on Isilon.


    Please describe how you did this in OneFS.


    > Copy From Client to Isilon NFS Server - 1m55s


    Please post here the way you did the copy (e.g. "cp", "dd", etc), as well as the size of the file you used.


    > TcpDumps show that when doing the same action via Isilon it requires Double amounts of Request/Ansers


    Maybe a block size difference.  Are the request sizes the same between the two copies?

  • Tsky

    1 Rookie

    •

    32 Posts

    3034

    0

    Posted April 13th, 2015 09:00

    Mount Exports:

    kipcranford wrote:

    > Mount Exports are same on both exports from the client side.


    Can you please post the mount string/options here?


    > Disabled Sync writes on Isilon.


    Please describe how you did this in OneFS.


    > Copy From Client to Isilon NFS Server - 1m55s


    Please post here the way you did the copy (e.g. "cp", "dd", etc), as well as the size of the file you used.


    > TcpDumps show that when doing the same action via Isilon it requires Double amounts of Request/Ansers


    Maybe a block size difference.  Are the request sizes the same between the two copies?

    rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.30.7,mountvers=3,mountport=300,mountproto=tcp,local_lock=none,addr=10.1.30.7 0 0

    Disabled On Isilon:

    isi nfs  settings export modify --setattr-asynchronous=yes

    How Copy was made:

    Untar of Linux.tar file  , 700MB , 50K files objects

    TcpDump :

    The block size are the same.

  • 3034

    2

    Posted April 13th, 2015 10:00

    You haven't discovered anything strange, nor have you discovered anything unexpected.

    The Isilon OneFS architecture has tradeoffs, as does every other platform.  One of the tradeoffs of a scale-out architecture is that creates and deletes will be slow.  Other operations will be fast.  The tradeoff you've made in your test is that you've thrown most of the redundancy out the window and provided no way for your environment to scale.  What happens when you have a lot of clients hitting your client?  Will it scale linearly?  Until what point?

    In our environment, what matters the most is wall-clock time of the entire application spread across hundreds of servers.  This is a mixture of a large number of metadata operations and many reads and writes.  The Isilon wins.  I've measured 1,000,000 NFS operations per second load-balanced across 28 S200 nodes.  I hit 200-300K NFS operations per second just about all day, every day.  This is getting mission critical production workloads done.  I regularly provide 80Gbps out of my Isilon cluster to my clients and it rarely hits below 20Gbps (24x7).

    Taking a single tarball and extracting it and saying you've "won" the benchmark is sadly misguided.  If that's the operation you do every day, and that's *all* you do, then yes, the Isilon will probably lose and you've bought the wrong platform.  On the other hand, if you actually want to do something with that tarball, and provide redundancy if your single Linux system fails, and scale as you grow, then you may have a competition on your hands.

    In my Isilon environment, I can lose 2 ENTIRE Isilon nodes - and all of their combined 48 spindles and SSDs - and not lose access to data.  The users may see a small pause as the IP addresses fail over, but the data is still there, along with access to that data.  In your case, you can have a single raid controller or chunk of memory go bad the wrong way and potentially lose the data.  If you need to upgrade the kernel, you'll lose access to the data.   If you need to restart your NFS server for any reason, you'll lose access to the data.  I can lose multiple drives and not lose access to the data, nor run out of redundancy while those drives are being repaired.  Imagine your single server having a drive fail.  Are you 100% sure you can rebuild a failed drive in a timely manner before the 2nd drive fails?  Are you willing to bet your job on it?   Does your little Linux server provide snapshots for data recovery?  Off-node replication at the block level?  Without significantly impacting performance?

    We had a DDN storage array here for our workloads a couple of years ago.  It's designed for high performance computing environments in supercomputer centers.  The Isilon kicked its butt out the door.

  • kipcranford

    125 Posts

    3034

    1

    Posted April 13th, 2015 12:00

    > When stated that there is no difference in the packetsize i would deepdive into the trace and take a look on Fails or something. There has to be an explanation for these.

    Indeed.  Using my git repository, I took packet captures when cloning to a Linux NFS server and to an Isilon NFS server.  The number of NFS packets was 30404 and 30406, respectively.  Both NFS servers were pretty close to "out of the box" in terms of configurations.

    So something does seem a bit strange...

  • Tsky

    1 Rookie

    •

    32 Posts

    3034

    0

    Posted April 13th, 2015 12:00

    Hi,

    Its much appreciated the informative response and of course i agree with you on all aspects regading Scale,Performance etc.

    The Question i ask is a very narrow scenario therfore i'm discarding any other "REAL" advantages and focusing

    on this "Scenario" .

    I just wanted to take the advanteages you mentioned out of the equation.

  • Tsky

    1 Rookie

    •

    32 Posts

    3034

    0

    Posted April 13th, 2015 12:00

    Sluetze,

    Thanks and after 12 Hours of DeepDive and no retransmisions/Failure its a Mistory

  • sluetze

    2 Intern

    •

    300 Posts

    3034

    0

    Posted April 13th, 2015 12:00

    ed,

    what you describe would expain higher latency / servertime needed to execute the commands provided by the client. And of course you are right at this point.

    But for me there is no explanation about the

    Since he has a double amount of requests / answers

    part.

    When stated that there is no difference in the packetsize i would deepdive into the trace and take a look on Fails or something. There has to be an explanation for these.

    But I do not have a lot of experience with NFS so im out

    Best Regards

    sluetze

  • Tsky

    1 Rookie

    •

    32 Posts

    3034

    0

    Posted April 13th, 2015 13:00

    I'll try to check if the TCP Capture is diffrent again altough i'm now counting on this to be the answer of the

    global behavior .

    I'll try to check it again and update regarding it ,

    other than that thank you for the answers you provided me .

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    3034

    0

    Posted April 14th, 2015 09:00

    Much simpler: Try unpacking the test tar file locally on the Linux server and on the Isilon.

    The difference comes mostly from the file system rather than from the NFS protocol or other network parameters...

    Linux filesystems create files seemingly at blazing speed BUT they lie to the user/applications -- "thanks" to write caching  in RAM(!) Linux reports successful file creations and data writes back to the applications LONG BEFORE the files will be safely written to disk...

    Isilon caches metadata and data writes in non-volatile memory (NVRAM) on multiple nodes, well protected and still pretty fast, but can't beat local RAM.

    -- Peter

  • kipcranford

    125 Posts

    1138

    0

    Posted April 14th, 2015 09:00

    > Isilon caches metadata and data writes in non-volatile memory (NVRAM) on multiple nodes, well protected and still pretty fast, but can't beat local RAM.

    While this is true, OneFS will still use its write cache too, depending on the filesystem configuration (the default is to use it).  So writes are actually coalesced in volatile RAM before they are flushed to the journal, which is non-volatile and considered stable storage as far as OneFS is concerned.

    This is also true when NFS is being used.  Client writes by default will use the OneFS write cache, unless disabled on the Isilon, via a 'sync' mount option, or via an O_DIRECT or O_SYNC open flag in the client application.