UNSOLVED

paviles1

updated

16 years ago

P

paviles1

11 Posts

0

1413

October 24th, 2010 12:00

Celerra Nic Teaming

Can the Celerra team the nics used for NFS?  We are planing on an implementation in which the sole purpose for the Celerra should be only to support a set of 4 ESX servers providing NFS storage to VMware.

Each ESX has a set dual nic cards dedicated for NFS, each connected to a dedicated swtich, so ESX NIC 1 connectes to Port 1 on SW1 and ESX NIC 2 connects to Port 2 on SW2 for example and then Port 1 on SW1 and Port 1 on SW 2 are combined into a logical port.

My issue is I dont see how the Celerra can team the data movers for link agregation, if so how does it scale?

I find it hard to see how it can provide better performance than other systems that have dual nics and allow for link agreegation and failover.

If so, a single 1 GB card it will be around 100 MB/s in max (sure will never get there), but even so, it will only take someone to dump an DVD ISO image or CD ISO image to one of the Virtual Machines acting as a file server for it to destroy any performance on the unit.

Is this the case? I believe NetApp does allow for the "data movers" to be clustered.

Any help is appreciated.

Regards,

Paul Aviles

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    743

    0

    Posted October 24th, 2010 12:00

    and multiple NFS mounts

    for ONE mount you will not get more than one interface worth of bandwidth from trunking no matter which vendor

    think of it as a multi-lane highway - if you. are alone and only have one truck it doesn't matter how many lanes there are

    if you have multiple trucks you can get more goods transported at the same time if you use multiple lanes

  • paviles1

    11 Posts

    743

    0

    Posted October 24th, 2010 12:00

    Rainer,

    Thanks for responding.

    I take from your answer that there is no way to do it with the Celerra. Your suggestion of only one truck is the reason for my question, that never happens.

    What multiple IP address or subnet has to do if there is no way to team the DM?  I can assign 50 IP address but the bandwidth will be the same.

    As my original post, the only client is the ESX servers, nothing else.

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    743

    0

    Posted October 24th, 2010 12:00

    one file system is accessible over one data mover only (same as for NetApp)

    you can use link aggregation of network interfaces within a data mover

    in order to get more then one interface worth of performance you need to understand how the trunking / load-balancing works (see the Celerra and switch manuals)

    in general if you have only one client you need to use multiple IP's or even subnets on both client and server

    If you don't want to go into that trouble then just use 10Gig Ethernet

    the VMware with Celerra TechBook available from Powerlink is also a good read

    Rainer

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    743

    0

    Posted October 24th, 2010 12:00

    look at the way trunking works

    for one file system you have typically only one mount with packets going to one IP address that has one MAC address going to one port (NFS 2048)

    the switch then decides based on a hash of either IP / MAC address or port number which interface in the trunk to put the packet on (there is no round-robin or random there)

    In your case it will always be the same interface

    Similar for the way the response packet from the server to the client takes

    So if you have multiple mounts - meaning multiple NFS data stores then in total you can use multiple interfaces worth of traffic

    with multiple ESX servers theoretically you could use different mounts and spread the load this way but I haven't tested if / how VMware allows you to configure that

    if you set the policy to use the port then other traffic that uses a different TCP port than NFS will also use different links

    Rainer

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    264

    0

    Posted October 24th, 2010 13:00

    just to be clear - this is NOT a Celerra limitation

    it's the same for any other vendors device

    we have actually implemented a reflect mode that send a response out the same interface that the request comes in

    in you scenario with only one mount it doesn't help since your client will only ever send it to one interface

    it's the way the trunking standards work on the switch side and it has good reasons

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    743

    0

    Posted October 24th, 2010 13:00

    do you mean multiple active data movers ?

    one file system is only mounted on one data mover read/write

    as far as utilizing multiple NICs within one data mover the Celerra works just like your Linux NFS server - just try it and see what results you get

    one client only uses one session and switches do not spread a single session over multiple links. if you do find one that does than it's fine with us and the Celerra will respond accordingly

  • paviles1

    11 Posts

    743

    0

    Posted October 24th, 2010 13:00

    Rainer, I dont follow you. I can today take any Unix/Linux distro converted into an NFS server and agreegate NIC cards on the server and also on the ESX side.

    >for one file system you have typically only one mount with packets going to one IP address that has one MAC address going to one port (NFS 2048)

    Correct for a single nic, if you bind two, you have created a new MAC for the bundled nics with a different IP address than the two separate nics and depending on how you group them you have failover and link agreegation. That is how teaming works....

    So forget about multiple ESX. Lets address if you could a single ESX server having 4 NICs. I create two separate vkernels for storage and have each one with 2 nics. I setup the switch with LACP for each set par of NICs. So on my client I have two separate "paths" if you will, each one with 2 nics connected to two separate switches.

    What do I do on the Celerra now to associate two DM?s as link agreegation?

    Otherwise everything else I just did and close to $80K become pointless now and I still have one single 1 GB on the DM side. Either I must be explaining this wrong or there is a big issue on scalability with the DM on my view.

    Regards,

    Paul

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    743

    0

    Posted October 24th, 2010 15:00

    Correct for a single nic, if you bind two, you have created a new MAC for the bundled nics with a different IP address than the two separate nics and depending on how you group them you have failover and link agreegation. That is how teaming works....

    correct

    just think aobut HOW your teaming driver or switch decides which path gets taken for which paket

    So forget about multiple ESX. Lets address if you could a single ESX server having 4 NICs. I create two separate vkernels for storage and have each one with 2 nics. I setup the switch with LACP for each set par of NICs. So on my client I have two separate "paths" if you will, each one with 2 nics connected to two separate switches.

    ok - if you can make the ESX server use *both* paths / vmkernel ports for outgoing traffic to one NFS mount then would be fine and you get load distribution that way.

    If these packet go to two different IP/Mac adresses on the NFS server or they have different source IP/Mac adresses then would be fine for the return path as well

  • DanJost

    190 Posts

    743

    0

    Posted October 25th, 2010 07:00

    "I can today take any Unix/Linux distro converted into an NFS server and agreegate NIC cards on the server and also on the ESX side."

    I think you don't understand how NIC teaming or LACP/etherchannel works.  There is no difference between creating an LACP trunk on your other "Unix/Linux distro " NFS server and on the Celerra.  You are still limited (for a single stream) to one network path ... in other words, 1Gbit assuming that is your link speed.  The switch participates in the load balancing with LACP/etherchannel.  Different switches have different capabilities (IP, MAC, TCP Port, etc) to determine which path to take.  If you are spreading your paths across different switches (non-stacked, non-chassis) you can't use LACP.  The teaming software on your Unix/Linux distro MIGHT be able to do transmit load balancing but when done across independent switches the switches will cry "foul" because the same MAC address will show up in different places on your L2 network.

    "Either I must be explaining this wrong or there is a big issue on scalability with the DM on my view."

    This is actually a "problem" with NFS and ethernet-based link aggregation.  For ESX without PowerPath/VE, you get the same effect with fiber channel in most instances (though you would generally have at least a 4Gbit connection that is storage optimized so it isn't that big of a problem).  With iSCSI and FC, you can get better load balancing for the storage connection over NFS.  10Gbit ethernet changes the rules because of the huge pipe it provides - similar to how 4Gbit FC let's you get by without load balancing (in the case of ESX and PowerPath/VE).

    I don't use iSCSI - perhaps someone else can offer some insights into the differences between that and NFS with an ESX implementation.

    Dan

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    580

    0

    Posted October 25th, 2010 08:00

    Well – NFS + TCP/IP isnt Fibre Channel

    TCP/IP was designed for large networks with hops of different speeds that also can loose packets.

    In such an environment the idea of distributing packets for one session over different routes would be bad since it leads to out-of-order packets and ties up I/O buffer space on clients and network equipment

    Fibre Channel on the other hand was designed for limited size storage networks where the links are the same speed, switches are non-blocking and don’t have to do fragmentation and reassembly

    So doing round-robin there isn’t a problem

    Sure you could change the NFS standard a bit and write a NFS client that uses multiple sessions for one mount similar to the ISCSI concepts – but currently NFS (and CIFS) do one session