Can the Celerra team the nics used for NFS? We are planing on an implementation in which the sole purpose for the Celerra should be only to support a set of 4 ESX servers providing NFS storage to VMware.
Each ESX has a set dual nic cards dedicated for NFS, each connected to a dedicated swtich, so ESX NIC 1 connectes to Port 1 on SW1 and ESX NIC 2 connects to Port 2 on SW2 for example and then Port 1 on SW1 and Port 1 on SW 2 are combined into a logical port.
My issue is I dont see how the Celerra can team the data movers for link agregation, if so how does it scale?
I find it hard to see how it can provide better performance than other systems that have dual nics and allow for link agreegation and failover.
If so, a single 1 GB card it will be around 100 MB/s in max (sure will never get there), but even so, it will only take someone to dump an DVD ISO image or CD ISO image to one of the Virtual Machines acting as a file server for it to destroy any performance on the unit.
Is this the case? I believe NetApp does allow for the "data movers" to be clustered.
I take from your answer that there is no way to do it with the Celerra. Your suggestion of only one truck is the reason for my question, that never happens.
What multiple IP address or subnet has to do if there is no way to team the DM? I can assign 50 IP address but the bandwidth will be the same.
As my original post, the only client is the ESX servers, nothing else.
one file system is accessible over one data mover only (same as for NetApp)
you can use link aggregation of network interfaces within a data mover
in order to get more then one interface worth of performance you need to understand how the trunking / load-balancing works (see the Celerra and switch manuals)
in general if you have only one client you need to use multiple IP's or even subnets on both client and server
If you don't want to go into that trouble then just use 10Gig Ethernet
the VMware with Celerra TechBook available from Powerlink is also a good read
for one file system you have typically only one mount with packets going to one IP address that has one MAC address going to one port (NFS 2048)
the switch then decides based on a hash of either IP / MAC address or port number which interface in the trunk to put the packet on (there is no round-robin or random there)
In your case it will always be the same interface
Similar for the way the response packet from the server to the client takes
So if you have multiple mounts - meaning multiple NFS data stores then in total you can use multiple interfaces worth of traffic
with multiple ESX servers theoretically you could use different mounts and spread the load this way but I haven't tested if / how VMware allows you to configure that
if you set the policy to use the port then other traffic that uses a different TCP port than NFS will also use different links
one file system is only mounted on one data mover read/write
as far as utilizing multiple NICs within one data mover the Celerra works just like your Linux NFS server - just try it and see what results you get
one client only uses one session and switches do not spread a single session over multiple links. if you do find one that does than it's fine with us and the Celerra will respond accordingly
Rainer, I dont follow you. I can today take any Unix/Linux distro converted into an NFS server and agreegate NIC cards on the server and also on the ESX side.
>for one file system you have typically only one mount with packets going to one IP address that has one MAC address going to one port (NFS 2048)
Correct for a single nic, if you bind two, you have created a new MAC for the bundled nics with a different IP address than the two separate nics and depending on how you group them you have failover and link agreegation. That is how teaming works....
So forget about multiple ESX. Lets address if you could a single ESX server having 4 NICs. I create two separate vkernels for storage and have each one with 2 nics. I setup the switch with LACP for each set par of NICs. So on my client I have two separate "paths" if you will, each one with 2 nics connected to two separate switches.
What do I do on the Celerra now to associate two DM?s as link agreegation?
Otherwise everything else I just did and close to $80K become pointless now and I still have one single 1 GB on the DM side. Either I must be explaining this wrong or there is a big issue on scalability with the DM on my view.
Correct for a single nic, if you bind two, you have created a new MAC for the bundled nics with a different IP address than the two separate nics and depending on how you group them you have failover and link agreegation. That is how teaming works....
correct
just think aobut HOW your teaming driver or switch decides which path gets taken for which paket
So forget about multiple ESX. Lets address if you could a single ESX server having 4 NICs. I create two separate vkernels for storage and have each one with 2 nics. I setup the switch with LACP for each set par of NICs. So on my client I have two separate "paths" if you will, each one with 2 nics connected to two separate switches.
ok - if you can make the ESX server use *both* paths / vmkernel ports for outgoing traffic to one NFS mount then would be fine and you get load distribution that way.
If these packet go to two different IP/Mac adresses on the NFS server or they have different source IP/Mac adresses then would be fine for the return path as well
"I can today take any Unix/Linux distro converted into an NFS server and agreegate NIC cards on the server and also on the ESX side."
I think you don't understand how NIC teaming or LACP/etherchannel works. There is no difference between creating an LACP trunk on your other "Unix/Linux distro " NFS server and on the Celerra. You are still limited (for a single stream) to one network path ... in other words, 1Gbit assuming that is your link speed. The switch participates in the load balancing with LACP/etherchannel. Different switches have different capabilities (IP, MAC, TCP Port, etc) to determine which path to take. If you are spreading your paths across different switches (non-stacked, non-chassis) you can't use LACP. The teaming software on your Unix/Linux distro MIGHT be able to do transmit load balancing but when done across independent switches the switches will cry "foul" because the same MAC address will show up in different places on your L2 network.
"Either I must be explaining this wrong or there is a big issue on scalability with the DM on my view."
This is actually a "problem" with NFS and ethernet-based link aggregation. For ESX without PowerPath/VE, you get the same effect with fiber channel in most instances (though you would generally have at least a 4Gbit connection that is storage optimized so it isn't that big of a problem). With iSCSI and FC, you can get better load balancing for the storage connection over NFS. 10Gbit ethernet changes the rules because of the huge pipe it provides - similar to how 4Gbit FC let's you get by without load balancing (in the case of ESX and PowerPath/VE).
I don't use iSCSI - perhaps someone else can offer some insights into the differences between that and NFS with an ESX implementation.
TCP/IP was designed for large networks with hops of different speeds that also can loose packets.
In such an environment the idea of distributing packets for one session over different routes would be bad since it leads to out-of-order packets and ties up I/O buffer space on clients and network equipment
Fibre Channel on the other hand was designed for limited size storage networks where the links are the same speed, switches are non-blocking and don’t have to do fragmentation and reassembly
So doing round-robin there isn’t a problem
Sure you could change the NFS standard a bit and write a NFS client that uses multiple sessions for one mount similar to the ISCSI concepts – but currently NFS (and CIFS) do one session
Rainer_EMC
6 Operator
•
8645 Posts
743
0
Posted October 24th, 2010 12:00
and multiple NFS mounts
for ONE mount you will not get more than one interface worth of bandwidth from trunking no matter which vendor
think of it as a multi-lane highway - if you. are alone and only have one truck it doesn't matter how many lanes there are
if you have multiple trucks you can get more goods transported at the same time if you use multiple lanes