UNSOLVED

SharatVTG

updated

15 years ago

S

SharatVTG

2 Posts

0

7520

July 5th, 2011 09:00

VNXe3100 ethernet ports & SP ports link aggregation and communication between them.?

Hi,

How to configure ethernet ports on VNXe3100 with dual SPs and configure the ports for link aggregation. When you create cifs server, you are assigning to SP eth2 or eth3 or both. How does the communication between ethernet port and the SP iscsi eth2, eth3 related? Can you provide the link or doc which has explanation of cifs/nfs server and ethernet ports.

Thanks

  • Kumar_A

    2 Intern

    •

    727 Posts

    1035

    0

    Posted July 5th, 2011 14:00

    There are two places where you would need to do configuration settings:

    1. At the VNXe storage system level - You can configure link aggregation from Settings -> More Configuration -> Advanced Configuration page in Unisphere. You can combine eth2 and eth3 ports from the VNXe system.

    2. You also need to configure the switch ports to make sure that link aggregation is enabled at the switch ports.

  • SharatVTG

    2 Posts

    1035

    0

    Posted July 6th, 2011 08:00

    Thanks for the reply.  Actually, I was referring to the 4 (1 Gbps) copper ethernet ports communication with iSCSI SP ports 1Gbps. When you create shared folder server, we assign SP eth2, and/or eth3 ports. How do you create aggregation on the 4 lan ports and IP assignments. There is not much VNXe documentation available except for hardware. Please provide me the link if you are aware of any.

    Thanks very much.

  • rajnvr1

    57 Posts

    1035

    0

    Posted July 15th, 2011 07:00

    The best practice that we can suggest for VNXe with 2 ports on each SP is to create a link aggregation from SPA Ports and SPB ports.

    I have an image attached to the post for an easier understanding.

    The Right side diagram is when both the SP’s are working.

    1. Create a LACP Group from all the ports on VNXe Side.
    2. Create a LACP Group 1 from port 5 and 7 and LACP Group 2 from port 6 and 8

    By this we get switch, SP, Port, and Host Level Redundancy .As this is a LACP trunk, performance will be good.

    The Left Side diagram is when SPB Fails.

    1. All the Interfaces move to SPA (the IP addresses)
    2. All the Services move to SPA(CIFS, NFS, ISCSI and Replication)
    3. SPA will be serving all the Data till SPB comes back up.
    4. In this state Port 6 and 8 will be down.

    Note: For the failover to be accurate and fast, the network convergence time should be very less (4sec).

    1 Attachment

  • tks11

    1 Message

    1035

    0

    Posted August 2nd, 2011 20:00

    Rohan,

    This is just the type of diagram I've been looking for. Do you think you could help me fill it in with more detail and sample IP addresses, so I can better understand how to set up our environment?

    Thanks.

    -Daniel

  • 1035

    0

    Posted August 3rd, 2011 23:00

    The diagram is very useful for explaining what happens if there is an SP failure/failover however it doesn't explain what happens if an Ethernet switch fails. I assume the peer port failover would not be possible as there would be a simultaneous failure of one port on each SP. Peer port failover of an individual port on an SP would not work if they were aggregated because it would result in ports connected to different port groups on the remaining Ethernet switch. Can someone please explain what would happen if there were an Ethernet switch failure with the depicted configuration?

    Thanks,

    Cameron

  • ghardy123

    6 Posts

    1035

    0

    Posted August 4th, 2011 09:00

    If your ESX hosts are setup as VMware recommends you would have traffic paths:

    ESXA

         iSCSI subnetA traffic passes to SwitchA through link 1 (ESXB through link2)

         iSCSI subnetB traffic passes to SwitchB through link 3 (ESXB through link4)

    SPA iSCSI server eth2 (just using eth2 and 3 for the ports drawn on the document) runs server-1 traffic to subnetA (link5)

    SPA iSCSI server eth3 runs server-1 traffic for subnetB (link7)

    SPB iSCSI server eth2 runs server-2 traffic to subnetA (link6)

    SPB iSCSI server eth3 runs server-2 traffic to subnetB (link8)

    ESX servers see 4 paths to the SAN

         ESXA: iSCSI subnetA portgroup can use paths 1-5 and 1-6 (subnetA has contact with both SPs and their storage), iSCSI subnetB portgroup can use paths 3-7 and 3-8.  This way ESXA can contact LUN1 through either 1-5 or 3-7.  On failover of SPA, ESXA contacts LUN1 through 1-6 or 3-8.  If SwitchA dies then ESXA can contact LUN1 through 3-7.

    After reading this, I'm sure it is clear as mud so let me know if I need to clarify anything.

    -George

  • 1035

    0

    Posted August 9th, 2011 23:00

    Thanks for the response George. The configuration in Rohan’s diagram is using LACP (more suited to CIFS or NFS) rather than separate IP addresses as you have suggested would be best for VMware. My concern is over what happens when an Ethernet Switch fails when using LACP.

    In the diagram, one of two configurations can be assumed because the ports haven’t been labelled.

    - Config option 1: SPA eth2 is connected to port 5, SPA eth3 is connected to port 7, SPB eth2 is connected to port 8 and SPB eth3 is connected to port 6 (i.e. both eth2 ports connected to the same switch)

    - Config option 2: SPA eth2 is connected to port 5, SPA eth3 is connected to port 7, SPB eth2 is connected to port 6 and SPB eth3 is connected to port 8 (i.e. eth2 on each SP connected to different switches)

    If a single Ethernet switch fails then essentially there are multiple port failures on the VNXe. My understanding is that the VNXe handles port failures in a different way to the VNX by using Fail Safe Networking (FSN) or peer port redirection as described in the VNXe Whitepaper and VNXe High Availability document. If SPA eth2 were to fail on its own, FSN would direct traffic through the Clariion Messaging Interface (CMI) and use the physical interfaces of SPB (I assume that the whole LACP group is redirected or you would be left with eth2 and eth3 in different port groups on the Ethernet switch - ouch).

    If the configuration is as I have described in option 1 then the FSN can’t occur through the CMI because the peer port has also failed e.g. If SPA eth2 and SPB eth2 fail simultaneously then SPA eth2 traffic obviously can’t be redirected to SPB eth2.

    If the configuration is as I have described in option 2 then the FSN would result in server_1 redirecting SPA eth2 traffic through the CMI to SPB eth2 but server_1 will be unable to redirect SPA eth3 traffic through SPB eth3 because the port is unavailable . This will result in server_1 having two ports connected to two different LACP port groups on the switches and communications will be broken. I assume server_2 would be in a similar situation.

    Neither of these cabling options seems to handle an Ethernet switch failure very well. Is there a better way to cable the system for VNXe? Should we go against common sense and have the ports on one SP connected to one switch and the ports on the other SP connected to the other switch just to cater for FSN? Am I missing something here? Is there some key part of FSN and how it handles LACP that I don’t understand? Does enabling LACP disable the FSN feature?

    Any ideas?

    Thanks,

    Cameron

  • Kumar_A

    2 Intern

    •

    727 Posts

    1035

    0

    Posted August 10th, 2011 07:00

    If you want to have protection against switch failure in the environment, you can assign two IP addresses to the iSCSI server. These two IPs can (should) be in two different subnets and therefore connected to two different switches.

  • 529

    0

    Posted August 10th, 2011 16:00

    Thanks for the response Avi. The configuration discussed in this thread and shown in Rohan’s diagram is using LACP. My concern is over what happens when an Ethernet Switch fails when using LACP. The workaround you have suggested (multiple addresses) may be suitable for an iSCSI Server but is not applicable to a Shared Folder Server.

    Please note that even if the issue being discussed was restricted to iSCSI, the use of LACP should still be possible. The VNXe High Availability document seems to skirt the issue of Ethernet switch failure which suggests that it is not supported for VNXe. Is this the case? If it is supported, which configurations will work?

    Thanks,

    Cameron

  • 529

    0

    Posted August 18th, 2011 16:00

    I think I have found the answer to my previous questions in the "VNXe Theory of Operations" training module. It appears that when a NIC is bonded, all of the physical interfaces in the bond belong to the same FSN interface. This implies that if a single interface in an LACP bond fails, the FSN interface still has a link and the I/O is not redirected.  If all of the NICs in the bond fail, the FSN interface no longer has a link and the I/O is redirected to the peer SP. Although this is not stated clearly in the training module, I have discussed this with an EMC engineer and this was his interpretation of how LACP and FSN work together.