UNSOLVED

michael_khadra

updated

15 years ago

1

6570

July 27th, 2011 09:00

iSCSI Connectivity with VNXe3300 Link Aggregation

I have a VNXe3300 storage with additional 4 x 1GB module, therefore i have 8 x 1GB network ports on each SP (Ports 2 to 5 and Ports 10 to 13).

I have two servers on which i need to install VMware and i want to connect each server to the storage using 6 ethernet ports (3 to each SP modules) and create one LUN to be shared by those two servers for clustering purposes (1 Datastore shared by 2 ESXs).

When i create an iSCSI server, i need to associate it with one ethernet interface on the VNXe.

In addition, with link aggregation, i can only aggregate ports on the same I/O module and therefore i can only aggregate ports 2 to 5 together, so the iSCSI server will be using 4 ports on each SP and not more.

What should be done in this case? How should be the physical connectivity and Aggregation configuration done?

Thanks in advance.

Regards,

  • ghardy123

    6 Posts

    999

    0

    Posted August 2nd, 2011 18:00

    Can you not get an ethernet switch?  Seems the design would be far easier with a gigabit switch...if not, that is a challenging question; I'll see if I can figure something out; unless someone else gets here first.

    -G

  • 999

    0

    Posted August 2nd, 2011 23:00

    OK, let us consider we are putting a GigabitEthernet switch between the servers and the VNXe for iSCSI connectivity.

    I configure the SP A ports 2-5 as an etherchannel and SP A ports 10-13 as another etherchannel.

    The same mapping is done automatically for ports on SP B by the VNXe.

    Now i connect all of them to the GigabitEthernet switch, and i configure 4 LACP etherchannels (4 ports each).

    I connect as well the two servers (Let say 6 ports from each server) to the switch and  i configure them on the switch as two LACP etherchannels (6 ports each), with Teaming configured on the servers.

    Now the questions is as follows:

    - LACP traffic is load balanced by source or by destination, but I guess when the same server is communicating with the same VNXe over iSCSI, it will be considered as one single communication and will use only one of the links and not all of them. Am i right?

    - When i configure an iSCSI server on the VNXe, i need to map it to a port (eth2 for example, which has 3,4,5 bundled with it). So at a maximum, i guess it can be associated with 4 ports on one of the SPs. Is this correct?

  • cergio

    13 Posts

    999

    0

    Posted August 3rd, 2011 12:00

    Regarding iscsi topology, there is no precise information. I downloaded a lot of EMC documentation but none explains in detail the best practices for high availability iscsi. my problem is similar to yours.It does not explains how to set the fail safe FSN Network Features

  • ghardy123

    6 Posts

    1000

    0

    Posted August 4th, 2011 09:00

    From my understanding for iSCSI implementation with VNXe and VMware...I have implemented and tested this for failover and everything worked flawlessly.  Configuration below...


    VNXe3300

    :eth 2/3/x/x from both SPs go to the same switch or switch stack.  You can LACP any ports that are on the same IO card but can't LACP the onboard with an add-on card

    :setup iSCSI server for SPA and SPB and give them IP addresses

    In a perfect world you would create two IO paths to each SP that have different subnet, the last implementation I did was setup with the following:

    SPA - iSCSI server : ISCSIA

    eth 2/3 - LACP

    eth 10/11 - LACP

    eth 2/3 on SPA iSCSI Server were assigned 10.10.12.21

    eth 10/11 on SPA iSCSI Server were assigned 10.10.16.21

    SPB - iSCSI server : ISCSIB

    eth 2/3 - LACP

    eth 10/11 - LACP

    eth 2/3 on SPB iSCSI server were assigned 10.10.12.22

    eth 10/11 on SPB iSCSI server were assigned 10.10.16.22

    Network:

    eth 2/3 on both SPs were plugged into CiscoSW1 and port channelled (port channel 2 for SPA and port channel 3 for SPB, if that helps)

    eth 10/11 on both SPs were plugged into CiscoSW2 and port channelled (port channel 2 for SPA and port channel 3 for SPB, if that helps)

    CiscoSW1 and CiscoSW2 had an LACP trunk between them to allow traffic to pass between switches if required (though it shouldn't be passing storage traffic based on the configuration)

    the iSCSI VLANs were isolated i.e. no traffic not originating from VLAN12 could communicate with VLAN12 (16 setup the same way).  Both VLANs are unrouted

    VMware:

    vSwitch1 was created for iSCSI and assigned vmnic 1 and vmnic 5 (two different NIC cards to keep a NIC failure from killing storage traffic)

         portgroup: iSCSIA - vmk2: 10.10.12.11 - failover nic override on pg set to only allow communications to vmnic1

         portgroup: iSCSIB - vmk3: 10.10.16.11 - failover nic override on pg set to only allow communications to vmnic5

    vmnic1 plugged into CiscoSW1

    vmnic5 plugged into CiscoSW2

    SW iSCSI hba used and assigned vmk2 and vmk3

    This gives the ESX host 2 paths to each SP, one to each subnet on each sp (so four paths to the SAN storage):

         portgroup iSCSIA can traffic to 10.10.12.21 and 10.10.12.22 on CiscoSW1

         portgroup iSCSIB can traffic to 10.10.16.21 and 10.10.16.22 on CiscoSW2

    In a failover event, everything fails properly

         ...if you lose a network switch then ESX will pick up traffic using the other subnet on the remaining switch using the remaining functional storage ports

         ...if you lose a SP then the VNXe will 'fail' its ports to the same port numbers on the opposing SP so SPA eth2/3 on failure will pass that traffic through SPB eth2/3 (same with eth10/11) to the LUNs

         ...if you lose a NIC port on the ESX server then the storage traffic will be passed on the remaining functional port-group

    Notes:

    I've never had good luck with making port-channels work effectively on ESX (though I haven't tried with 4.1), so I stick with the vmware recommendation of one vmnic per storage vmk.

    I believe VMware now fully supports Jumbo frames, so if you have the capability I'd attempt to get it to work as it theoretically moves your throughput threshold from around 100MB/s per link to somewhere between 200-500MB/s (I've seen different people post different results, I've not done this myself with the VNXe yet so YMMV).

    As far as FSN, I suppose you could create an FSN with eth2/3 -> eth4/5; I've never tried for that much redundancy with the VNXe as I'm unsure of exactly what you're trying to protect against with that configuration (bad network switch blade?).

    Hope this makes sense.

    -George

  • jablonde

    9 Posts

    1000

    0

    Posted August 4th, 2011 09:00

    Great post.

    I've tried to get jumbo frames to work w/ VMware 4.1 & the VNXe 3300, but so far haven't been successful.  Everything is configured, but ESX won't see the LUNs.  Once I switch back to 1500 MTU, everything works.  Could be the switches that I'm using though.

  • ghardy123

    6 Posts

    1000

    0

    Posted August 4th, 2011 10:00

    Same here...the implementation I was referencing has Cisco 3650X switches; they advertise Jumbo support but the routing cpu (not sure if that is the right term) only supports a 1500 mtu and I think that ends up breaking the Jumbo frames.  The literature on the switch says the cpu shouldn't be used in non-routed traffic but my experience disagrees with them.  vmkping -s 9000 would work from any jumbo configured port to storage and I could always see the VNXe LUNs...saw the paths and the device properly through esxcfg-scsidev and fdisk -l /blahblahblah but could never actually access the disks.  Formatting for vmfs, etc always responded with 'access denied' or some other unhelpful error.

    I didn't have time to keep playing with it so I never got further than accepting jumbo 'failure' .

    -G

  • ghardy123

    6 Posts

    1001

    0

    Posted August 4th, 2011 14:00

    Only if eth2/3 on SPA fail to SPB in which case there won't be a duplication of IP address since eth2/3 on SPA would be failed (thus not online).  eth2/3 on SPB will never have the IP address assigned to eth2/3 of SPA unless the ports on SPA are down (or if you configure SPB with the same IP address, but that would be a misconfiguration issue, not a functionality one).

    FSN (as with the old Celerra, etc) only fails to the 'standby' ports when the 'primary' ports are down.

    Your statement is not following through on the whole process.  Let me restate what you said more accurately, that may clarify the confusion:

    If you configure the ports eth2/3 of the SPA with X IP address the ports eth2/3 of SPB will assume the same ip address of SPA ON FAILOVER.  Putting these same ports on the same switch will not fail on duplication of ip address since ports eth2/3 on SPA are no longer functional.  Your network switch will deal with the fact that the MAC address recieving X IP address has changed.

  • cergio

    13 Posts

    1001

    0

    Posted August 4th, 2011 14:00

    I do not understand, according to FSN feature (fail safe network), if you configure the ports eth2/3 of the SPA with a X IP address the ports eth2/3 of SPB will assume the same ip address of SPA. Putting these same ports on the same switch fails for duplication of ip address.

  • cergio

    13 Posts

    1001

    0

    Posted August 4th, 2011 14:00

    Thank you very much for you explanation Ghardy. Now it becomes clearer to me how it works.

  • mworkman

    2 Posts

    224

    0

    Posted August 23rd, 2011 18:00

    Can multiple interfaces be added to an iSCSI server when eth2-5 are being used in link aggregation? I do not have an add-in card so I only have 4 ports per controller.

    I'm concerned that I only have one path listed on my ESXi servers for each datastore. Adding another interface would solve this I think, but not sure.