We have an MD3200i loaded with 12 x 600GB 15k SAS drives. Two Raid 10 arrays with 4 LUNs per array. Using 2 x R710 servers with 4 NICs per server for iSCSI traffic using different subnet on each port -
Server1
192.168.130.1
192.168.131.1
192.168.132.1
192.168.133.1
Server2
192.168.130.2
192.168.131.2
192.168.132.2
192.168.133.2
MD3200i Controller1
192.168.130.101
192.168.131.101
192.168.132.101
192.168.133.101
MD3200 Controller2
192.168.130.102
192.168.131.102
192.168.132.102
192.168.133.102
We have 2 switches for redundancy with the correct connections made. All LUNS are available, 16 paths per NIC, 8 active i/o, 8 standby, round robin enabled and showing 8 paths per LUN so all appears fine. When using ATTO benchmark, the read performance is crippled from 64k upwards whereas write shows up to 250MB/s. Having read many threads with similar issues, I have yet to find a fix. Delayed ack is disabled, we have tried Dell's old recommended way of a single vSwitch with all NIC's in active inactive configuration as per their best practices (confirmed correct by Dell engineer a while ago). I now have their ESXi 5 guide for MD3200i which recommends single vSwitch per NIC but then goes on to say add each NIC to the iSCSI initiater through VMKernel port bindings but they don't appear unless I put them in the same vSwitch.
Ok, finally sorted. Both hosts performing as they should. A combination of things. Updating vmware to use the Broadcom 2.2.1l.v50.1 driver then set up port binding as per latest Dell MD3200i deployment guide for ESXi 5 does indeed work. The signed VMWare driver for the Broadcom does not work for me so if anyone else has this issue, this was my fix. IOMeter reporting squential reads of 400MB/s when run at the same time on two hosts accessing separate disk groups through individual controllers in active/active scenario.
I have tried disabling delayed ack on VShere, switching flow control off on the switches, setting jumbo frames with an MTU of 6500 (seemed quite optimal for some, better than 9000). Still the read performance drops to ~2MB/s after 32K while write speeds are hitting 260MB/s.
It sounds like your config is all correct... But obviously something isn't right. You should get better read performance than that with only a single connection...
Does this behavior apply to LUNs on every RAID group? (could be problems with a disk).
Perhaps experiment with a direct-connect from a server to one controller port (temporarily remove the switch from the equation) and see if it improves?
The segment size is 128k. On Friday I was experimenting with different Broadcom VIB's and having totally lost all paths with one VIB, I quickly reconfigured VMWare ending up with a rather strange setup with a single vSwitch with VSphere reporting 3 NICs as unused, only a single storage adapter showing but with 64 paths and round robin definitely working. I'm now getting blazing speeds-
IOMeter is showing sequential read speeds of just over 400MB/s. Didn't think an MD3200i would even manage these sorts of speed. IOPS are still set to default 1000. I could do with optimising the middle ranges a little and will be increasing the spindle count anyway. I have another attached host that I am now going to update the Broadcom VIB and try out a few different VMWare configs as Dell can't seem to decide on what is optimal. The VIB that made the difference was from the BCM-NetXtremeII-3.0-735414 uncertified bundle.
Nicely done Jonathan. I certainly wouldn't have expected it to be driver related, haven't seen it cause a problem like that before.
As far as "optimal" configuration goes, having a single vSwitch vs. one vSwitch per adapter shouldn't make any difference unless you're implementing a storage heartbeat vmkernel port. What's more important is that you have a "one-to-one" setup between the NICs and vmkernel ports, each NIC only being used by a single vmkernel port (which it sounds like you have based on those unused adapters).
Thanks for your help in this matter. It turns out it was not the driver at all as you suspected. My second host still has the same issue and it is down to the config of the vSwitch. No matter how I try, I cannot add more than one VMKernal to the port bindings of the iSCSI adapter. On my working host, there is only a single adapter with 4 VMKernals added with 8 LUNs showing 64 paths. The unused adapters was misleading, it was referring to the NICs on the vSwitch. I moved the unused ones up to used and all remains the same on the working host. On the non working host, there are permanently 4 unused port identifiers on the san, one for each NIC, I would like to remove these and have a single port identifier to a single vSwitch with all four NICs. It is currently showing 8 iSCSI adapters that I am unable to remove whereas my working host only has a single adapter. I have tried rebooting, disconnecting through iscsi session remove. How do I disable the adapters when it is greyed out? I've readded the NICs but it remains greyed out.
Ok, I see what's happened, updating the broadcom driver knocked out all the hardware adapters leaving me with only the iSCSI software adapter which is working great. This allows me to use port binding. On the non working host, I now have it set up the same using the software adapter but the hardware ones are still showing and I can't connect to the san. Maybe worth removing the iscsi-bnxi driver to remove the hardware adapters and only use the software one? Have heard of problems with Broadcom NICs before.
jonnyjonah
9 Posts
1452
1
Posted October 10th, 2012 04:00
Ok, finally sorted. Both hosts performing as they should. A combination of things. Updating vmware to use the Broadcom 2.2.1l.v50.1 driver then set up port binding as per latest Dell MD3200i deployment guide for ESXi 5 does indeed work. The signed VMWare driver for the Broadcom does not work for me so if anyone else has this issue, this was my fix. IOMeter reporting squential reads of 400MB/s when run at the same time on two hosts accessing separate disk groups through individual controllers in active/active scenario.
Thanks
Jonathan