DurkinR

updated

14 years ago

D

DurkinR

50 Posts

0

195020

November 1st, 2012 13:00

ESXi 5.1, Intel X520, PS6110XV - Latency issue

Hello,

Configuration

Setting up our first R710 host running ESXi 5.1, with a dual-port Intel X520 10GbE card for sw iSCSI, connected to a PS6110XV through Force10 S4810 switches.

DCB is disabled on the PS array. No warnings/errors in the logs. It's running the 5.2.5 firmware.

The EQ MEM 1.1.1 is installed with storage heartbeat properly configured on the ESXi host, and appears to be working fine. DelayedAck is disabled, and iSCSI login timeout is set to 60s.

The Force10 switches have jumbo frames and flow control rx on tx on setup, and they are stacked.

A test virtual machine is setup and running on a test volume.

Problem

Seeing high latency issues on the host. Using the vSphere Client to watch read/write latency on the sw iSCSI HBA. In the past hour, the max has shot up to 7000 ms, with a couple of other bumps in the 1000-2000 range. In between these peaks is generally seems low, less than 10 ms. During the configuration of the Intel NICs, I noticed I cannot enable flow control. The default setting is autonegotiate on, rx off, tx off. I've tried a couple ways to enable it and even added it to the startup script and it still says it is disabled, when viewed with the ethtool -a vmnicx command.

Can someone point me in the right direction on where to look for a problem? This is my first run with 10GbE, our current production setup uses ESX 4.1u2 at 1GbE, and it runs fairly well, certainly no latency above 100ms.

  • DurkinR

    50 Posts

    3246

    1

    Posted November 16th, 2012 17:00

    It looks like we have resolved the issue, and by we, I mean Force10 tech support.

    To recap, after setting up our new EqualLogic array , connected to two Force10 S4810 switches and an ESXi 5.1 host using an Intel X520 NIC, we were having connection issues. 5-8% retransmits (as measured from SAN HQ), high latency spikes on the ESXi host, The EqualLogic MPIO was not working properly, usually only one connection per volume at a time was working, frequent re-connections to the VMFS volume, pings would drop to the SAN, basically it was unusable.

    We began by troubleshooting the ESXi host, but that lead nowhere, other than a helpful driver update that allowed me to enable FlowControl on the NIC.

    Eventually we figured out that if we shut off one of the switches, the problem went away. Then we figured out that if we left both switches up but pulled the plug on one of the Port Channel connections, the problem went away.

    The F10 S4810 switches are connected to our 1 gig SAN via a 10 GbE port channel, connected to two interfaces, one on each F10 switch. The F10 tech support agent figured out by looking at each stack config that the Cisco 3750Es had a dynamic port-channel config, and the F10s had a static config. So he re-built the port-channel on the F10s and made it dynamic, and for the last couple of hours, it has been working correctly, no re-transmits, no latency spikes, no connection drops. I would have never guessed the port-channel was working right because it never dropped a link. But the Force10 tech was very methodical and eventually figured out the port-channel was to blame.

    Anyway, I hope that this helps someone out there someday. Definitely a strange problem that required turning on and off  switches and pulling cables to track down.

  • DurkinR

    50 Posts

    2948

    0

    Posted November 2nd, 2012 09:00

    Don,

    Thanks for the tip...I ran this command to disable interrupt moderation for the Intel 10GbE card:

    # esxcli system module parameters set -m ixgbe -p "InterruptThrottleRate=0"

    I found it in this tech white paper www.vmware.com/.../VMW-Tuning-Latency-Sensitive-Workloads.pdf

    I'm still seeing issues, spikes of super-high latency when watching the performance counters on the sw iSCSI VMHBA, re-transmits being reported by SAN HQ. I still can't enable flow control on the X520 which seems odd.

    Perhaps it is a switch config issue. I followed the guidelines for configuring the S4810s in the EqualLogic rapid config guides...en.community.dell.com/.../3615.rapid-equallogic-configuration-portal-by-sis.aspx

  • DurkinR

    50 Posts

    2948

    0

    Posted November 2nd, 2012 11:00

    No, DCB is disabled on the array, that was one of the first things I did when troubleshooting. The switches do have automatic iSCSI configuration, that is something I'm looking into. when I run the show iscsi command I get this:

    FTOS#show iscsi

    iSCSI is enabled

    iSCSI session monitoring is disabled

    iSCSI COS : dot1p is 4 no-remark

    Session aging time: 10

    Maximum number of connections is 256

    ------------------------------------------------

    iSCSI Targets and TCP Ports:

    ------------------------------------------------

    TCP Port    Target IP Address

    3260

    860

    FTOS#

    Th switch config guide says that this is what happens when iSCSI is enabled:

    • At the first detection of an EqualLogic array, an MTU of 12000 is enabled on all ports and

    port-channels (if it is has not already been enabled).

    • Spanning-tree portfast is enabled on the interface identified by LLDP.

    • Unicast storm control is disabled on the interface identified by LLDP.

  • Joe S586

    9 Technologist

    •

    729 Posts

    2948

    0

    Posted November 2nd, 2012 12:00

    Are the iSCSI interfaces on a separate VLAN (other than the default VLAN).  I noticed that the configuration portal guide for the S4010 doesn’t specify this, however it is recommended per the EqualLogic Configuration guide.  If not, configure a separate VLAN for the iSCSI interfaces, and test again.  You also need to add the vlan to the stacking ports and/or ISL / Lag ports

    Also, take a look at your input and output throttles, you may need to tweak this (show interface XX), look for high numbers in the throttles, if you see them you want to test by changing the flow control threshold setting, (the example below is the highest values, and if this helps, you can always lower the thresholds to a point where you still don’t see the high latency, and are comfortable in keeping or go back to the defaults if this doesn’t help)

    change this: flowcontrol rx on tx on

    to this: flowcontrol rx on tx on threshold 2047 2013 2013

    The automatic iSCSI settings aren’t recommended.  Stick to what the configuration portal suggests.

     

    -joe

  • DurkinR

    50 Posts

    2948

    0

    Posted November 2nd, 2012 16:00

    Hey Joe,

    Yes, the iSCSI are on a different VLAN than the default. The guide should include this, as well as the need to enable "protocol spanning-tree rstp" with "no disable" at the global config level in addition to setting "spanning-tree rstp edge-port" on the edge ports. Another issue, if the iSCSI settigns aren't recommended, why are tehy there? They were added specifically for EqualLogic and Compellent equipment, and the guides don't mention it at all, even though it's enabled by default.

    I'm not sure how to verify the VLAN is configured on the stacking ports, I'll have to look into that and make sure it's working....right now show vlan does not include the stacked ports.

    Right now the throttles on the ports that are in-use are 0, for both input and output.

    Any ideas why I cannot enable flowcontrol on hte intel x520 NIC?

    Thanks

  • DurkinR

    50 Posts

    2948

    0

    Posted November 2nd, 2012 17:00

    The speed and duplex are not hard coded. But checking one of the ports that the Intel x520 is uing, shows 29,162 output throttles, so I'll have to take a look at Joe's advice again on setting a threshold.

  • DurkinR

    50 Posts

    2948

    0

    Posted November 2nd, 2012 17:00

    The Threshold command is not available on the S4810 switch with this FTOS version. I have verified the ports speeds are set to auto (the default). when I look at the NIC in ESXi, it shows 10000 full, and negotiate is not an option. not sure if that has any bearing on not being able to enable flowcontrol on the NICs.

  • joergriether

    127 Posts

    660

    0

    Posted November 4th, 2012 04:00

    Even if it sounds strange to you because you think that simply can´t be - if you can find the time and muse, please do a little experiment for me: disable jumbo frames completely on the source (system that carries the x520s), disable stp/rstp/rstpportfast on the f10 connected to the eql completely, place it all in default vlan. I would start at the very beginning. reason: 7000ms is NOT AT ALL "standard behavior" and has in my opinion serious causes. Thus, i would start at rstp and jumbos.

    Regards, Jörg

  • DurkinR

    50 Posts

    660

    0

    Posted November 5th, 2012 14:00

    Jorg,

    I have remvoed stp fromt eh ports in use and also auto-config of the iSCSI on the switch and reloaded the switches. The constant re-transmits have have slowed, but are still happening, and just had another spike of 8000ms latency. So that improved some but not all of the issue. I just flashed the firmware on the intel x520 too and going to see if that helps. I will try some of your other suggestions one by one if this doesn't help.

    Definitely a strange problem I have not seen before. No latency to speak of for 20 minutes and then an 8000ms spike. 3-4% re-transmits then it levels out near 0.1%.

  • joergriether

    127 Posts

    660

    0

    Posted November 6th, 2012 02:00

    8000ms is SO bad, i don´t think lro or delayed ack is the causation for this behaviour.

    Did you already disable jumbo frames on the esx and put everything in default vlan as i suggested?

    Best regards,

    Joerg