d_ready

updated

6 years ago

D

d_ready

14 Posts

0

2766

March 30th, 2021 10:00

m1000e no traffic between A2 and blades with esxi

Enivornment: m1000e with m6220 switches in the A1/A2. Blades are M630

our ESXI hosts are teamed between A1 and A2. A1 and A2 are not stacked. The interfaces for vmnic0(a1) and vmnic1(a2) on the m6220 are configured the same.

Right now, all the blades function when only assigned to vmnic0(a1), however traffic stops when we add in vmnic1(a2) as a redundant link. if we just use vmnic1 and remove vmnic0, they also do not send traffic.

Running show interface counters gi1/0/9, i'm seeing minimal traffic, with all traffic being discarded on transmit.

 

Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Gi1/0/9 2685 434 58 2193

Port OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
--------- ---------------- ---------------- ---------------- ----------------
Gi1/0/9 158902 2380 48510 108012

FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Received Oversize Packets: .................... 0
Transmitted Oversize Packets: ................. 0
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 559886

I'm having trouble identifying if there is a configuration issue on the ESXI host, or the m1000e, or if there is a hardware failure.

Everything was working fine until some weeks ago. The server admin noted that no changes were made on his end in ESXI or in the dell CMC, and no configuration changes were made on the switches.

The A2 switch shows as connected, and no errors in the Dell CMC for it. CDP is successful over the link between the A2 and the blades.

I'm trying to find out why the transmits are all being discarded. Any help is appreciated

  • d_ready

    14 Posts

    1342

    0

    Posted April 5th, 2021 10:00

    I don't mean to Triple Post, but I've found what appears to be the cause. Root still to be determined by the server Admin.

     

    With noticing traffic failing on the link between my c9500 and the m6220, I looked into the counters and saw  the TX pause frames:

    BldChs1A2#show interfaces counters te1/1/1

    Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
    --------- ---------------- ---------------- ---------------- ----------------
    Te1/1/1 1128521 18698 375403 734420

    Port OutTotalPkts OutUcastPkts OutMcastPkts OutBcastPkts
    --------- ---------------- ---------------- ---------------- ----------------
    Te1/1/1 9363 1208 7828 327

    FCS Errors: ................................... 0
    Single Collision Frames: ...................... 0
    Late Collisions: .............................. 0
    Excessive Collisions: ......................... 0
    Multiple Collisions: .......................... 0
    Received Oversize Packets: .................... 0
    Transmitted Oversize Packets: ................. 0
    Internal MAC Rx Errors: ....................... 0
    Received Pause Frames: ........................ 0
    Transmitted Pause Frames: ..................... 13641269
    Receive Packets Discarded: .................... 32
    Transmit Packets Discarded: ................... 0

     

     

    I then checked all my interface counters on the m6220, to see if there was excessive traffic coming from any of it's active interfaces (Only blade 9 was supposed to have any vnic's attached to a2).

    BldChs1A2#show interfaces counters

    Port InTotalPkts InUcastPkts InMcastPkts InBcastPkts
    --------- ---------------- ---------------- ---------------- ----------------
    Gi1/0/1 0 0 0 0
    Gi1/0/2 0 0 0 0
    Gi1/0/3 108 0 108 0
    Gi1/0/4 3775806 0 3775806 0
    Gi1/0/5 0 0 0 0
    Gi1/0/6 0 0 0 0
    Gi1/0/7 0 0 0 0
    Gi1/0/8 0 0 0 0
    Gi1/0/9 637 224 113 300
    Gi1/0/10 0 0 0 0
    Gi1/0/11 86 0 20 66
    Gi1/0/12 0 0 0 0
    Gi1/0/13 0 0 0 0
    Gi1/0/14 0 0 0 0
    Gi1/0/15 0 0 0 0
    Gi1/0/16 0 0 0 0
    Gi1/0/17 0 0 0 0
    Gi1/0/18 0 0 0 0
    Gi1/0/19 0 0 0 0
    Gi1/0/20 0 0 0 0

    Te1/1/1 1134693 18753 377354 738586
    Te1/1/2 0 0 0 0

     

    Gi1/0/9 was supposed to have traffic, as well as TE 1/1/1. Gi1/0/11 as low, and negligible, but Gi1/0/4 stuck out.

     

    I shut down interface gi1/0/4, and immediately  the pause frames stopped. I was then able to reach the m6220 from my c9500, as well as reach all my test VMs.

     

    We're currently checking out what on blade 4 would be causing this kind of traffic that caused the pause frames. It's al older 610 half height blade, but vcenter doesn't show any NIC's that would appear on the A2 switch (vmnic1). We might just decommission this blade, but I'd like to find the actual cause first.

    For now, it appears the issue is resolved.

  • DELL-Josh Cr

    Community Manager

    •

    9801 Posts

    •

    45350 Points

    1067

    0

    Posted March 30th, 2021 14:00

    Hi,

    What does "show spanning-tree” show for that port? Page 756 https://dell.to/3fwPLl9

  • d_ready

    14 Posts

    1001

    0

    Posted March 31st, 2021 06:00

    Hi Josh,

     

    Here is the Spanning Tree output:

     


    BldChs1A2#show spanning-tree gi1/0/9

    Port Gi1/0/9 Enabled
    State: Forwarding Role: Designated
    Port id: 128.9 Port Cost: 20000
    Port Fast: Yes Root Protection: No
    Designated bridge Priority: 32768 Address: 0025.6413.AB6E
    Designated port id: 128.9 Designated path cost: 0
    CST Regional Root: 80:00:00:25:64:13:AB:6E
    Root Guard..................................... FALSE
    Loop Guard..................................... FALSE
    TCN Guard...................................... FALSE
    Auto Portfast.................................. TRUE
    Port Up Time Since Counters Last Cleared....... 0 day 1 hr 42 min 38 sec
    BPDU: sent 43268, received 0

     

  • Dell-Stephane T

    11 Legend

    •

    3699 Posts

    998

    0

    Posted March 31st, 2021 07:00

    Hi,

     

    Spanning tree seems to be activated, could you send the command show spanning-tree detail for deeper analysis? Could you also check/deactivate all firewall rules on the VM?

     

    Regards,

  • d_ready

    14 Posts

    997

    0

    Posted March 31st, 2021 09:00

    Here's the output of the detail for the port:

    Spanning tree Enabled (BPDU flooding : Disabled) Portfast BPDU filtering Disabled mode rstp
    CST Regional Root: 80:00:00:25:64:13:AB:6E
    Regional Root Path Cost: 0
    ROOT ID
    Address 80:00:00:25:64:13:AB:6E
    This Switch is the Root.
    Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
    Bridge Max Hops 20

    Number of topology changes 0 last change occurred 13d21h49m42s ago
    Times: hold 6, hello 2, max age 20, forward delay 15


    Port Gi1/0/9 Enabled
    State: Forwarding Role: Designated
    Port id: 128.9 Port Cost: 20000
    Root Protection: No
    Designated bridge Priority: 32768 Address: 0025.6413.AB6E
    Designated port id: 128.9 Designated path cost: 0
    CST Regional Root: 80:00:00:25:64:13:AB:6E CST Port Cost: 0
    BPDU: sent 48271, received 0

     

    I'll need to get a request in to disable the firewall on esxi instance, however here is the firewall rule list

    My test VM in this blade has it's windows firewall disabled.

    esxcli network firewall ruleset list
    Name Enabled
    ---------------------- -------
    sshServer true
    sshClient false
    nfsClient false
    nfs41Client false
    dhcp true
    dns true
    snmp true
    ntpClient false
    CIMHttpServer true
    CIMHttpsServer true
    CIMSLP true
    iSCSI true
    vpxHeartbeats true
    updateManager true
    faultTolerance true
    webAccess true
    vMotion true
    vSphereClient true
    activeDirectoryAll true
    NFC true
    HBR true
    ftpClient false
    httpClient false
    gdbserver false
    DVFilter false
    DHCPv6 false
    DVSSync true
    syslog false
    WOL true
    vSPC false
    remoteSerialPort false
    rdt false
    cmmds false
    vsanvp false
    rabbitmqproxy true
    ipfam false
    vvold false
    iofiltervp true
    esxupdate false
    vit false
    vsanEncryption false
    pvrdma false
    vsanhealth-unicasttest false

     

  • DELL-Josh Cr

    Community Manager

    •

    9801 Posts

    •

    45350 Points

    993

    0

    Posted March 31st, 2021 10:00

    Spanning-tree looks fine. Can you reboot that switch?

  • d_ready

    14 Posts

    992

    0

    Posted March 31st, 2021 10:00

    We've rebooted the switch multiple times, and also loaded the config onto a spare unit and swapped them out. No success with either option.

  • d_ready

    14 Posts

    947

    0

    Posted March 31st, 2021 11:00

    It was updated in one of our troubleshooting steps. We still have the old version saved under the 2nd image.

     

    I don't have evidence for it, as I'm not very familiar with the m1000e, however I'm suspecting an issue on the Chassis A2 slot. The VM we have attached to it currently is able to get a DHCP address, but no ICMP or other traffic is passing. First I need to rule out any possible network issues.

  • DELL-Josh Cr

    Community Manager

    •

    9801 Posts

    •

    45350 Points

    989

    0

    Posted March 31st, 2021 11:00

    Was it updated as a troubleshooting step or was it always a different version from the other switch?

     

  • DELL-Josh Cr

    Community Manager

    •

    9801 Posts

    •

    45350 Points

    991

    0

    Posted March 31st, 2021 11:00

    What firmware version are you on?