A large percentage of dropped frames typically indicates an issue with the Ethernet interconnect.   Your post indicates this is not necessarily a cable issue, however it can be one if cables are handmade and there is near end crosstalk.    It would help if you could indicate the number of packets transferred as well as the number of dropped frames to give a real indication of the packet retry rate.

Please check to ensure your network: 5709 NICs, 5448 switch ports, and MD3000i are all configured for Jumbo frames (MTU=9000).  If one component is not configured, then the whole interconnect will  be at a default MTU=1500 which can decrease performance.  Flow control should also be enabled to minimize the impact of dropped frames on overall performance.   Spanning Tree should be disabled (or set to portfast) at the server and storage ports.

Also make sure your switch has the latest firmware (if new this is not an issue), as there was a firmware revision about 5-6 months ago.

Recovered packets is a fact of life in Gbit Ethernet systems, and won't show up in Event Viewer.  You might be able to get more statistics via the CLI of the switch itself.  It is only when the retry rate goes 'way up that it is an issue.

Your observation that only two of the paths to the MD3000i, one per controller, are in use is OK - the other ports are typically used for failover.

Thank you for the reply.  I'll see if I can answer your questions here...

Cabling.  All Ethernet cabling between the servers in question and this MD3000i are brand new Amphenol CAT-6 (pre-made).

Regarding packets transmitted vs. dropped I re-ran my three types of backups I've been testing with and have recorded those numbers below.  This is viewed from the MD3000i.  The baseline on the MD3000i was reset prior to each job starting.  All Backup Exec jobs were initiated from a Dell 2900, to the primary node on the Oracle RAC, and all data (be it datatbase or "flat files") was pulled from the MD3000i via the iSCSI connection between the MD3000i and the primary Oracle node.

Backing up 2.4GB of small flat files (8,823 total files)

2,586,173,359 Bytes

632MB/Min

4:19 elapsed

Total Frames: 483,128

Dropped Frames:  7

Dropped Percent:  .0014%

 

RAID Controller Module 1 Port 0

Frames:  486

Total Bytes:  1,294,156

Dropped Frames:  0

Jumbo Frames:  142

 

RAID Controller Module 1 Port 1

Frames:  6762

Total Bytes:  23,678,036

Dropped Frames:  0

Jumbo Frames:  2549

 

RAID Controller Module 0 Port 0

Frames:  22,320

Total Bytes:  127,500,118

Dropped Frames:  0

Jumbo Frames:  19,831

 

RAID Controller Module 0 Port 1

Frames:  453,560

Total Bytes:  3,440,775,726

Dropped Frames:  7

Jumbo Frames:  446,792

 

 

 

Backing up a single 7.4GB File

7,457,487,140 Bytes

5,203 MB/Min

2:33 elapsed

 

RAID Controller Module 1 Port 0

Frames:  498

Total Bytes:  617,016

Dropped Frames:  0

Jumbo Frames:  49

 

RAID Controller Module 1 Port 1

Frames:  3992

Total Bytes:  14,986,002

Dropped Frames:  0

Jumbo Frames:  1640

 

RAID Controller Module 0 Port 0

Frames:  399,139

Total Bytes:  3,289,908,116

Dropped Frames:  0

Jumbo Frames:  398,480

 

RAID Controller Module 0 Port 1

Frames:  512,499

Total Bytes:  4,227,152,756

Dropped Frames:  18

Jumbo Frames:  512,004

 

 

 

Backing up an Oracle Database

44.22G Input (raw database size)

2.46G Output (Oracle-compressed size that came across to the Backup Exec server)

119 MB/Min

20:55 elapsed

 

RAID Controller Module 1 Port 0

Frames:  1,146,172

Total Bytes:  9,342,220,304

Dropped Frames:  24,383

Jumbo Frames:  1,131,080

 

RAID Controller Module 1 Port 1

Frames:  1,315,829

Total Bytes:  10,434,808,433

Dropped Frames:  24,741

Jumbo Frames:  1,260,219

 

RAID Controller Module 0 Port 0

Frames:  1,809,416

Total Bytes:  14,879,055,640

Dropped Frames:  25,030

Jumbo Frames:  1,802,152

 

RAID Controller Module 0 Port 1

Frames:  1,689,448

Total Bytes:  13,894,084,685

Dropped Frames:  24,089

Jumbo Frames:  1,682,936

 

 

 Here are the relevant portions of how both 5448's are configured.  The iSCSI VLANs are not physically connected between the two.  I configured ports 1 - 24 all the same and have a connection from each MD3000i RAID module into each 5448 (4 cables total coming out of the MD3000i).  I have a two 5709 NIC's in each Oracle node server, and one cable coming out of each 5709 into each switch (one cable per iSCSI subnet per server).  Aside from a management connection to each 5448 in VLAN1, they have no other connections or purpose.

 

interface range ethernet g(1-24)
port storm-control broadcast enable
exit
interface ethernet g1
port storm-control include-multicast
exit
interface ethernet g2
port storm-control include-multicast
exit

interface ethernet g24
port storm-control include-multicast
exit
interface range ethernet g(1-24)
spanning-tree disable
exit
interface range ethernet g(1-24)
flowcontrol on
exit
interface range ethernet g(1-24)
no mdix
exit
port jumbo-frame
vlan database
vlan 243
exit
interface range ethernet g(1-24)
switchport access vlan 243
exit
interface vlan 243
name iSCSI_Oracle_Side_B
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
interface range ethernet g(1-18,21-24)
no negotiation
exit
iscsi target port 860 address 0.0.0.0
iscsi target port 3260 address 0.0.0.0
interface vlan 1
ip address 10.80.111.45 255.255.255.0
exit
ip default-gateway 10.80.111.1

 

 

(I've been dabbling with negotiation is why it is currently showing 19 & 20 (where MD3000i is) excluded...  Tried both though)

 

The server NICs and MD3000i are both set to MTU = 9000.  Flow control is enabled on the NIC's.

 

I did update the 5448's firmware prior to starting all of this a few weeks back.  They are at 2.0.0.32

 

When I mentioned Event Viewer, I was actually referring to looking to see if the iSCSI session itself was dropping rather than looking for dropped frames.  It is a little hard to see in the way I presented the data above, but you'll notice that they two flat file stats are very heavily weighted on two of the four MD3000i ports (which was my understanding of the way it should look).  Then if you look at the Oracle capture, you can see that it is split very evenly between the 4 ports of the MD3000i.  On the surface that seems ideal, but there's only 2GB of wiring coming to the server from the MD3000 (although I could fire up more if it's technically possible).  So I ultimately was just wondering if it was dropping iSCSI sessions, causing it to move on to one of the two failover sessions (and ultimately an evenly distributed load).  I didn't find anything indicating that though.

 

The only other thing I've noticed since my first post was that the Transmitted Pause Frames counters on each of the four switch ports that go to the MD3000 accumulate during the Oracle backup.  Between the 4 ports, a total of 479,345 pause requests were sent (125,749 + 126,596 + 113,460 + 113,540).  When running the single 7GB file backup, there were a total of 66 (0 + 0 + 0 + 66), and when running the job on the 8000 "average" files it was 48 (0 + 0 + 0 + 48).

 

Thanks again for the response and suggestions

 

 

 

 

 

 

 

 

 

Total Frames: 5,960,865

Dropped Frames:  98,243

Dropped Percent:  1.648%

 

Total Frames: 916,128

Dropped Frames:  18

Dropped Percent:  .0019%

 

Oops, something zigged when it should have zagged and some of my frame stats wound up out of place.  Each section should have included the following, but they wound up at the bottom of the message instead.

Backing up 2.4GB of small flat files (8,823 total files)

Total Frames:  483,128

Dropped Frames:  7

Dropped Percent:  .0014%

 

 

Backing up a single 7.4GB File

Total Frames: 916,128

Dropped Frames:  18

Dropped Percent:  .0019%

 

 

Backing up an Oracle Database

Total Frames: 5,960,865

Dropped Frames:  98,243

Dropped Percent:  1.648%

 

 

PowerConnect 5424 Ethernet switch

   I am assuming that you have some experience with network administration.   If none of this makes any sense to you, you should ask for help from someone who has experience with this stuff.
  
   You should be running the newest switch firmware version (2.0.0.35).   If not - download the package from the Dell support site and follow the upgrade directions.   The examples here are from the Console serial port on the rear of the switch.
  
*********************  
console> show version
SW version    2.0.0.35 ( date  27-Jan-2009 time  18:13:34 )
Boot version    2.0.0.0 ( date  12-Nov-2008 time  12:56:52 )
HW version    00.00.01
console>
*********************

Next, to have a known starting point, delete the current startup configuration and reboot the switch.   This will return the switch to the factory default configuration.

*********************
console> enable
console# delete startup-config
Delete startup-config [y/n]? y

console# reload
You haven't saved your changes. Are you sure you want to continue ? (Y/N):emotion-45: y
This command will reset the whole system and disconnect your current session. Do you want to continue ? (Y/N):emotion-45: y
Shutting down ...
*********************

When the switch prompts you for the setup wizard, say no.   You need to choose appropriate IP address, mask, and gateway for your network.   In the future, you can use telnet to manage the switch instead of the Console serial port.

*********************
Would you like to enter the setup wizard (you must answer this question within
60 seconds)? (Y/N):emotion-21: N

console> enable
console# configure
console(config)# port jumbo-frame
console(config)# spanning-tree mode rstp
console(config)# interface range ethernet g1-24
console(config-if)# flowcontrol on
console(config-if)# spanning-tree portfast
console(config-if)# exit
console(config)# interface vlan 1
console(config-if)# sntp client enable
console(config-if)# ip address xxx.xxx.xxx.xxx 255.255.255.0
console(config-if)# exit
console(config)# ip default-gateway xxx.xxx.xxx.xxx
console(config)# enable password level 15 xxxxxx
console(config)# line telnet
console(config-line)# password xxxxxx
console(config-line)# exit
console(config)#
*********************

This will configuration will give you a good working switch for your back-end iSCSI network.    Since this is a back-end network, there won't be any VoIP traffic, so you can remove this stuff from the default config.

*********************
console(config)# voice vlan oui-table remove 00036b
console(config)# voice vlan oui-table remove 00096e
console(config)# voice vlan oui-table remove 0001e3
console(config)# voice vlan oui-table remove 000fe2
console(config)# voice vlan oui-table remove 0060b9
console(config)# voice vlan oui-table remove 00d01e
console(config)# voice vlan oui-table remove 00e075
console(config)# voice vlan oui-table remove 00e0bb

*********************

And again, since this is a back-end iSCSI network, you really have no need to prioritize iSCSI, (there is no other traffic) so you can remove this stuff too.

*********************
console(config)# no iscsi enable
console(config)# no iscsi target port 860
console(config)# no iscsi target port 3260
*********************

Now that you have finished the configuration, copy the running config to flash.   You will need to reboot for the "jumbo-frame" command to take effect.

*********************
console# copy running-config startup-config
Overwrite file [startup-config] ?[Yes/press any key for no] y ....01-Jan-2000 00:04:10 %COPY-I-FILECPY: Files Copy - source URL running-config destination URL flash://startup-config
01-Jan-2000 00:04:17 %COPY-N-TRAP: The copy operation was completed successfully
Copy succeeded
console#
console# reload
This command will reset the whole system and disconnect your current session. Do you want to continue ? (Y/N):emotion-45: y
Shutting down ...

*********************
You may want to add a few more creative things to your config to customize it to your environment, but add them one at a time then test.
 

Thank you Thomas.  At first glance, I'm pretty much inline with those settings today.  It's sound advice though and I'm out of other things to try so I will give that a whirl.  One thing I did do differently was leave a management port in VLAN 1 and created a VLAN on each 5448 for the iSCSI segments to live in (with no physical interconnect or trunking between those two iSCSI VLANs).  I wouldn't think that would make any difference, but I'll certainly give it a try with your down-to-basics config and respond with the results.

Thanks again!

Well, I've made the changes and unfortunately that is all that changed.  I just left everything on VLAN1 on both switches and went over and physically unplugged the uplink I was using for the management VLANs on both, so they were both 100% standalone/disconnected.  Still getting what I would consider a high percentage of dropped frames on all 4 MD3000i controller ports.  I did the math over the weekend and when doing the Oracle job, it drops about 1.6% of the total frames.  When doing the other types of backups, it was something like 0.003%.  I had started a parallel thread in the Powerconnect forum here because I also found that the 4 ports on the two 5448's where the MD3000i plugs in were counting up a high number of "Transmitted Pause Frames" during these Oracle jobs.  Wasn't 100% certain if that meant that the switch itself was slowing things down, or if it was simply counting pause frames that were passing through it.  The same counter on the database server switch ports are clean.

We gave Symantec another try today and I think I'm able to let them off the hook now.  Wound up doing a straight RMAN backup from the database itself to a RAID set on the same database (so it took our Backup Exec Media server, the BUE Oracle Agent, and the network between them out of the equation).  We got the same results there...  Slow and lots of dropped frames.  Going bare-bones on the switch configs is letting me feel relatively good that they aren't at fault.  There's not a whole heck of a lot that can really be configured on the MD3000i network-wise, so I'm kind of left with the server's 5709 NICs and/or Oracle itself. 

The "pause" frames are the Layer-2 flow confrol mechanism used by the NICs and switch ports.    They send a pause frame when their recieve buffers are nearly full.   The pause frame is a request to the transmitting host to stop sending frames temporally, allowing the reciever some time to process some frames and free up more buffer space.   Pause frames indicate that a reciever is becomming congested.

A few pause frames is a good thing to prevent frame loss.     High numbers of pause frame are an indication of a network design problem - number of frames sent exceeds the available bandwidth of the path, or the ability of the receiver to process the frames at wire speed.   In an ethernet switch, if frames are arriving on multiple ports and destin for a single outgoing port, the switch will originate pause frames in an attempt to slow down the senders.

If a host is a bottleneck, you may need more CPU or memory resources, or possibly adjust the process priorities to improve the throughput.   In some NIC drivers, you can increase the number of recieve buffers (also called descriptors) - sometimes this helps, sometimes not.   There is a DOS command "netstat -s" that will give you some stats on the network prorocol stack.    In the switch, "show interface counters" will give you stats on the individual ports.

I hope this helps.

-Thomas

We made some progress on this, but I'm not quite sure I understand why.

To recap the setup, I have two Powerconnect 5448 switches.  Call them an "A" side and a "B" side.  They are strictly for iSCSI between this particular MD3000i and my two Oracle RAC servers.  The VLAN's that the iSCSI ports are in are not trunked or cross connected....  Totally isolated.  The only reason I had VLAN's was to have a management VLAN that I tie into the rest of my LAN.

So on the MD3000i, I have a cable from Controller 0, Port 0 to the Side A switch, as well as a cable from Controller 1, Port 0.  Controller 0, Port 1 and Controller 1, Port 1 both go to the Side B switch. 

On the servers, for the iSCSI connections, I had two ports on each server live...  One to the Side A switch, the other to the Side B and of course in the same VLAN as the MD3000i.

Microsoft's iSCSI initiator is used on both, and MPIO was not installed as per Dell's instructions (they have their own flavor that installs from the MD3000i CD).  On the "Side A" iSCSI NIC in the server, an iSCSI connection was created to each of the two MD3000i controller ports that are in the side A switch.  Likewise for the "Side B" NIC.  And likewise again for the other Oracle RAC server.  So one physical port on the server for Side A, two logical iSCSI connections to the Side A MD3000i ports.  Same for B.

Maybe I grossly misread, but that was my understanding of how it all should be wired and configured.  This setup spews dropped frames on the MD3000i (equally among all 4 controller ports) and causes pause frames to be transmitted from the switch ports where the MD3000i connects though.

Today, we tried removing the iSCSI initiator connections that go to the second controller on the MD3000i.  That left it set up as 1 physical port and one logical iSCSI connection on Side A, and the same for Side B.  Running the same Oracle backup as before, it ran right at 4 times faster and did not drop a single frame on the MD3000i. 

We wound up firing up a couple more NIC's on each of the servers and now have 1:1 physical ports and iSCSI connections (so 1 NIC connects to only 1 MD3000i controller port).  This didn't boost speed much, but is at least painfully redundant.  Traffic rotates pretty evenly among all 4 ports when doing the Oracle backups, but favors 1 and sometimes 2 NICs with non Oracle backups.  Under this 4-port configuration we are still not getting any dropped frames (yay!).

I suppose this is "solved" now, but I'm still pretty curious what the underlying cause is/might be.  ??

 

We run our a couple of non virtualized windows servers just as you describe.

 

The packet loss and slow speed,  sure sound like two subnets have gotten crossed either phycially or in your Vlan setup.

The way your doing it now surely would make that nearly impossible.

 

Some applications including some high level backup operations just mayt not be conducive to true load balancing either.

 

A very similar post on the VMware forums had me suggesting take it down to one connection and then run the backup and see what happens.   Your reconfiguration basically did this, but in a more permanent way.