UNSOLVED

Kingsley Mok

updated

19 years ago

0

16700

February 7th, 2007 20:00

AX150i - SAN - Slow Data Througput - From SAN to TAPE?

 
Well, we've finally have the Virtual Drive in place for the PrimaryBackup Server (dedicated to backing up data to SAN solely) and the Snapshot of the Virtual for the Secondary Backup Server (dedicated to backup the snapshot directly to tape).
 
We've ran a Backup To Disk job test to the SAN.  For a 373 GB file, the data throughput averaged 708 MB / Min.  This is less than backing up to tape directly with averaged 900 MB to 975 MB / min.
 
After the backup job was complete, did a snapshot via Navsphere Utility as instructed in the online help document.
 
Now we are doing a backup of the snapshot to the tape to test the performance...
 
[....staring at awe in disbelief...]
...
...
...
.
Holy Smoking Tokes! :smileysurprised:
 
For a 373 GB backup of the Snapshot of Virtual Drive, transfer to tape is averaging 100 mb / Min?!?!??? :smileyindifferent:
 
At this rate, this will take 2 to 3 days to complete the job!!!!  I hate to see what a Snapshot of 1.8TB will take! :robotmad:
 
Worse, I look at Powerpath and the IO is showing one node active at 14.
 
One node active only?!?  At 14 only?!?!? :smileyindifferent:
 
This performance is unbearable!!!!  :robotmad:
 
 


Message Edited by Kingsley Mok on 02-07-2007 04:12 PM
  • JTGlenn

    96 Posts

    278

    0

    Posted February 8th, 2007 20:00

    What version of FLARE (OS on array) are you running?
    If you are not at 02.20.150.5.022, I would recommend upgrading.
     
  • 278

    0

    Posted February 8th, 2007 21:00

    Hi JTGlenn,
     
    Thanks for your input.  I've called in and checked with Dell on this issue.  The problem was with RFC 1122-delayed acknowledgement coming from the way TCP/IP works within Windows 2000 and Windows 2003.  Therefore, the problem is not Dell or Symantec's Backup Exec 10.1d.  It was the default design in Microsoft Windows 2000/2003. 
     
    As long as there is connection, you're fine.  It's all a matter of tune up like Windows XP Tune Ups we all see on the Internet--fiddling around with settings and modifying the registry.
     
    This slow data throughput problem is otherwise known as "Microsoft iSCSI slow performance."
     
    There is a "Hot Fix" for this issue.  There's not much on the "Hot Fix" as it is really just a tune up in the system settings for performance.
     
    I've applied the hot fix and it's soooooo nice to see that the transfer rate from SAN to TAPE is now up at 1.7 TB/min.  Waaaaaay better than 100 mb/min.  BIG difference.
     
     
    I will post up the document I've received from Dell on this so that others to review:
     
     
    ==========================
    ==========================
    From: blanked out
    Sent:  blanked out
    To:     Kingsley Mok
    Subject:  Mandatory TCP/IP settings for Microsoft iSCSI clusters to fix slow performance and unreliable failover
     
     
    The following is a Primus(R) eServer solution:

    ID: emc150702
    Domain: EMC1
    Solution Class: 3.X Compatibility

    Goal       Mandatory TCP/IP settings for Microsoft iSCSI clusters to fix slow performance and unreliable failover

    Goal       Recommended TCP/IP settings for Microsoft iSCSI configurations to fix slow performance

    Fact       Product: CLARiiON CX3-40c

    Fact       Product: CLARiiON CX3-20c

    Fact       EMC SW: PowerPath

    Symptom    Slow performance occurs during network congestion when attempting to perform high bandwidth read operations from a CLARiiON iSCSI array using the Microsoft iSCSI initiator.

    Symptom    Unreliable Microsoft iSCSI cluster failover during period of network congestion

    Symptom    Slow performance.

    Change     

    Run a program that performs high bandwidth read operations from an iSCSI CLARiiON using the Microsoft iSCSI initiator during network congestion

    Cause      

    Slow performance may occur during network congestion when RFC 1122-delayed acknowledgements extend the error recovery process.  In these situations, the default 200 millisecond delay on the acknowledgement can significantly impact read bandwidth.  PowerPath, by load balancing read requests across multiple array ports, increases the likelihood that simultaneous read completions from multiple ports will result in network congestion.  This increases the likelihood of experiencing the problem. 

    As specified in RFC 1122, Microsoft TCP uses delayed acknowledgments to reduce the number of packets that are sent on the media. Instead of sending an acknowledgment for each TCP segment received, TCP in Windows 2000 and later takes a common approach to implementing delayed acknowledgments. As data is received by TCP on a particular connection, it sends an acknowledgment back only if one of the following conditions is true:  

    • No acknowledgment was sent for the previous segment received.
       
    • A segment is received, but no other segment arrives within 200 milliseconds for that connection.  

    Typically, an acknowledgment is sent for every other TCP segment that is received on a connection unless the delayed ACK timer (200 milliseconds) expires. You can adjust the delayed ACK timer by editing the registry as outlined in the workaround below. 

    Fix        

    Modify the TCP/IP settings for the network interfaces carrying iSCSI traffic to immediately acknowledge incoming TCP segments.  This workaround solves the read performance issue.    The procedure to modify the TCP/IP settings is different for Windows 2000 servers and Windows 2003 servers.  Follow directions appropriate for the version you are running on your servers. 

    These setting are MANDATORY for all Microsoft iSCSI clusters and are RECOMMENDED for all Microsoft iSCSI configurations.

    Microsoft iSCSI clusters also require array software revision 3.22.xxx.5.511 or 3.24.xxx.5.007 or later.

    Note: These TCP/IP settings should not be modified for network interfaces not carrying iSCSI traffic as the increased acknowledgement traffic may negatively affect other applications.

    Caution! This workaround contains information about modifying the registry. Before you modify the registry, make sure to back it up and make sure that you understand how to restore the registry if a problem occurs. For information about how to back up, restore, and edit the registry, click the following link to view the following article in the Microsoft Knowledge Base:  http://support.microsoft.com/kb/256986/.

     

    On a server that runs Windows 2000 SP3 or later, follow these steps:

    1. Start Registry Editor (Regedit.exe).
       
    2. Locate and then click the following registry subkey:

      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces

      the interfaces will be listed underneath by automatically generated GUIDs like {064A622F-850B-4C97-96B3-0F0E99162E56}
       
    3. Click each of the interface GUIDs and perform the following steps:

      a. Check the IPAddress or DhcpIPAddress parameters to determine whether the interface is used for iSCSI traffic.  If not, skip to the next interface. 

      b. On the Edit menu, point to New and then click DWORD value.

      c.  Name the new value TcpDelAckTicks and assign it a value of 0.
       
    4. Exit the Registry Editor.
       
    5. Restart Windows for this change to take effect. 

    On a server that runs Windows Server 2003 SP1 or later, follow these steps:

    1. Start Registry Editor (Regedit.exe).
       
    2. Locate and then click the following registry subkey:

      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces

      the interfaces will be listed underneath by automatically generated GUIDs like {064A622F-850B-4C97-96B3-0F0E99162E56}
       
    3. Click each of the interface GUIDs and perform the following steps:

      a.  Check the IPAddress or DhcpIPAddress parameters to determine whether the interface is used for iSCSI traffic.  If not, skip to the next interface.

      b.  On the Edit menu, point to New, and then click DWORD value.

      c.  Name the new value TcpAckFrequency, and assign it a value of 1.
       
    4. Exit the Registry Editor.
       
    5. Restart Windows for this change to take effect.


    Message Edited by Kingsley Mok on 02-08-2007 05:13 PM
  • JTGlenn

    96 Posts

    278

    0

    Posted February 8th, 2007 22:00

    It's good to see support is on top of it.  I would still recommend the latest FLARE if you are not there.
  • orlantech

    1 Message

    278

    0

    Posted October 11th, 2007 14:00

    I applied this fix to my servers about 2 months ago and it was working great. My backups went from 30+ hours to 4 hours. However, I installed Service Pack 2 for Windows 2003 on my servers this week and I am back to the same problem of slow transfer rates. I verified that the registry key is still there. Are there any known issues with Service Pack 2 that would cause this?
  • gtrplyr

    9 Posts

    278

    0

    Posted November 27th, 2007 16:00

    you might try an I/O monitoring tool like this one that helps nail down the root cause of issues like this with AX or CX arrays:  http://www.vmware.com/appliances/directory/1084