AX150i - SAN - Slow Data Througput - From SAN to TAPE?
Well, we've finally have the Virtual Drive in place for the PrimaryBackup Server (dedicated to backing up data to SAN solely) and the Snapshot of the Virtual for the Secondary Backup Server (dedicated to backup the snapshot directly to tape).
We've ran a Backup To Disk job test to the SAN. For a 373 GB file, the data throughput averaged 708 MB / Min. This is less than backing up to tape directly with averaged 900 MB to 975 MB / min.
After the backup job was complete, did a snapshot via Navsphere Utility as instructed in the online help document.
Now we are doing a backup of the snapshot to the tape to test the performance...
[....staring at awe in disbelief...]
...
...
...
.
Holy Smoking Tokes! :smileysurprised:
For a 373 GB backup of the Snapshot of Virtual Drive, transfer to tape is averaging 100 mb / Min?!?!??? :smileyindifferent:
At this rate, this will take 2 to 3 days to complete the job!!!! I hate to see what a Snapshot of 1.8TB will take! :robotmad:
Worse, I look at Powerpath and the IO is showing one node active at 14.
One node active only?!? At 14 only?!?!? :smileyindifferent:
This performance is unbearable!!!! :robotmad:
Message Edited by Kingsley Mok on 02-07-2007 04:12 PM
Thanks for your input. I've called in and checked with Dell on this issue. The problem was with RFC 1122-delayed acknowledgement coming from the way TCP/IP works within Windows 2000 and Windows 2003. Therefore, the problem is not Dell or Symantec's Backup Exec 10.1d. It was the default design in Microsoft Windows 2000/2003.
As long as there is connection, you're fine. It's all a matter of
tune uplike Windows XP Tune Ups we all see on the Internet--fiddling around with settings and modifying the registry.
This slow data throughput problem is otherwise known as "Microsoft iSCSI slow performance."
There is a "Hot Fix" for this issue. There's not much on the "Hot Fix" as it is really just a tune up in the system settings for performance.
I've applied the hot fix and it's soooooo nice to see that the
transfer rate from SAN to TAPEis now up at
1.7 TB/min. Waaaaaay better than
100 mb/min. BIG difference.
I will post up the document I've received from Dell on this so that others to review:
==========================
==========================
From:
blanked out
Sent:
blanked out
To: Kingsley Mok
Subject: Mandatory TCP/IP settings for Microsoft iSCSI clusters to fix slow performance and unreliable failover
Goal Mandatory TCP/IP settings for Microsoft iSCSI clusters to fix slow performance and unreliable failover
Goal Recommended TCP/IP settings for Microsoft iSCSI configurations to fix slow performance
Fact Product: CLARiiON CX3-40c
Fact Product: CLARiiON CX3-20c
Fact EMC SW: PowerPath
Symptom Slow performance occurs during network congestion when attempting to perform high bandwidth read operations from a CLARiiON iSCSI array using the Microsoft iSCSI initiator.
Symptom Unreliable Microsoft iSCSI cluster failover during period of network congestion
Symptom Slow performance.
Change
Run a program that performs high bandwidth read operations from an iSCSI CLARiiON using the Microsoft iSCSI initiator during network congestion
Cause
Slow performance may occur during network congestion when RFC 1122-delayed acknowledgements extend the error recovery process. In these situations, the default 200 millisecond delay on the acknowledgement can significantly impact read bandwidth. PowerPath, by load balancing read requests across multiple array ports, increases the likelihood that simultaneous read completions from multiple ports will result in network congestion. This increases the likelihood of experiencing the problem.
As specified in RFC 1122, Microsoft TCP uses delayed acknowledgments to reduce the number of packets that are sent on the media. Instead of sending an acknowledgment for each TCP segment received, TCP in Windows 2000 and later takes a common approach to implementing delayed acknowledgments. As data is received by TCP on a particular connection, it sends an acknowledgment back only if one of the following conditions is true:
No acknowledgment was sent for the previous segment received.
A segment is received, but no other segment arrives within 200 milliseconds for that connection.
Typically, an acknowledgment is sent for every other TCP segment that is received on a connection unless the delayed ACK timer (200 milliseconds) expires. You can adjust the delayed ACK timer by editing the registry as outlined in the workaround below.
Fix
Modify the TCP/IP settings for the network interfaces carrying iSCSI traffic to immediately acknowledge incoming TCP segments. This workaround solves the read performance issue. The procedure to modify the TCP/IP settings is different for Windows 2000 servers and Windows 2003 servers. Follow directions appropriate for the version you are running on your servers.
These setting are MANDATORY for all Microsoft iSCSI clusters and are RECOMMENDED for all Microsoft iSCSI configurations.
Microsoft iSCSI clusters also require array software revision 3.22.xxx.5.511 or 3.24.xxx.5.007 or later.
Note: These TCP/IP settings should not be modified for network interfaces not carrying iSCSI traffic as the increased acknowledgement traffic may negatively affect other applications.
Caution! This workaround contains information about modifying the registry. Before you modify the registry, make sure to back it up and make sure that you understand how to restore the registry if a problem occurs. For information about how to back up, restore, and edit the registry, click the following link to view the following article in the Microsoft Knowledge Base: http://support.microsoft.com/kb/256986/.
On a server that runs Windows 2000 SP3 or later, follow these steps:
Start Registry Editor (Regedit.exe).
Locate and then click the following registry subkey:
I applied this fix to my servers about 2 months ago and it was working great. My backups went from 30+ hours to 4 hours. However, I installed Service Pack 2 for Windows 2003 on my servers this week and I am back to the same problem of slow transfer rates. I verified that the registry key is still there. Are there any known issues with Service Pack 2 that would cause this?
JTGlenn
96 Posts
278
0
Posted February 8th, 2007 20:00