VNX File System replication speeds - How fast can it transfer?
We have 10 Gb networking between the Source 5700 and destination 5700 and almost no latency <1 ms. For all practical purposes they might as well be side by side in the datacenter plugged into the same switch.
I have never been able to get the transfer to even 1 Gb. I am replicating a new file system and current transfer is 44712 KB/sec which is .34 Gb/sec
No bandwidth limits have been applied to the interconnect.
What kinds of transfer rates are people seeing over what speed netowork links? to me this seems unreasonable.
Replication performance is largely bound by network speed, but in some cases, can be bound by disk latency. In cases I've observed, an idle VNX will replicate a single filesystem on a single GbE at the speed you've described (you didn't specify if you had 10GbE interfaces in your 5700s, only that you had 10GbE networking between them).. However, I can continue to add more replication sessions (other filesystems) and continue to scale until I can saturate the network pipe (you'll see transfer rates fall off all replications when you do so).
Re-tranmission timeouts (RTOs) can sometimes be a problem on fast networks. If your 10GbE switches have any QoS or other bandwidth limitations enabled, that could contribute as well.
Have you checked tcp stats to see your retransmit rates? Are they below .01%?
The thing is even from the beginning where there were barely any users on the vNX the speed was always stuck at 80000 KB/sec like we somehow had a 1Gb bottleneck somewhere. To answer your question we do have 10GbE interfaces in both vnx5700s we are using both configured together as an LACP. There is no Quality of service.
When one replication was running it would transfer @ 80000KB/sec
With 2, each was @ 40000 with 3, each was @25000.
The disks are SATA 2TB at source 3 TB SATA at destination.
There is no way we can saturate our network pipe.
I have run 6 256 thread emcopies while backups over Ethernet are running and replication and can never get more than 20% of their bandwidth used per networking.
Ouch... Take a look at the performance threads about SnapSure snapshots - I think they might have covered some of what's happen. This is the end result. It's the COW behavior that bogs down filesystem performance horrendously. Creating replication sessions with the SavVol in the same pool could be the culprit. Any chance you have a separate pool on the destination side you can drop the SavVol into with the -pool flag during set? Of course, your source SavVol will be on the source fs - no easy way to change that....
ok, i'll buy that for continuous replication but this is the first full copy, there should be no blocks to protect on the DR side (maybe a bunch of 0's). Dig dipper man, 48MB/s is laughable . I just tried to robocopy something to the same file system on the source, got 200MB/s ..so we know the plumbing is there but so far replicator is sucking wind.
I was under the impression that all the COW activity is on the DR SavVol, because of the internal checkpoints. Replicator creates two checkpoints on the source and two on the DR side. The block changes between the filesystem state and the internal source ckpt (the delta) gets transferred to the DR side and refreshes the destination internal ckpt. Unless I'm mistaken, the changes are all being written to the DR ckpt, then pushed into the filesystem. If for some reason we rebooted the DR datamover, we could resume the replication without having to start from scratch - we can start from the last consistent state of the checkpoint. Again, I could be off my rocker on this one....
if this is the first fully copy, there should be no COW activity ? Unfortunately i only have one pool on the target VNX so i can't test that theory. Target VNX is very lightly utilized by Block activity, i can't imagine if there was any COW it would be slowed down by disk. Any other ideas/tweaks ?
umichklewis
4 Apprentice
•
1190 Posts
1192
0
Posted October 31st, 2012 10:00
Replication performance is largely bound by network speed, but in some cases, can be bound by disk latency. In cases I've observed, an idle VNX will replicate a single filesystem on a single GbE at the speed you've described (you didn't specify if you had 10GbE interfaces in your 5700s, only that you had 10GbE networking between them).. However, I can continue to add more replication sessions (other filesystems) and continue to scale until I can saturate the network pipe (you'll see transfer rates fall off all replications when you do so).
Re-tranmission timeouts (RTOs) can sometimes be a problem on fast networks. If your 10GbE switches have any QoS or other bandwidth limitations enabled, that could contribute as well.
Have you checked tcp stats to see your retransmit rates? Are they below .01%?