Issue with random IO speeds with more than one NIC
I'm running into a weird issue with ScaleIO 2.0 on Windows Server 2012 R2, three nodes with 3x 960GB SanDisk Extreme Pro SSDs per node.
With just 1GbE, 10GbE or 40Gb Infiniband (with IPoIB), I get speeds as expected. For example, over 10GbE or IB I get around 40MB/s read and 20MB/s write random 4K.
However, when I enable a second NIC on the MDM and SDCs (on a different subnet and different physical NIC and switch, all separate), the speeds drop to <1MB/s read/write for the random 4K test.
What is going on? If I remove either network out, the speed goes back up.
The second network is an additional NIC on a different network. The networks involved are:
172.16.32.0/24 is the 1GbE management network, 1500 MTU
10.0.212.0/24 is the 10GbE network, 9000 MTU
10.0.213.0/24 is the IPoIB network over 40/32Gb IB, getting around 15-20Gb/s over IPoIB
Three nodes (hv1, hv2, hv3) that are all identically configured hardware wise and running Windows Server 2012 R2 Hyper-V in a hyperconverged setup with ScaleIO as both SDS+SDC on all nodes.
If I use one of the networks individually, I get speeds as such:
Seq Q32T1: 1937MB/s read, 1103MB/s write
Random 4K Q32T1: 216MB/s read, 116MB/s write
Seq: 941MB/s read, 430MB/s write
Random 4K: 45MB/s, 19MB/s
If I use both at the same time (or even add in the 1GbE network), speeds for Random 4K tank:
Seq Q32T1: 2344MB/s read, 997MB/s write
Random 4K Q32T1: 240MB/s read, 139MB/s write
Seq: 475MB/s read, 402MB/s write <-- about 1/2 speed read
However, the ping will fail as 9000 doesn't account for the UDP and ICMP overhead. With the MTU set at 9014 I can ping on the 212 10GbE subnet at 8972 bytes, switch is set to 9216.
On the IPoIB, the MTU is 4092 so I can't do a ping of 8972. This is an IPoIB best practice.
I also tried lowering the MTU down to 1514 on all interfaces and I still end up with very slow 4K writes when using any two interfaces. For example, the 1GbE and 10GbE interfaces (on different switches) both at 1514.
pawelw1
306 Posts
803
0
Posted April 21st, 2016 23:00
Hi,
How exactly is your second network configured? Are you using some kind of a NIC teaming, or you just add another NIC in different network?
You mentioned you tried to add second IPs to the MDM and SDCs, how about the SDS?
Can you please provide scli --query_all and scli --query_all_sds outputs?
Thanks!
Pawel