I would like to hear your opinion on the possibility of further maximizing our Celerra CIFS performance.
Typical use scenario varies from large sequential reads to large sequential writes, to an indexing/searching process on the data (where the I/O pattern is fairly unpredictable).
Setup is as follows:
- Celerra NS40G, 5.6.44-4
- Backend is a Clariion CX3-80
- Two DAE's with 15x500Gb SATA
- Each DAE has three RAID5 (4+1 disks) RAID groups, in total 6 RAID groups.
- Each RAID group has 4 LUN's which are all exported to the Celerra
- At the Celerra the first LUN of every RAID group was put in a stripeset (stripesize=64k).
- The same for every second, every third, and every fourth LUN.
- That gives 4 stripesets at the Celerra
- The stripesets are put together in a storage pool
- On the storage pool, one big filesystem is created, and shared via a dedicated virtual datamover on the Celerra
- The Celerra is connected to a Cisco 2970 Gigabit switch by via an 802.3ad aggregated link of 3x 1Gbit Ethernet.
- Clients are connected via 1Gbit Ethernet to the same switch
Currently, in a multi-client test, we reach a cumulative read speed of 220-230Mb/s.
What do you think of this setup, and in which areas would we have to look to achieve more performance gain?
Thanks for your answer. We are indeed planning to have a look at MPFS.
But I am still wondering if there isn't anything we can do in the Celerra itself. For example: - Would it make a difference to use a metavolume in stead of a storage pool? (What is the difference anyway?) - What mount properties (sync writes, prefetch, direct writes, oplocks, virus checking etc) are advised? - What advanced datamover parameters (capabilities, fastRTO, indBlkHashSize, prealloc, updatemode) are interesting?
considering you only got 30 SATA disks your numbers dont look bad
the probably best shot at improving this significantly would be to use MPFS over FC or ISCSI
with MPFS the driver can do very aggressive caching and prefetching and drive the Clariion harder with large I/Os
Ask your local EMC contact to give you a copy of the MPFS Technical Presentation - it contains better descriptions on how it works and some performance comparisons.
You did not mention anything about this, but: The SATA raidgroups should be "balanced" across the SPs, f.e. all Luns from the first one belonging to SPA, all Luns from the second one belonging to SPB, ...
We do not use your mentioned stripesize of 64k, but 32k. But I can not give you an idea, how far this would change your performance numbers. This would also be dependend from the type of IOs you are doing (sequential, random, single threads, multiple threads, ...). I assume you are doing random IO, because when using multiple clients against the same physical disks, there are no longer sequential IOs from the disc perspective. But random IO is the weak point of SATA discs ...
In a standard config, you would only present 2 Lun per raidgroup to the Celerra (as long as they are smaller than 2 TB)
Would it make a difference to use a metavolume in stead of a storage pool? (What is the difference anyway?)
If you are speaking of a Clariion Metalun, this is no option, because the Celerra does not support it. The difference is the volume handling. The Celerra is designed to do the volume handling by itself using its volume manager, f.e. striping and slicing using backend volumes. A Celerra storage pool is just a container of a number of similar storage objects, f.e. plain luns or "volumes".
Currently, in a multi-client test, we reach a cumulative read speed of 220-230Mb/s.
Is this 220 Megabyte/s or Megabit/s?
What real bandwidth is your network and the clients used for the test able to handle? How is the load distributed across the trunk during the test? What tools are used for testing or how is the testing done?
"The SATA raidgroups should be "balanced" across the SPs, f.e. all Luns from the first one belonging to SPA, all Luns from the second one belonging to SPB, ..."
Out of interest, does this recommendation also apply to DAE with FC disks. I have 2 LUNs per raid group, but 1 LUN on SPA and the other LUN on SPB.
You did not mention anything about this, but: The SATA raidgroups should be "balanced" across the SPs, f.e. all Luns from the first one belonging to SPA, all Luns from the second one belonging to SPB, ...
we had discussions about this in Clariion forum and it was mentioned by EMC folks that it's no longer a requirement as the new SATA II drives are using the new "paddle" card.
You did not mention anything about this, but: The SATA raidgroups should be "balanced" across the SPs, f.e. all Luns from the first one belonging to SPA, all Luns from the second one belonging to SPB, ...
True, I forgot to mention. What we did is link all LUN's on all RAID groups on DAE#1 to SPA, and all LUN's on all RAID groups on DAE#2 to SPB. I would think this would have the same effect because of the striping. Or do you think dividing the load by LUN will be beneficial to dividing the load by DAE?
We do not use your mentioned stripesize of 64k, but 32k.
64k seems to be recommended for SATA, that's why we chose this.
In a standard config, you would only present 2 Lun per raidgroup to the Celerra (as long as they are smaller than 2 TB)
I have changed to 2 LUN's per RAID group now, and will test this out.
If you are speaking of a Clariion Metalun, this is no option, because the Celerra does not support it.
No, I was talking about Celerra metavolumes.
Is this 220 Megabyte/s or Megabit/s?
Megabyte/s
What real bandwidth is your network and the clients used for the test able to handle?
The network connections of the clients are able to handle 1Gbit flat, the trunk I am not sure.
How is the load distributed across the trunk during the test?
This is something we need to investigate further. I would not expect a bottleneck here, but I'd have to verify that.
What tools are used for testing or how is the testing done?
We use IOmeter on 6 different Windows clients simultanuously. We do reading and writing tests with 64k and 1M blocksizes. Also we do an indexing test with irregular I/O patterns.
... True, I forgot to mention. What we did is link all LUN's on all RAID groups on DAE#1 to SPA, and all LUN's on all RAID groups on DAE#2 to SPB. I would think this would have the same effect because of the striping. Or do you think dividing the load by LUN will be beneficial to dividing the load by DAE?
We do not use your mentioned stripesize of 64k, but
32k. 64k seems to be recommended for SATA, that's why we chose this.
This is not best practice
If you are speaking of a Clariion Metalun, this is
no option, because the Celerra does not support it. No, I was talking about Celerra metavolumes.
Your original question was: "- Would it make a difference to use a metavolume in stead of a storage pool? (What is the difference anyway?)" The difference is: If you do not use a storage pool, then you have to do the volume management (FS and Checkpoint creation and extension) manually. Then you cut off the automated volume management. F.e. if you want to create a FS using a pool you can use the GUI or the nas_fs ... -create size= pool= without a pool, you need to create a Meta of the desired size (using a few commands) and then create the FS on top of the meta. I do not think autoextend is working in this case.
What real bandwidth is your network and the clients
used for the test able to handle? The network connections of the clients are able to handle 1Gbit flat, the trunk I am not sure.
I was asking for the real bandwidth, not the advertised speed. So are the clients/the network able to deliver 120 Megabyte/s against the Celerra network port? Or is it just about 70 Megabyte/s which would explain the 220 MB/s you are mentioning for 3 Ports?
kouprie-fox-it.
4 Posts
450
0
Posted September 2nd, 2009 09:00
Thanks for your answer. We are indeed planning to have a look at MPFS.
But I am still wondering if there isn't anything we can do in the Celerra itself. For example:
- Would it make a difference to use a metavolume in stead of a storage pool? (What is the difference anyway?)
- What mount properties (sync writes, prefetch, direct writes, oplocks, virus checking etc) are advised?
- What advanced datamover parameters (capabilities, fastRTO, indBlkHashSize, prealloc, updatemode) are interesting?
Thanks again