UNSOLVED
Idea for LUN configuration- is this a good plan?
So I've got a 3-80 w/ FLAIR 22 on it and all 4 busses full. We've got a whole lot of LUNs to put in the next few months for a big rollout of VMware servers, and I'm trying to figure out how best to use the space. Most of the machines (60 or 70) will be a standard size of 30 Gb each, with a few large data base servers, the biggest being 500 Gb. Here's what I'm thinking. Please take a look and let me know if I'm walking off a cliff or something here, as I'm a bit green on the SAN admin thing.
What I'd like to do is something like the following: Build 4 RAID5 RAID groups of a half a tray each on each of the 4 back end busses. With 300 Gb drives they'd be 1878 each. Then I'd cut each of them into 7 268 GB LUNs. For the LUNs that will actually be added to the Storage Groups for the VMware hosts I'd make MetaLuns composed of 4 of 268 Gb LUNs, one from each of 4 RAID groups spread across the 4 BE busses. Once one of these big MetaLuns is presented to a VMware cluster VMFS will take care of who gets what space.
Why I think this is a good idea is that by using a MetaLun composed of a bunch of smaller LUNs spread across all of the BE busses is that the I/O would be able to use all 4 busses and the LUNs would be striped across (8 x 4 = ) 32 disks. Furthur, if there is a drive failure the rebuild time would only be what would be required for the smaller 268 Gb base LUN that holds the failed disk, not the much longer time that would be required had I used a 1072 Gb base LUN. It would also cut down on administration because I wouldn't have to keep carving, and keeping track of, dozens of smaller LUNs.
So does this all hold water? Am I right about the rebuild time being shorter? Would this really have better performance than just putting a bunch of base LUNs in the Storage Groups? And- the SPs in the 3-80 are running at about 55% on average. MetaLuns just have to use more processor cycles than a base LUN for I/O. Would I kill my SPs by doing this?
Thanks in advance for anything you can tell me.
What I'd like to do is something like the following: Build 4 RAID5 RAID groups of a half a tray each on each of the 4 back end busses. With 300 Gb drives they'd be 1878 each. Then I'd cut each of them into 7 268 GB LUNs. For the LUNs that will actually be added to the Storage Groups for the VMware hosts I'd make MetaLuns composed of 4 of 268 Gb LUNs, one from each of 4 RAID groups spread across the 4 BE busses. Once one of these big MetaLuns is presented to a VMware cluster VMFS will take care of who gets what space.
Why I think this is a good idea is that by using a MetaLun composed of a bunch of smaller LUNs spread across all of the BE busses is that the I/O would be able to use all 4 busses and the LUNs would be striped across (8 x 4 = ) 32 disks. Furthur, if there is a drive failure the rebuild time would only be what would be required for the smaller 268 Gb base LUN that holds the failed disk, not the much longer time that would be required had I used a 1072 Gb base LUN. It would also cut down on administration because I wouldn't have to keep carving, and keeping track of, dozens of smaller LUNs.
So does this all hold water? Am I right about the rebuild time being shorter? Would this really have better performance than just putting a bunch of base LUNs in the Storage Groups? And- the SPs in the 3-80 are running at about 55% on average. MetaLuns just have to use more processor cycles than a base LUN for I/O. Would I kill my SPs by doing this?
Thanks in advance for anything you can tell me.
- I use the following setup for 3 x ESX server with 40 VMs
3 raid groups made up of 6 x 300Gb 15K.
The MetaLUNs are made up of 3 x 110Gb luns, one LUN from each raid group.
The VMFS volume size is 330Gb, which is 10 x 30Gb size VM's with 30Gb left over for snapshots. This setup works ok. I worked out the VMFS volume size then worked out the LUN size and disk setup.
This ends up wtith 12 x 330Gb meta Luns and 20Gb left over on each raid group.
I would also watch scsi locks on the VMFS volume, the size and number of ESX host accessing the same LUN will affect this. Which is why I ended up with a 330Gb VMFS volume size. - Have you got any IOPS calculations/requirements as I think having potentially 60-70 VMs all on the same spindles could cause performance issues...
I fully agree with you on having smaller MetaLUNs is better then a 1TB FLARE LUN but with your proposed setup a single disk failure would disrupt all hosts, as all MetaLUNs share the same RGs.
Also from a performance point of view each physical spindle is capable of one op at a time which means ¿potentially¿ one server could hog all available disks¿
Therefore what I think I would do is to create 8 R5 4+1 RGs (2 per DAE, let¿s say spindles 0-4 and 5-9) instead. Furthermore, I would create half of the MetaLUNs on the 0-4 spindle RGs and the other half on the 5-9 spindle RGs.
This way also expanding the MetaLUNs would be easier as you would need to add another RG of same size (only 4+1 required) or use existing space in the RGs.
I would also distribute the hot-edge of the MetaLUN evenly amongst the RGs as well depending on the total spindle amount try to have hot spares distributed evenly among the backend busses. Also remember to divide the MetaLUNs evenly among the SPs (goes with the base LUN = also the hot-edge).
If you really want to try to optimize the LUN layout then put your busiest LUNs in the middle of the RG and the next busiest either side of them, that way the head has to travel as little as possible.
Finally remember to align your disks!
I hope this helps
Carl - Thanks guys! There are some good ideas here.
At this point I'm leaning away from the MetaLUN idea and instead thinking of building base LUNS with disks from different busses to spread the load a bit, and staying w/ more conventional 5+1 or 7+1 Groups.
And I've see it mentioned several places but I still haven't absorbed the thing about "Aligning disks". What's the big deal? Will I really see a difference? - I recommend you read the document that Glen referenced. It is a very good resource for all Clariion admins. In it you will see that binding LUNs across buses and DAEs will not provide performance gains. It can however help with the availability of LUNs and rebuild of LUNs.
There is also a selection on File System Alignment that you should read, it will explain it all. This is a recommendation for any intel based host attached to the array, and it will remove the measurable negative impact seen from not aligning a file system partition.
Aran - Thanks. I read that, and I've got it now.
But it's lead me to another question- in looking at a few of my LUNs I see lots and lots of stripe crossings. So I'm going to infer from that that I could increase performance and the efficiency of space usage by aligning my disks. But it doesn't look like this is something that can be done to an existing LUN.
So how do I do it? Create new, aligned LUNs and migrate to them? I sure hope there's an easier way than that. - You guessed right, it cannot be done to an existing LUN. The alignment of the partition can only be set when a partition is created on a disk with no existing partitions on it. The only way to move data from an existing LUN to a LUN with an aligned partition is to backup and restore the data. A LUN migration using the clariion virtual LUN technology will not work.
kelleg
6 Operator
•
4537 Posts
543
0
Posted January 29th, 2008 12:00
When using metaLUNs, try to keep all metaLUNs in the same original Raid Groups that you use for the first metaLUN and try not to put non-metaLUNs in these Raid Groups. Also, use components from all the original 4 Raid Groups, ie. don't create a metaLUN with three components rather than four - one from each Raid Group.
There is a very good Document on PowerLink called "EMC CLARiiON Best Practices for Fibre Channel Storage: CLARiiON Release 26 Update Firmware"
Home > Support > Technical Documentation and Advisories > White Papers > Configuration/Administration
or do a search on "emc clariion best practiices for fibre"
regards,
glen kelley