UNSOLVED

dynamox

updated

11 years ago

D

dynamox

11 Legend

•

20419 Posts

•

87439 Points

0

1723

October 28th, 2015 08:00

Cluster expansion and change protection

Hello guys/gals,

Current Environment:

Version: OneFS 7.1.1.5

Cluster size: 11 x  X400 nodes  (all 3TB NL-SAS SED drives)

Current Protection: +2:1

We are getting ready to add two more nodes and would like to change protection to +3.  What is the best way to accomplish both tasks (cluster expansion and protection change) ? Should i join the two nodes first and them immediatly change protection ? Just trying to avoid un-nessary churn in terms of restripe.

Thank you

  • carlilek

    2 Intern

    •

    205 Posts

    679

    1

    Posted October 28th, 2015 09:00

    I'm not sure if you can really do it in one restripe. Because there's two separate jobs involved here... smartpools/setprotectplus (to set the protection level), and multiscan/autobalance (to rebalance the nodes). My choice is usually to run the smartpools first, because it will move some data at least to the new nodes, but you'll also need to run a multiscan after that to get it truly balanced.

  • chughh27

    39 Posts

    679

    0

    Posted October 28th, 2015 20:00

    Hello,

    Join 2 nodes first and then change protection level to +3 run smartpool job.

    Thanks

  • BernieC

    76 Posts

    679

    1

    Posted October 29th, 2015 00:00

    I agree with carlilek...if your cluster is relatively balanced before the 2 nodes are added then SmartPools/SetProtectPlus may keep it relatively balanced after completing its run. With 2 empty nodes, and a change in protection, expect to see full stripes of data and parity written across all 13 nodes. It's been my experience that this sort of change usually keeps the cluster relatively balanced so long as you don't go about deleting huge amounts of data in the middle of the restripe. So, add the nodes, change the protection level, let SP/SPP run and then (if needed), you can run MultiScan to finish balancing the cluster (and clearing orphaned blocks/inodes).

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    679

    0

    Posted October 29th, 2015 06:00

    Thank you.  So what you are recommending is this:

    1) Add 2 nodes to the cluster and wait for autobalance to finish (could be a couple of weeks)

    2) After #1 is complete, change protection to +3.  Will this automatically start setprotect job or do i have to manually kick it off ?

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    679

    0

    Posted October 29th, 2015 07:00

    Hi Dynamox

    the message from carlilek and Bernie would be that you can go

    from 9+2 to 10+3 in one single step by running SmartPools first,

    which will do all the restriping in a balanced manner.

    Not much left to do for a subsequent MultiScan operating at the then established 10+3 layout.

    No need for two full restripe journeys 9+2 ==> 11+2 ==> 10+3  as would happen with Multiscan first, SmartPool second.

    Makes sense?

    -- Peter

  • carlilek

    2 Intern

    •

    205 Posts

    679

    0

    Posted October 29th, 2015 09:00

    My understading, Peter, is that the multiscan will not make that restripe change until a smartpools has been run. So what I would do is add the nodes, cancel the automatic multiscan/autobalance, make the change in the smartpools interface, then run the smartpools job.

    --Ken

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    679

    0

    Posted October 29th, 2015 19:00

    Thanks Ken, of course the +3 protection needs to be set at some point in time before starting the SmartPools job.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    679

    0

    Posted October 30th, 2015 08:00

    Peter, when you say "at some point" , what do you mean by that ?  Goal is to avoid multiple restripes. So do i join both nodes to the cluster, immediately cancel autobalance job, go into pool properties and change protection to +3  and then start setprotect job manually ?

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    679

    0

    Posted October 30th, 2015 09:00

    Yes that's a reasonable plan. Good luck!