UNSOLVED

Dzmitryj

updated

13 years ago

D

Dzmitryj

5 Posts

0

3346

June 12th, 2013 08:00

What is a proper way to move a node from the cluster

What is a proper way to remove a node from a cluster ?

The node has a "down" state in the cluster and smartfail procedure does not work.

Thanks  a lot

  • 1255

    0

    Posted June 12th, 2013 13:00

    Hi

    Can you shed more details on why SmartFail is not working? That is the proper way to remove a node with out compromising data integrity.

  • Dzmitryj

    5 Posts

    1255

    0

    Posted June 12th, 2013 15:00

    The node shows in the web interface like down (red circle and like being smartfailed (letter S in the sircle).  You can see infiniband interfaces of this node in the web UI as well but not the ethernet ones. After smartfail initiation procedure  has been initiated couple of hours has been passed.

  • osaddict

    110 Posts

    1255

    0

    Posted June 12th, 2013 15:00

    What type of nodes and how many do you have? It can take many hours or even days to smartfail a node out. All the data that was on all the drives in that node has to be re-calculated. Which is why we usually try to avoid it, unless you are retiring hardware or shrinking a cluster. If the chassis has failed, we can do a "disk tango", which allows us to move the drives to a new chassis to without having to rebuild all that data.

  • Zubrania

    2 Posts

    1255

    0

    Posted June 13th, 2013 00:00

    I have 4x NL_ nodes. Cluster has no data yet.

  • Zubrania

    2 Posts

    1255

    0

    Posted June 13th, 2013 01:00

    Hi Peter

    isi job status

    No running jobs.

    No paused or waiting jobs.

    No failed jobs.

    Recent job results:                                                            

    Time            Job                        Event                         

    --------------- -------------------------- ------------------------------

    06/12 14:37:13  ShadowStoreDelete[5]       Succeeded (LOW)

    06/12 14:37:26  MultiScan[7]               Succeeded (LOW)

    06/12 14:37:37  SetProtectPlus[6]          Succeeded (LOW)

    06/12 15:39:40  FlexProtect[8]             Succeeded (MEDIUM)

    06/12 17:05:26  FlexProtect[9]             Succeeded (MEDIUM)

    06/12 17:56:40  MultiScan[10]              Succeeded (LOW)

    06/12 22:29:03  FlexProtect[11]            System Cancelled

    06/13 17:05:35  FlexProtect[12]            Succeeded (MEDIUM)

    I can not ssh to the node, Only console access is available,

    What I can see from the console of the node is that it boots to recovery with the following error

    Test Journal exited with error - Checking Isilon Journal integrity...          

    The journal state is internally consistent, but the journal state does not matc.

    luckily the node has been successfully smartfailed from the cluster

    I think it would be a good idea to reinitialize the node

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    1255

    0

    Posted June 13th, 2013 01:00

    What does (on any node)

    isi job status

    say, some job still ongoing, waiting or failed?

    Can you login with ssh to the node in question?

    (From another node via IB: ssh - )

    And then check the  /var/log/messages for local entries?

    Peter

  • Peter_Sero

    6 Operator

    •

    1169 Posts

    1255

    1

    Posted June 13th, 2013 02:00

    Good. You might double-check in the WebGUI"

    "When the smartfail process is complete, the node no longer appears on the Cluster Status page."


    If the node is still listed, try  Cluster->Cluster Management->Remove Node

    (not sure whether this is what you did in the first place, or just initiated the smartfail.)


    Peter