UNSOLVED

hei-san

updated

14 years ago

HS

hei-san

19 Posts

0

4977

May 14th, 2013 14:00

Replication stopped after losing power to primary control station

During a maintenance window, we needed to swap out our main network LAN switch at our DR site.  In the process of trying to get the old switch out of the rack, we accidentally unplugged our CS0 device.  We did not notice it off at first.  Once we did we pligged it back in.   I thought I managed to get that as the primary again, but I may have done something wrong.  And now, when i checked my replication pairs, they are saying they last updated May 10, the same night the CS0 went offline...

So, how do I make sure my CS0 is set to primary?

Why do these CS seem to be very sensitive when one fails?  Or do I just not understand how this works... (which could very well be the case)

Now that replication is in some "paused" state, how do I get it going again?  Will it just work once the CS0 and CS1 are straightened out?

  • christopher_ime

    6 Operator

    •

    1962 Posts

    1200

    1

    Posted June 25th, 2013 18:00

    Hei-san,

    Firstly, please consider moving this question to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time.  You can do so by selecting "Move" under ACTIONS along the right.  Then search for and select: "Celerra Support Forum" or "VNX Support Forum" depending on the generation of array you own but the steps are the same (for the sake of this response I will assume VNX but by using the commands, this will be portable between the two).  Here also is a recent post providing detailed instructions on how to move this post and will assure that you get the most (quality and quantity) feedback by the community.

    EMC Community Network Frequently Asked Questions

    The issue here is that you likely don't have an IPalias configured so your replication sessions rely on a specific Control Station to be primary/active.  In a dual CS install, an IPalias is highly recommended for convenience (when logging into Unisphere or the shell you only need to remember one IP and not worry about which is the current primary/active) and for Replication, I will go on record as suggesting it is required.  When you setup the CNS (Celerra Network Server) peer relationship which consists of: peer name, IP, and shared secret, it was likely configured with the actual IP of a single Control Station.  Instead, in a dual CS setup, you should setup and use the IPalias of the remote array which is a single IP that represents simply the currently assigned primary/active CS; just like the Cluster (Virtual) IP you are familiar with on traditional clusters.

    Creating (and listing) an IPalias

    ===========================

    The initialization tools used during the initial install of the array do not create an IPalias; you need to create and test it manually.  You can review the short steps in the "VNX System Operations" guide:


    https://mydocs.emc.com/VNXDocs/System_Operations.pdf

    You can also can also search for the document by name on support.emc.com but I was looking to mix it up a bit and bring attention to the My Documents (UPD) portal also:

    https://mydocs.emc.com/VNX/requestMyDoc.jsp

    With the document downloaded, skip to the section "Create an IP alias".

    In relation to Replicator, this is discussed in the "Using VNX Replicator" document which you can search for on support.emc.com by name.  Specifically you will want to review the following:

    Section: "Using dual Control Stations"

    Sub-Section: "IP aliasing"

    Steps are basically (run as root):

    1) Verify first if you have an IPalias created:

    /nas/sbin/nas_config -IPalias -list

    2) For each dual control station configuration, create an IPalias

    /nas/sbin/nas_config -IPalias -create 0

    3) Verify again that it was correctly created (and state = UP):

    /nas/sbin/nas_config -IPalias -list

    You can now log into Unisphere or CLI via the IPalias.

    Determining which is primary/active and which is secondary/standby

    ======================================================

    Now to your other question about determining which Control Station is currently the assigned primary, there are several ways:

    1) If you were to setup and use the IPalias, it will of course always log you into the primary/active CS

    2) If you use the actual IP of either CS, if after logging in it drops you immediately to a command prompt that is also indication of the same. 


    3) If instead you get a prompt suggesting the following, this would be what you'd expect of the secondary:

    [...]

    NAS service is not running. Root privileges are required to run this command.

    Please wait...

            *** slot_1 secondary control station ***

    Would you like to log into the Primary Control Station? (yes or [no])

    [...]

    3) Additionally when logged into either, you could run:

    /nasmcd/sbin/getreason

    NOTE: I provide the absolute path of /nasmcd (instead of /nas) which will work on either control station.  /nas is only mounted on the primary control station which isn't the case on the secondary (NAS service isn't running).  Therefore, consider the above command more portable.

    In the output, always remember that the slot_# represents the physical location of the CS (and data movers) and never changes


    CS0 = slot_0

    CS1 = slot_1

    The only things that will change is the reason code:

    10 = primary control station

    11 = secondary control station

    For instance, if CS0 was the primary the response would be:

    10 - slot_0 primary control station

    11 - slot_1 secondary control station

    If relationship is switched (CS1 is primary) you would see:

    11 - slot_0 secondary control station

    10 - slot_1 primary control station

    Failing over manually

    =================

    As for failing over manually, you have two options (run as root):

    1) /nas/sbin/cs_standby -failover

    a) run from the primary/active control station

    b) current primary CS will reboot in the process

    2) /nasmcd/sbin/cs_standby -takeover

    a) run from the acting standby, but should only be used when its peer is inaccessible

    b) again note the use of /nasmcd since /nas would not be mounted on the secondary CS

    This is covered in the following (as well as the "Command Line Reference for File" document and will mix it up a bit and provide the URL to a search string on support.emc.com):

    "Configuring Standbys on VNX"

    https://mydocs.emc.com/VNXDocs/Standbys.pdf

    "Command Line Reference for File"

    https://support.emc.com/search/?resource=DOC_LIB&AlloftheseWrds=Command%20Line%20Reference%20for%20File&SearchWithin=true&adv=y

    Failover and "Failback" (note the quotes)

    ==================

    Unlike the data movers which will not failback automatically (always a manual process to restore), with the Control Stations the system will automatically failover between the two as needed.  Once configured with an IPalias and properly referenced via browser bookmarks, putty shortcuts, scripts, Replicator, etc, there also really isn't any urgency to failover again just so that CS0 is primary.  In fact, when discussing Control Stations, I don't even use the term failback as the hardware is identical and one shouldn't be considered inferior to the other.  Please note this discussion changes when talking about data movers but I digress.

    NOTE: the exception is when you go to do any NDU task like upgrading code.  In fact, when using the USM, it clearly states not to use the IPalias but instead the IP of CS0 (which implies it needs to be the primary)

    What isn't available if primary Control Station is down (other than ability to manage temporarily)

    ===========================================

    In anticipation of another question, when a primary control station is down anything that relies on the NAS service will not be performed automatically; however, the CIFS shares and/or NFS exports as served via the data movers will remain accessible throughout so you aren't down (per se).  The process of failing over also can take 10 to 15 minutes before the primary is assigned, so it is important to understand what is affected.  As for tasks that do require an active control station, those include (but are not limited to) things such as:

    1) Scheduled checkpoints

    2) Automatic extension of filesystems

    3) Failover of data movers

    etc.

    Testing

    ======

    It goes without saying that you should also test the IPalias by performing a series of failovers: CS0 -> CS1 and CS1 -> CS0 to make sure it works as expected.  From experience though on rare occasions the IPalias doesn't come online on its own, but is easily rectified on the active CS using the ifup command (a consequence of its Linux foundation)

    You can see this by running the following command (as root):

    /nas/sbin/nas_config -IPalias -list

    1) alias = eth3:#

    2) state = UP or DOWN

    If DOWN, only need to run the following command

    /sbin/ifup eth3:#

    NOTE: hash symbol represents the (Linux kernel) virtual IP assigned to eth3

    Replication considerations

    =====================

    As for your replication sessions, you will want to review the IP assigned to the CNS (Celerra Network Server) peer objects on each array.  Remember, a replication session is comprised of:

    1) CNS relationship (name assigned to remote peer, IP, and shared secret)

    2) Interconnect

    3) FS or VDM session

    Run the following command on both arrays:


    nas_cel -list

    Specifically you'll be interested in:

    1) name

    - name of the peer which btw does not have to match exactly the name of a CS which allows you to instead use an IPalias

    2) net_path

    - I'm going to assume this is referencing the actual IP of one of the control stations but as you've now learned should be the IPalias

  • cincystorage

    2 Intern

    •

    467 Posts

    1200

    0

    Posted June 26th, 2013 02:00

    Great post,  I was about to start working on a blog post about this very subject!

  • christopher_ime

    6 Operator

    •

    1962 Posts

    1200

    0

    Posted June 26th, 2013 09:00

    Thanks Mark, much appreciated.

  • darrellx

    10 Posts

    1200

    0

    Posted June 26th, 2013 16:00

    Chris

    Thanks for the comprehensive update. However, this leads to a few more questions.

    You say that it is confusing to setup the IPalias to use one of the actual CS IPs, but it can be done.

    Since I have a dual CS setup that was configured with no IPAlias, would I be better off in the long-run to use a different IP for the alias, or just use the CS0 IP?  Since there are only two of us that do any administration on the system, I am looking for te easy route, and I doubt that confusion will be an issue.

    Once the IPAlias has been configured, do both control stations need a reboot?

    Thanks

    darrell

  • christopher_ime

    6 Operator

    •

    1962 Posts

    1200

    0

    Posted June 26th, 2013 17:00

    Nevermind, turns out that the Command Reference for File (mentioned above) has also the session output demonstrating the command so no need to wait for me to return home to capture a session:

    "Command Line Reference for File"

    https://support.emc.com/search/?resource=DOC_LIB&AlloftheseWrds=Command%20Line%20Reference%20for%20File&SearchWithin=true&adv=y

    [...]

    EXAMPLE #1 To create an IP alias for the Control Station, type:

    # /nas/sbin/nas_config -IPalias -create 0

    Do you want slot_0 IP address <1.2.3.4> as your alias [yes or no]: no

    Please enter an IP address to use as an alias: 1.2.3.6

    Do you want slot_0 IP address <1.2.3.4> as your alias [yes or no]: yes

    Please enter a new IP address for slot_0: 1.2.3.6

    [...]


    You will be interested in the second half.  Note the flow:


    1) yes is answered

    2) You are then prompted for a new IP address for slot_0


    So, you still need 3 IP's.  Now, from the example in the PDF, you can tell:


    1) CS0/slot_0 is the current primary Control Station


    2) This was run from CS0

    Otherwise if run instead from CS1 because it was the primary CS, the prompts above would ask on behalf of slot_1 (and you be stealing and reassigning the IP for CS1 if you said yes)

  • christopher_ime

    6 Operator

    •

    1962 Posts

    1200

    0

    Posted June 26th, 2013 17:00

    darrellx,

    So just to clarify, you still need 3 IP's.  You are just stealing either slot_0 or slot_1 IP's (depending on which CS is primary and that you are running it from), but still need to reassign an IP to the CS you just stole it from.  Sorry that was my point about it being confusing, but I realize now that the only thing confusing was my use of the word confusing

    I would say unless you just had so many scripts, browser bookmarks, Putty shortcuts, etc. that you couldn't feasibly update, then do so, but if that isn't the case then I'd leave your current setup as-is and assign the new IP for the IPalias.  Even hei-san can update the CNS object with the new IP via the command:

    nas_cel -modify

    No it doesn't require a reboot; however, I would remind you to keep the following command handy just-in-case:

    /nas/sbin/nas_config -IPalias -list

    1) alias = eth3:#

    2) state = UP or DOWN

    If DOWN, only need to run the following command

    /sbin/ifup eth3:#

    NOTE: hash symbol represents the (Linux kernel) virtual IP assigned to eth3


    However, as recommended above, I'd highly suggest testing failover CS0 -> CS1 and CS1 -> CS0.  During that time, 10-15 minutes may elapse without a primary control station assigned.  When I need to run the command above (ifup) it is during the initial failover test.  It seems to behave after that.



    Later tonight, I'll paste a session demonstrating this.  I'm not in a position currently (client network) to log into a lab.


  • christopher_ime

    6 Operator

    •

    1962 Posts

    1200

    0

    Posted June 26th, 2013 20:00

    darrell,

    I removed reference from the main post to my comments about setting the IPalias to that of either CS0 or CS1 (when prompted).  The follow-up responses discuss the actual behavior in detail and a use case of why you may want to do so.

    Thanks for the feedback.

  • hei-san

    19 Posts

    1200

    0

    Posted July 23rd, 2013 13:00

    Another odd effect on this whole problem is that my replication speeds have gone down too.  Part of the changes made the night this all broke was swapping out the core LAN switch at the DR site where we lost our CS.  We had a 48 port 10/100 switch in there that we removed and installed a 48 port 10/100/1000 switch.  Our uplink at this site runs at 100Mbps.  I have looked at the network ports I used prior to the change (with trends collected in Cacti) and I would get up in the 60-80Mbps speeds.  Now that replication is moving again, I only seem to get up to 20-30Mbps...  I can post some graphs, but is there anything I need to double check on my network settings on the SAN or recommendations for switch settings?  I have 4 ports per DM and they are setup as LACP group.

  • hei-san

    19 Posts

    1200

    0

    Posted July 23rd, 2013 13:00

    Thanks for your help!!!  I thought the EMC tech that came onsite had done this, but the IP I gave him never did work and I never pursued it... anyway, I have added the alias information like you described and now I have a common IP address for my control stations at each site VNXA = xx.xx.xx.154 and VNXB = xx.xx.xx.174 (both SANs are in the same subnet streched across our layer 2 metro ethernet setup).

    Here is what "nas_cel -list" shows on primary SAN:

    [root@VNXA-CS0 sbin]# nas_cel -list

    id    name          owner mount_dev  channel    net_path                                      CMU

    0    VNXA-CS0      0                          xxx.xx.x.150                                  APM001203020982007

    1    VNXB-CS0      0                          xxx.xx.x.170                


    I see above you mention using "nas_cel -modify" command to update the CNS.

    Do I do this on each side (i.e.  each SAN)?  What about updating host names?

  • christopher_ime

    6 Operator

    •

    1962 Posts

    84

    0

    Posted September 28th, 2013 23:00

    hei-san,

    Apparenetly, I'm doing a poor job keeping up with the threads that still have open questions.  Sorry about that.  I do see that you posted the eventual resolution in the following post:

    https://community.emc.com/message/737266#737266

    I would concur and is what I also refer to as "firehose down a straw".  Coincidentally I had made that comment also here (but wasn't the final resolution for that user):

    https://community.emc.com/message/759853#759853