Replication stopped after losing power to primary control station
During a maintenance window, we needed to swap out our main network LAN switch at our DR site. In the process of trying to get the old switch out of the rack, we accidentally unplugged our CS0 device. We did not notice it off at first. Once we did we pligged it back in. I thought I managed to get that as the primary again, but I may have done something wrong. And now, when i checked my replication pairs, they are saying they last updated May 10, the same night the CS0 went offline...
So, how do I make sure my CS0 is set to primary?
Why do these CS seem to be very sensitive when one fails? Or do I just not understand how this works... (which could very well be the case)
Now that replication is in some "paused" state, how do I get it going again? Will it just work once the CS0 and CS1 are straightened out?
Firstly, please consider moving this question to the proper forum for maximum visibility. Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time. You can do so by selecting "Move" under ACTIONS along the right. Then search for and select: "Celerra Support Forum" or "VNX Support Forum" depending on the generation of array you own but the steps are the same (for the sake of this response I will assume VNX but by using the commands, this will be portable between the two). Here also is a recent post providing detailed instructions on how to move this post and will assure that you get the most (quality and quantity) feedback by the community.
The issue here is that you likely don't have an IPalias configured so your replication sessions rely on a specific Control Station to be primary/active. In a dual CS install, an IPalias is highly recommended for convenience (when logging into Unisphere or the shell you only need to remember one IP and not worry about which is the current primary/active) and for Replication, I will go on record as suggesting it is required. When you setup the CNS (Celerra Network Server) peer relationship which consists of: peer name, IP, and shared secret, it was likely configured with the actual IP of a single Control Station. Instead, in a dual CS setup, you should setup and use the IPalias of the remote array which is a single IP that represents simply the currently assigned primary/active CS; just like the Cluster (Virtual) IP you are familiar with on traditional clusters.
Creating (and listing) an IPalias
===========================
The initialization tools used during the initial install of the array do not create an IPalias; you need to create and test it manually. You can review the short steps in the "VNX System Operations" guide:
You can also can also search for the document by name on support.emc.com but I was looking to mix it up a bit and bring attention to the My Documents (UPD) portal also:
With the document downloaded, skip to the section "Create an IP alias".
In relation to Replicator, this is discussed in the "Using VNX Replicator" document which you can search for on support.emc.com by name. Specifically you will want to review the following:
Section: "Using dual Control Stations"
Sub-Section: "IP aliasing"
Steps are basically (run as root):
1) Verify first if you have an IPalias created:
/nas/sbin/nas_config -IPalias -list
2) For each dual control station configuration, create an IPalias
/nas/sbin/nas_config -IPalias -create 0
3) Verify again that it was correctly created (and state = UP):
/nas/sbin/nas_config -IPalias -list
You can now log into Unisphere or CLI via the IPalias.
Determining which is primary/active and which is secondary/standby
Now to your other question about determining which Control Station is currently the assigned primary, there are several ways:
1) If you were to setup and use the IPalias, it will of course always log you into the primary/active CS
2) If you use the actual IP of either CS, if after logging in it drops you immediately to a command prompt that is also indication of the same.
3) If instead you get a prompt suggesting the following, this would be what you'd expect of the secondary:
[...]
NAS service is not running. Root privileges are required to run this command.
Please wait...
*** slot_1 secondary control station ***
Would you like to log into the Primary Control Station? (yes or [no])
[...]
3) Additionally when logged into either, you could run:
/nasmcd/sbin/getreason
NOTE: I provide the absolute path of /nasmcd (instead of /nas) which will work on either control station. /nas is only mounted on the primary control station which isn't the case on the secondary (NAS service isn't running). Therefore, consider the above command more portable.
In the output, always remember that the slot_# represents the physical location of the CS (and data movers) and never changes
CS0 = slot_0
CS1 = slot_1
The only things that will change is the reason code:
10 = primary control station
11 = secondary control station
For instance, if CS0 was the primary the response would be:
10 - slot_0 primary control station
11 - slot_1 secondary control station
If relationship is switched (CS1 is primary) you would see:
11 - slot_0 secondary control station
10 - slot_1 primary control station
Failing over manually
=================
As for failing over manually, you have two options (run as root):
1) /nas/sbin/cs_standby -failover
a) run from the primary/active control station
b) current primary CS will reboot in the process
2) /nasmcd/sbin/cs_standby -takeover
a) run from the acting standby, but should only be used when its peer is inaccessible
b) again note the use of /nasmcd since /nas would not be mounted on the secondary CS
This is covered in the following (as well as the "Command Line Reference for File" document and will mix it up a bit and provide the URL to a search string on support.emc.com):
Unlike the data movers which will not failback automatically (always a manual process to restore), with the Control Stations the system will automatically failover between the two as needed. Once configured with an IPalias and properly referenced via browser bookmarks, putty shortcuts, scripts, Replicator, etc, there also really isn't any urgency to failover again just so that CS0 is primary. In fact, when discussing Control Stations, I don't even use the term failback as the hardware is identical and one shouldn't be considered inferior to the other. Please note this discussion changes when talking about data movers but I digress.
NOTE: the exception is when you go to do any NDU task like upgrading code. In fact, when using the USM, it clearly states not to use the IPalias but instead the IP of CS0 (which implies it needs to be the primary)
What isn't available if primary Control Station is down (other than ability to manage temporarily)
===========================================
In anticipation of another question, when a primary control station is down anything that relies on the NAS service will not be performed automatically; however, the CIFS shares and/or NFS exports as served via the data movers will remain accessible throughout so you aren't down (per se). The process of failing over also can take 10 to 15 minutes before the primary is assigned, so it is important to understand what is affected. As for tasks that do require an active control station, those include (but are not limited to) things such as:
1) Scheduled checkpoints
2) Automatic extension of filesystems
3) Failover of data movers
etc.
Testing
======
It goes without saying that you should also test the IPalias by performing a series of failovers: CS0 -> CS1 and CS1 -> CS0 to make sure it works as expected. From experience though on rare occasions the IPalias doesn't come online on its own, but is easily rectified on the active CS using the ifup command (a consequence of its Linux foundation)
You can see this by running the following command (as root):
/nas/sbin/nas_config -IPalias -list
1) alias = eth3:#
2) state = UP or DOWN
If DOWN, only need to run the following command
/sbin/ifup eth3:#
NOTE: hash symbol represents the (Linux kernel) virtual IP assigned to eth3
Replication considerations
=====================
As for your replication sessions, you will want to review the IP assigned to the CNS (Celerra Network Server) peer objects on each array. Remember, a replication session is comprised of:
1) CNS relationship (name assigned to remote peer, IP, and shared secret)
2) Interconnect
3) FS or VDM session
Run the following command on both arrays:
nas_cel -list
Specifically you'll be interested in:
1) name
- name of the peer which btw does not have to match exactly the name of a CS which allows you to instead use an IPalias
2) net_path
- I'm going to assume this is referencing the actual IP of one of the control stations but as you've now learned should be the IPalias
Thanks for the comprehensive update. However, this leads to a few more questions.
You say that it is confusing to setup the IPalias to use one of the actual CS IPs, but it can be done.
Since I have a dual CS setup that was configured with no IPAlias, would I be better off in the long-run to use a different IP for the alias, or just use the CS0 IP? Since there are only two of us that do any administration on the system, I am looking for te easy route, and I doubt that confusion will be an issue.
Once the IPAlias has been configured, do both control stations need a reboot?
Nevermind, turns out that the Command Reference for File (mentioned above) has also the session output demonstrating the command so no need to wait for me to return home to capture a session:
EXAMPLE #1 To create an IP alias for the Control Station, type:
# /nas/sbin/nas_config -IPalias -create 0
Do you want slot_0 IP address <1.2.3.4> as your alias [yes or no]: no
Please enter an IP address to use as an alias: 1.2.3.6
Do you want slot_0 IP address <1.2.3.4> as your alias [yes or no]: yes
Please enter a new IP address for slot_0: 1.2.3.6
[...]
You will be interested in the second half. Note the flow:
1) yes is answered
2) You are then prompted for a new IP address for slot_0
So, you still need 3 IP's. Now, from the example in the PDF, you can tell:
1) CS0/slot_0 is the current primary Control Station
2) This was run from CS0
Otherwise if run instead from CS1 because it was the primary CS, the prompts above would ask on behalf of slot_1 (and you be stealing and reassigning the IP for CS1 if you said yes)
So just to clarify, you still need 3 IP's. You are just stealing either slot_0 or slot_1 IP's (depending on which CS is primary and that you are running it from), but still need to reassign an IP to the CS you just stole it from. Sorry that was my point about it being confusing, but I realize now that the only thing confusing was my use of the word confusing
I would say unless you just had so many scripts, browser bookmarks, Putty shortcuts, etc. that you couldn't feasibly update, then do so, but if that isn't the case then I'd leave your current setup as-is and assign the new IP for the IPalias. Even hei-san can update the CNS object with the new IP via the command:
nas_cel -modify
No it doesn't require a reboot; however, I would remind you to keep the following command handy just-in-case:
/nas/sbin/nas_config -IPalias -list
1) alias = eth3:#
2) state = UP or DOWN
If DOWN, only need to run the following command
/sbin/ifup eth3:#
NOTE: hash symbol represents the (Linux kernel) virtual IP assigned to eth3
However, as recommended above, I'd highly suggest testing failover CS0 -> CS1 and CS1 -> CS0. During that time, 10-15 minutes may elapse without a primary control station assigned. When I need to run the command above (ifup) it is during the initial failover test. It seems to behave after that.
Later tonight, I'll paste a session demonstrating this. I'm not in a position currently (client network) to log into a lab.
I removed reference from the main post to my comments about setting the IPalias to that of either CS0 or CS1 (when prompted). The follow-up responses discuss the actual behavior in detail and a use case of why you may want to do so.
Another odd effect on this whole problem is that my replication speeds have gone down too. Part of the changes made the night this all broke was swapping out the core LAN switch at the DR site where we lost our CS. We had a 48 port 10/100 switch in there that we removed and installed a 48 port 10/100/1000 switch. Our uplink at this site runs at 100Mbps. I have looked at the network ports I used prior to the change (with trends collected in Cacti) and I would get up in the 60-80Mbps speeds. Now that replication is moving again, I only seem to get up to 20-30Mbps... I can post some graphs, but is there anything I need to double check on my network settings on the SAN or recommendations for switch settings? I have 4 ports per DM and they are setup as LACP group.
Thanks for your help!!! I thought the EMC tech that came onsite had done this, but the IP I gave him never did work and I never pursued it... anyway, I have added the alias information like you described and now I have a common IP address for my control stations at each site VNXA = xx.xx.xx.154 and VNXB = xx.xx.xx.174 (both SANs are in the same subnet streched across our layer 2 metro ethernet setup).
Here is what "nas_cel -list" shows on primary SAN:
[root@VNXA-CS0 sbin]# nas_cel -list
id name owner mount_dev channel net_path CMU
0 VNXA-CS0 0 xxx.xx.x.150 APM001203020982007
1 VNXB-CS0 0 xxx.xx.x.170
I see above you mention using "nas_cel -modify" command to update the CNS.
Do I do this on each side (i.e. each SAN)? What about updating host names?
Apparenetly, I'm doing a poor job keeping up with the threads that still have open questions. Sorry about that. I do see that you posted the eventual resolution in the following post:
I would concur and is what I also refer to as "firehose down a straw". Coincidentally I had made that comment also here (but wasn't the final resolution for that user):
christopher_ime
6 Operator
•
1962 Posts
1200
1
Posted June 25th, 2013 18:00
Hei-san,
Firstly, please consider moving this question to the proper forum for maximum visibility. Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time. You can do so by selecting "Move" under ACTIONS along the right. Then search for and select: "Celerra Support Forum" or "VNX Support Forum" depending on the generation of array you own but the steps are the same (for the sake of this response I will assume VNX but by using the commands, this will be portable between the two). Here also is a recent post providing detailed instructions on how to move this post and will assure that you get the most (quality and quantity) feedback by the community.
EMC Community Network Frequently Asked Questions
The issue here is that you likely don't have an IPalias configured so your replication sessions rely on a specific Control Station to be primary/active. In a dual CS install, an IPalias is highly recommended for convenience (when logging into Unisphere or the shell you only need to remember one IP and not worry about which is the current primary/active) and for Replication, I will go on record as suggesting it is required. When you setup the CNS (Celerra Network Server) peer relationship which consists of: peer name, IP, and shared secret, it was likely configured with the actual IP of a single Control Station. Instead, in a dual CS setup, you should setup and use the IPalias of the remote array which is a single IP that represents simply the currently assigned primary/active CS; just like the Cluster (Virtual) IP you are familiar with on traditional clusters.
Creating (and listing) an IPalias
===========================
The initialization tools used during the initial install of the array do not create an IPalias; you need to create and test it manually. You can review the short steps in the "VNX System Operations" guide:
https://mydocs.emc.com/VNXDocs/System_Operations.pdf
You can also can also search for the document by name on support.emc.com but I was looking to mix it up a bit and bring attention to the My Documents (UPD) portal also:
https://mydocs.emc.com/VNX/requestMyDoc.jsp
With the document downloaded, skip to the section "Create an IP alias".
In relation to Replicator, this is discussed in the "Using VNX Replicator" document which you can search for on support.emc.com by name. Specifically you will want to review the following:
Section: "Using dual Control Stations"
Sub-Section: "IP aliasing"
Steps are basically (run as root):
1) Verify first if you have an IPalias created:
/nas/sbin/nas_config -IPalias -list
2) For each dual control station configuration, create an IPalias
/nas/sbin/nas_config -IPalias -create 0
3) Verify again that it was correctly created (and state = UP):
/nas/sbin/nas_config -IPalias -list
You can now log into Unisphere or CLI via the IPalias.
Determining which is primary/active and which is secondary/standby
======================================================
Now to your other question about determining which Control Station is currently the assigned primary, there are several ways:
1) If you were to setup and use the IPalias, it will of course always log you into the primary/active CS
2) If you use the actual IP of either CS, if after logging in it drops you immediately to a command prompt that is also indication of the same.
3) If instead you get a prompt suggesting the following, this would be what you'd expect of the secondary:
[...]
NAS service is not running. Root privileges are required to run this command.
Please wait...
*** slot_1 secondary control station ***
Would you like to log into the Primary Control Station? (yes or [no])
[...]
3) Additionally when logged into either, you could run:
/nasmcd/sbin/getreason
NOTE: I provide the absolute path of /nasmcd (instead of /nas) which will work on either control station. /nas is only mounted on the primary control station which isn't the case on the secondary (NAS service isn't running). Therefore, consider the above command more portable.
In the output, always remember that the slot_# represents the physical location of the CS (and data movers) and never changes
CS0 = slot_0
CS1 = slot_1
The only things that will change is the reason code:
10 = primary control station
11 = secondary control station
For instance, if CS0 was the primary the response would be:
10 - slot_0 primary control station
11 - slot_1 secondary control station
If relationship is switched (CS1 is primary) you would see:
11 - slot_0 secondary control station
10 - slot_1 primary control station
Failing over manually
=================
As for failing over manually, you have two options (run as root):
1) /nas/sbin/cs_standby -failover
a) run from the primary/active control station
b) current primary CS will reboot in the process
2) /nasmcd/sbin/cs_standby -takeover
a) run from the acting standby, but should only be used when its peer is inaccessible
b) again note the use of /nasmcd since /nas would not be mounted on the secondary CS
This is covered in the following (as well as the "Command Line Reference for File" document and will mix it up a bit and provide the URL to a search string on support.emc.com):
"Configuring Standbys on VNX"
https://mydocs.emc.com/VNXDocs/Standbys.pdf
"Command Line Reference for File"
https://support.emc.com/search/?resource=DOC_LIB&AlloftheseWrds=Command%20Line%20Reference%20for%20File&SearchWithin=true&adv=y
Failover and "Failback" (note the quotes)
==================
Unlike the data movers which will not failback automatically (always a manual process to restore), with the Control Stations the system will automatically failover between the two as needed. Once configured with an IPalias and properly referenced via browser bookmarks, putty shortcuts, scripts, Replicator, etc, there also really isn't any urgency to failover again just so that CS0 is primary. In fact, when discussing Control Stations, I don't even use the term failback as the hardware is identical and one shouldn't be considered inferior to the other. Please note this discussion changes when talking about data movers but I digress.
NOTE: the exception is when you go to do any NDU task like upgrading code. In fact, when using the USM, it clearly states not to use the IPalias but instead the IP of CS0 (which implies it needs to be the primary)
What isn't available if primary Control Station is down (other than ability to manage temporarily)
===========================================
In anticipation of another question, when a primary control station is down anything that relies on the NAS service will not be performed automatically; however, the CIFS shares and/or NFS exports as served via the data movers will remain accessible throughout so you aren't down (per se). The process of failing over also can take 10 to 15 minutes before the primary is assigned, so it is important to understand what is affected. As for tasks that do require an active control station, those include (but are not limited to) things such as:
1) Scheduled checkpoints
2) Automatic extension of filesystems
3) Failover of data movers
etc.
Testing
======
It goes without saying that you should also test the IPalias by performing a series of failovers: CS0 -> CS1 and CS1 -> CS0 to make sure it works as expected. From experience though on rare occasions the IPalias doesn't come online on its own, but is easily rectified on the active CS using the ifup command (a consequence of its Linux foundation)
You can see this by running the following command (as root):
/nas/sbin/nas_config -IPalias -list
1) alias = eth3:#
2) state = UP or DOWN
If DOWN, only need to run the following command
/sbin/ifup eth3:#
NOTE: hash symbol represents the (Linux kernel) virtual IP assigned to eth3
Replication considerations
=====================
As for your replication sessions, you will want to review the IP assigned to the CNS (Celerra Network Server) peer objects on each array. Remember, a replication session is comprised of:
1) CNS relationship (name assigned to remote peer, IP, and shared secret)
2) Interconnect
3) FS or VDM session
Run the following command on both arrays:
nas_cel -list
Specifically you'll be interested in:
1) name
- name of the peer which btw does not have to match exactly the name of a CS which allows you to instead use an IPalias
2) net_path
- I'm going to assume this is referencing the actual IP of one of the control stations but as you've now learned should be the IPalias