Online Enginuity code upgrade from 5771 to 5772 caused system hang
Recently one of the DMX frame with enginuity 5771 was upgraded to 5772 in our environment. All the hosts connected to the frame has at least 2 paths (17 rule) to the storage and all hosts are with either powerpath or with veritas DMP enabled to take care of any spof with the SAN path. However, during the upgrade, only one host out of 6 connected to the this pair (7BA/10BA) was hung and all other stayed back. However, when checked it looks like both the paths were not available to the host at the same time. When I checked on the other hosts, which stayed online, I could see the same that both paths were taken down at the same time.
My understanding with enginuity upgrade is that it will be done on per directory basis and all at the same time.
I get a query from the platform support that how could both the path can go down at the same time as we have confirmed to them that one of the path will stay online during the code upgrade and its only necessary for them to do the failover configurations (powerpath/Veritas).
However, when checked it looks like both the paths were not available to the host at the same time. When I checked on the other hosts, which stayed online, I could see the same that both paths were taken down at the same time.
This is what the OS perceived. Probably the FC driver have wrong timeouts thus the FAST-IML caused the OS to mark each single path as unavailable.
My understanding with enginuity upgrade is that it will be done on per directory basis and all at the same time.
When you upgrade the code (or when you load a new binfile) each processor does a Fast-IML, one by one. AFAIK no more then a single processor reboots at the same time. Thus the 4 processors never ever goes offline at the same time.
I get a query from the platform support that how could both the path can go down at the same time as we have confirmed to them that one of the path will stay online during the code upgrade and its only necessary for them to do the failover configurations (powerpath/Veritas).
Simply they didn't go down at the same time.. Probably due to bad timings in configuration file of emulex driver, the OS caught the FAST-IML and marked the first path as dead. And when the code fastimled (a few seconds or minutes later) the second processor the OS caught again the event and marked second path as dead...
Can you please check if you can find same errors in logs of hosts that didn't hang? Are you running PowerPath on the host ??
We suffered from a similar issue with about 10 Windows hosts when we upgraded from 70 to 71 code on DMX3. All other hosts never noticed anythnig, but out of these 10 Windows hosts (which all needed reboots) I think 2 actually suffered from data corruption (due to the loss of disks of course, but still...).
After this incident I noticed that EMCGRAB/HEAT since then included remarks about when it was safe to do enginuity upgrades or not. Since that incident.... I always thought it was our specific problem that caused the HEAT output to have those extra lines
We had the same problem when we went from 71 to 72 code. We had 1 out of 110 servers go down. After a month of calls with EMC a Veritas we figured it was just another bug from Veritas.
When you do an out of family upgrade I thought the system could not do a fast IML on the procs. It issues the commands for all procs do a full IML and you will have both paths offline. This is why you need to make sure your time out settings are correct. If you look at the servers that stayed up you should see the disconnect and reconnect errors within seconds of each other and stay down about 10 or so seconds
The host uses Veritas DMP for the failover and no powerpath is installed.
Yesterday I got an incident report from local EMC support which is confusing as it states that all the FA ports will be IMLed at about the same time, which is difficult to be reported to the management as all the while it was updated them that only one processor at a time while others stay up.
Report: ======================
During the 71 to 72 upgrade, all FA ports will be IMLed at about the same time and this is the normal process for a 71 to 72 code upgrade. It took 13 seconds for the FA ports to come back online as can be seem from the DMX log below. FA2A-10b 503FD3E00 x01 SEP/19/09 13:01:48:208:965 PC = 0026C014 -----------> local time 01:56 am FA2A-10b 530BC3E06 x01 SEP/19/09 13:01:52:853:770 PC = 0016526C ---------- RSCN FA2A-10b 222172906 x01 SEP/19/09 13:02:01:512:985 PC = 00379A0C -----------> local time 02:04 am ================================================= FA2A-07b 503FD3E00 x01 SEP/19/09 13:01:48:527:634 PC = 0026C014 FA2A-07b 530BC3E06 x01 SEP/19/09 13:01:53:172:486 PC = 0016526C FA2A-07b 222172906 x01 SEP/19/09 13:02:01:831:772 PC = 00379A0C Both FA 7b and 10b were brought back online at about 02:04 am. No other error was found in the DMX-4 #2337 and upgrade was completed successfully.
Analysis Results and Root Cause
1) From the server log, following scsi error, "tran_err" and dmpnode 271/0x8 (trimsdg01) was disabled at 01:56:20 am.
2) As all available paths/dmpnode was disabled, disks were not accessible, and file system got disabled.
3) We have observed that DMP has disabled the paths immediately on following the scsi errors detection. This can only happen when the fcp driver does not have a timeout delay.
There is a FCP_OFFLINE_DELAY setting found missing in the /etc/system file. It controls the wait time to off-line a SCSI target when the target may temporarily "disappear" due to a switch RSCN or other asynchronous events. This setting is designed to safeguard against the accidental or inappropriate off-lining of a SCSI target due to a short-duration "disappearance" from the fabric while SAN reconfiguration information is being exchanged. Although you will find documents which say that the following parameters ¿ set fcp:fcp_offline_delay=20 ¿ set fcp:ssfcp_enable_auto_configuration=1 are no longer required for Solaris 10, it was proven in our EMC labs that, 4 times out of 6 fcp_offline_delay is set to 0 if the "set fcp:fcp_offline_delay=20" line is not also included in /etc/system. Therefore EMC strongly recommends adding these 2 entries in /etc/system.
A reboot -- -r must be performed to enable new settings
When you upgrade the code (or when you load a new binfile) each processor does a Fast-IML, one by one. AFAIK no more then a single processor reboots at the same time. Thus the 4 processors never ever goes offline at the same time.
That is an incorrect statement - when micrococde is upgraded the Fast IML is done on ALL directors at the same time. This is specifically why you need the correct timeout settings on the host HBA's.
No an in-family code upgrade will also IML all processors at the same time it does not step through them one at a time. The fast IML is very quick and is tracked in milliseconds.
When it's an out of family upgrade the DMX will IML all procs at the same time. This is done to prevent access to the backend with different out of family code levels. When you do an in family upgrade the DMX will IML as you posted.
Symmetrix Enginuity code upgrade will fast IML all FAs, the time is around 10s at most time from I heard and observed. You might argue then it can't be qualified as NDU, my view is you can't talk about NDU without looking at the whole environment like SAN and Host OS, because what you really care is whether it's disruptive to host function.
IML all FAs will not cause paths being unavailable to OS, providing OS has timeout setting set correctly. Default FC/SCSI timeout value for major OS are typically conservative and will not be an issue. So if you said some hosts lost all paths, my guess is it's likely the timeout setting on that host has been changed.
Before talking about the timeout setting for different drivers later, I would also like to point out the 'tran_err' in initial post does not means path went offline, it only mean SCSI command timeout and the command will be retried later.
Below message will show you when the path start go offline:
emcp: [ID 801593 kern.notice] Error: Path Bus XXXX Tgt 50060XXXXXXXXXXX Lun 33 to 00019010XXXX is dead (if you are using Powerpath)
or
[ID 408114 kern.info] /scsi_vhci/ssd@g60060e8004a4ad000000a4ad00000004 (ssd49) offline (if you are using MPxIO)
For performance sensitive environment, it's not rare SA want to change the FC/SCSI timeout to get better failover time, however, you have to know your stroage environment better so the change won't sacrifice the availability. Here are a few critical timeout value can be set in Solaris environment:
1. Timeout value controls the number of seconds that the HBA waits before it off-lines a port due to "link down" event:
For Leadville, this is controlled by fp_offline_ticker, by default it's set to 90s, it can be changed in /kernel/drv/fcp.conf. This setting correspond to R_T_TOV in FC-FS2 standard, it real world, it shouldn't be set below to R_A_TOV(10s). For LPFC driver, this is controled by "linkdown-tmo" set in /kernel/drv/lpfc.conf, default is 30s.
2. Timeout value controls the wait time to off-line a SCSI target when the target may temporarily "disappear" due to a switch RSCN or other asynchronous events:
For Leadville driver, this is controlled by fcp_offline_delay, which is 2*R_A_TOV as specified in FCP4 Chapter 11. It's set to 20s by default, it can be changed in /kernel/drv/fcp.conf, however, I don't think you want to change it unless you want to modify the R_A_TOV for your entire SAN environment(Change this setting on SAN switch will require a reboot typically as well). For LPFC driver, this is controled by "nodev-tmo" set in /kernel/drv/lpfc.conf, default is 30s.
Above 2 value are most critical timeout setting will determine how your system respond during array fast IML.
3. There are also SCSI timeout (sd_io_time for LPFC and ssd_io_time for Leadville) , which determine how long an I/O can be outstanding before an error condition is returned. Each SCSi retry default to 100ms, that's why you might syslog spewed with SCSI retry constantly. SCSI queue depth can also play a role here, that's why storage vendor alway recommend to do the array upgrade during low activity, since you might hit queue_full issue even though other parameter mentioned above set correctly.
There is a lot a gap in what I know, what was told to me and what was happened in this whole activity. As, I was under the belief that only one processor will be touched while others stay up. And I believe this is only applicable for inter family Code upgrade and not for the ePaks and regular bin changes.
xe2sdc
6 Operator
•
2831 Posts
686
0
Posted September 23rd, 2009 01:00
paths were not available to the host at the same
time. When I checked on the other hosts, which
stayed online, I could see the same that both paths
were taken down at the same time.
This is what the OS perceived. Probably the FC driver have wrong timeouts thus the FAST-IML caused the OS to mark each single path as unavailable.
will be done on per directory basis and all at the
same time.
When you upgrade the code (or when you load a new binfile) each processor does a Fast-IML, one by one. AFAIK no more then a single processor reboots at the same time. Thus the 4 processors never ever goes offline at the same time.
could both the path can go down at the same time as
we have confirmed to them that one of the path will
stay online during the code upgrade and its only
necessary for them to do the failover configurations
(powerpath/Veritas).
Simply they didn't go down at the same time..
Probably due to bad timings in configuration file of emulex driver, the OS caught the FAST-IML and marked the first path as dead. And when the code fastimled (a few seconds or minutes later) the second processor the OS caught again the event and marked second path as dead...
Can you please check if you can find same errors in logs of hosts that didn't hang?
Are you running PowerPath on the host ??