UNSOLVED
SLES11 Linux systems crash during NDU
We have 25 Oracle DB servers running on HP hardware, everything from DL385G1 to BL460G7.
All currently running under SLES11 SP1. Using Powerpath for multipathing (version 5.5.0.00.00-275)
For the second time in 6 months we have had all these boxes crash during a Flare update
process. Storage is a VNX5500. All Windows, VMware, and Solaris servers survived the SP
failovers. Here are the steps we took prior to this latest disaster:
Verified BIOS settings for the Qlogic HBAs per EMC recommendation
Installed the 'latest' set of OS patches. This is restricted by the fact that Powerpath is
not certified with the current set of Suse patches. We're up to kernel version 2.6.32.54-0.3
Configured hosts to capture crash dumps
I've made a stab at looking at the crash dumps but I'm not a kernel guru. In 2 cases the
backtrace showed the crash occurred during a vfs function in the ASM module. Another case
showed the reiser_fs module running at the crash.
It appears that EMC wants to pass this problem off to Oracle since they own ASM. Suse doesn't
want to look at it since we're not at the current patch level.
We never had this problem running with a CX700 and CX3-80 and SLES9. All survived numerous NDU
maintenance events and a couple of FC switch failures. Now we're running our production DB
environment without effective SAN redundancy.
Has anyone else experienced this problem and might have a workaround or fix ??
thanks,
Don M.
And I'll reply to my own question ...
I started some testing of this problem yesterday with a spare HP box and a CX3-80. Load the system with SLES11 SP1, Oracle DB code, and powerpath. Attach 2 luns to the HP, create Linux filesystems, and start up 2 copies from local disk -> SAN disk. Then reboot an SP.
With powerpath loaded, this test would fail every time. One of the copy jobs would fail, the filesystem would become inaccessible. The /dev/emcpower__ device would become inaccessible. No type of rescanning would get it to show up again. Only came back with a reboot. The host did see paths going dead and then back alive as the SP rebooted.
So I removed powerpath and setup native mpio for multipathing. Repeated test. All survived and the copies ran to completion. Multipath showed paths dropping and resuming as the SP came back.
So it appears that there's something inherently flawed with powerpath in this configuration.
Anonymous User
63 Posts
257
0
Posted August 17th, 2012 18:00
That good news, you can send this to support, but I think they will look your testing case first. Hopefully you can link the mis-behavior on the CX3-80 to your VNX5500.
Are you able to test new PowerPath version?
Anonymous User
63 Posts
257
0
Posted August 7th, 2012 00:00
Frankly, EMC doesn't have the ablity to point out what was wrong.
From my personal opionion, Oracle and SUSE owns the responsibilities to point out what was wrong.