UNSOLVED

Magic999

updated

17 years ago

M

Magic999

11 Posts

0

164282

January 4th, 2010 13:00

OMSA daemons appear to crash a minute after startup

I am having a (yet another) peculiar problem with OMSA on Dell PE 2950 running CentOS 5.4 (64 bit). (I list the details of configuration further in my post.) As installed, OMSA daemons seem to work only for around 30-60 seconds, and then some of them quit (crash?).

Immediately after reboot of the machine:

# srvadmin-services.sh status
dell_rbu (module) is running
ipmi driver is stopped
dsm_sa_datamgrd (pid 4398 4128) is running
dsm_sa_eventmgrd (pid 4375) is running
dsm_om_shrsvcd (pid 3575) is running
dsm_om_connsvcd (pid 4394 4393) is running

# omreport storage controller
Controller  PERC 5/i Integrated (Embedded)

[...]

Web-based GUI also works OK at this point.

A minute later (without doing anything else on the machine):

# srvadmin-services.sh status
dell_rbu (module) is stopped
ipmi driver is running
dsm_sa_datamgrd is stopped
dsm_sa_eventmgrd is stopped
dsm_om_shrsvcd (pid 3575) is running
dsm_om_connsvcd (pid 4394 4393) is running

# omreport storage controller
No controllers found

The web-based GUI shows "Main Chassis" and "Storage" empty, while "Software" has only "Operating System", which is also empty.

However, everything works fine (indefinitely) if I restart the daemons:

# srvadmin-services.sh restart

Shutting down DSM SA Shared Services:                      [  OK  ]
Shutting down DSM SA Connection Service:                   [  OK  ]
Stopping Systems Management Data Engine:
Stopping dsm_sa_eventmgrd: Not started                     [FAILED]
Stopping dsm_sa_datamgrd: Not started                      [FAILED]
Stopping Systems Management Device Drivers:
Stopping dell_rbu: Not started                             [  OK  ]
Starting Systems Management Device Drivers:
Starting dell_rbu:                                         [  OK  ]
Starting ipmi driver: Already started                      [  OK  ]
Starting Systems Management Data Engine:
Starting dsm_sa_datamgrd:                                  [  OK  ]
Starting dsm_sa_eventmgrd:                                 [  OK  ]
Starting DSM SA Shared Services:                           [  OK  ]
Starting DSM SA Connection Service:                        [  OK  ]

# omreport storage controller
Controller  PERC 5/i Integrated (Embedded)
[...]

To make the OMSA daemons start properly without manual intervention, I disabled all deamons by chkconfig at all runlevels, and instead put the following lines into /etc/rc.d/rc.local:

/opt/dell/srvadmin/sbin/srvadmin-services.sh start
/opt/dell/srvadmin/sbin/srvadmin-services.sh restart

Both lines are necessary, so, in essence, OMSA needs to be started twice in order to work properly. It seems to me that this is a result of some circular dependencies in initialization order between OMSA components. The workaround works, but I find it quite ugly and would prefer some nicer solution. Has anyone else encountered this problem?

 

Details of my configuration:

Hardware: DELL PowerEdge 2950

System:
Centos 5.4 64 bit

# uname -a
Linux peta.swmed.edu 2.6.18-164.9.1.el5 #1 SMP Tue Dec 15 20:57:57 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

 

PERC Controller: PERC 5/i Integrated

# omreport storage controller
 Controller  PERC 5/i Integrated (Embedded)

Controllers
ID                                            : 0
Status                                        : Ok
Name                                          : PERC 5/i Integrated
Slot ID                                       : Embedded
State                                         : Ready
Firmware Version                              : 5.2.2-0072
Minimum Required Firmware Version  : Not Applicable
Driver Version                                : 00.00.04.08-RH2


OMSA Version
: 6.2.0

# rpm -qa | grep -i srvadmin
srvadmin-storelib-6.2.0-1.11.el3
srvadmin-omilcore-6.2.0-1.9.el5
srvadmin-omcommon-6.2.0-1.19.el5
srvadmin-cm-6.2.0-677
srvadmin-rac4-6.2.0-1.5.el5
srvadmin-storelib-libpci-6.2.0-1.1.el5
srvadmin-smcommon-6.2.0-1.29.el5
srvadmin-racadm5-6.2.0-677
srvadmin-storage-6.2.0-1.29.el5
srvadmin-storageservices-6.2.0-1.5.el5
srvadmin-rac5-6.2.0-1.5.el5
srvadmin-hapi-6.2.0-1.17.el5
srvadmin-sysfsutils-6.2.0-2.1.el5
srvadmin-xmlsup-6.2.0-1.17.el5
srvadmin-racsvc-6.2.0-677
srvadmin-fsa-6.2.0-1.6.el3
srvadmin-omacore-6.2.0-1.18.el5
srvadmin-itunnelprovider-6.2.0-1.6.el5
srvadmin-standardAgent-6.2.0-1.5.el5
srvadmin-rac4-components-6.2.0-677
srvadmin-racdrsc5-6.2.0-677
srvadmin-all-6.2.0-1.5.el5
srvadmin-racadm4-6.2.0-677
srvadmin-idracadm-6.2.0-677
srvadmin-isvc-6.2.0-1.16.el5
srvadmin-iws-6.2.0-1.18.el5
srvadmin-base-6.2.0-1.5.el5
srvadmin-idrac-6.2.0-1.5.el5
srvadmin-rac5-components-6.2.0-677
srvadmin-webserver-6.2.0-1.5.el5
srvadmin-jre-6.2.0-1.17.el5
srvadmin-storage-populator-6.2.0-1.25.el3
srvadmin-idracdrsc-6.2.0-677
srvadmin-smweb-6.2.0-1.29.el5
srvadmin-megalib-6.2.0-1.6.el3
srvadmin-deng-6.2.0-1.6.el5
srvadmin-idrac-components-6.2.0-677
srvadmin-racdrsc4-6.2.0-677
srvadmin-storelib-sysfs-6.2.0-1.1.el5

The OMSA software was installed as follows:
wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
yum install srvadmin-all srvadmin-storelib-sysfs srvadmin-storelib-libpci

 

OMSA Daemons

# chkconfig --list
dataeng            0:off    1:off    2:on    3:on    4:on    5:on    6:off
dsm_om_connsvc     0:off    1:off    2:on    3:on    4:on    5:on    6:off
dsm_om_shrsvc      0:off    1:off    2:on    3:on    4:on    5:on    6:off
dsm_sa_ipmi        0:off    1:off    2:on    3:on    4:on    5:on    6:off
instsvcdrv         0:on    1:off    2:on    3:on    4:on    5:on    6:on
openwsmand         0:off    1:off    2:on    3:on    4:on    5:on    6:off
[List edited to show only OMSA-related daemons]

  • phantomtiger

    1 Message

    3196

    0

    Posted January 6th, 2010 15:00

    I'm having this same exact problem on a Dell PE 6850 running CentOS 5.4 (32 Bit).

  • ArTourter

    3 Posts

    3196

    0

    Posted January 11th, 2010 11:00

    I get a similar problem on every single Dell server I have (4x PE2900, 1x PE1950) all running CentOS 5.4 64bit or 32bit and OMSA 6.2.0 and wih PERC{5,6}/{i,e} controlers, except a restart for me doesn't fix it.

    Every thing has been working fine for 3 years with all previous versions of OMSA, but 6.2.0 is broken.

    for me, dsm_sa_datamgrd takes about 10 minutes to decide whether it can start or not and then end up with a red FAILED, although when doing a status it claims it is started.

    and dsm_sa_snmpd just segfaults:

    dsm_sa_snmpd[10108]: segfault at 0000000000000008 rip 0000000000b4e49d rsp 00000000ffe6fad4 error 4
    dsm_sa_snmpd[12040]: segfault at 0000000000000008 rip 0000000000b4e49d rsp 00000000ff86f644 error 4
    dsm_sa_snmpd[16331]: segfault at 0000000000000008 rip 0000000000b4e49d rsp 00000000ffb12b84 error 4

    When accessing the web interface, every thing is blank, you can login, get the normal interface and the tree on the left but no information is available at all.

    I know CentOS is not officially supported but as it used to work just fine until now, it would be nice if this got fixed at some point in the fairly near future.<

  • node13h

    1 Message

    3196

    0

    Posted January 12th, 2010 03:00

    Similar problem here. two servers PE 1950 with PERC5/i and PERC6/i CentOS 4.8 (64 bit), OMSA 6.2, srvadmin-services.sh restart fixes some things, but temperatures and storage still not shown. Earlier OMSA versions were working fine (5.5 was fine for sure).

    # omreport chassis temps
    Error! No temperature probes found on this system.

  • Magic999

    11 Posts

    3196

    0

    Posted January 12th, 2010 09:00

    In my case omreport chassis temps works, though it shows one temperature reading only. I wonder if this is correct...

    # omreport chassis temps
    Temperature Probes Information

    ------------------------------------
    Main System Chassis Temperatures: Ok
    ------------------------------------

    Index                     : 0
    Status                    : Ok
    Probe Name                : System Board Ambient Temp
    Reading                   : 22.0 C
    Minimum Warning Threshold : 8.0 C
    Maximum Warning Threshold : 42.0 C
    Minimum Failure Threshold : 3.0 C
    Maximum Failure Threshold : 47.0 C

  • bretticus

    4 Posts

    3195

    0

    Posted January 12th, 2010 19:00

    Just wondering if anyone found a resolution to this. I just installed this with a new Centos 5.4 build on a PE 2850 with a Perc 4e/di RAID controller. I am having the same frustrations.

    This issue is intermittent. Once in a while, I will get a report. More than often I get "no controllers found."

    At the moment (mysteriously) datamgrd is running, usually it is not and when I run srvadmin-services.sh restart I get an error message reporting that it is not running.

    # srvadmin-services.sh status
    Checking Remote Access Controller (RAC4) Service... not run[  OK  ]
    dell_rbu (module) is stopped
    ipmi driver is stopped
    dsm_sa_datamgrd (pid 20088) is running
    dsm_sa_eventmgrd (pid 20106) is running
    dsm_om_shrsvcd (pid 17618) is running
    dsm_om_connsvcd (pid 17652 17651) is running

    # uname -a
    Linux ******* 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

     

    # omreport storage controller
     Controller  PERC 4e/Di (Embedded)

    Controllers
    ID                                            : 0
    Status                                        : Ok
    Name                                          : PERC 4e/Di
    Slot ID                                       : Embedded
    State                                         : Ready
    Firmware Version                              : 5B2D
    Minimum Required Firmware Version             : Not Applicable
    Driver Version                                : Not Applicable
    Minimum Required Driver Version               : Not Applicable
    Storport Driver Version                       : Not Applicable
    Minimum Required Storport Driver Version      : Not Applicable
    Number of Connectors                          : 2
    Rebuild Rate                                  : 30%
    BGI Rate                                      : Not Applicable
    Check Consistency Rate                        : Not Applicable
    Reconstruct Rate                              : Not Applicable
    Alarm State                                   : Not Applicable
    Cluster Mode                                  : Not Applicable
    SCSI Initiator ID                             : 7
    Cache Memory Size                             : 256 MB
    Patrol Read Mode                              : Auto
    Patrol Read State                             : Stopped
    Patrol Read Rate                              : Not Applicable
    Patrol Read Iterations                        : 2836
    Abort check consistency on error              : Not Applicable
    Allow Revertible Hot Spare and Replace Member : Not Applicable
    Auto replace member on predictive failure     : Not Applicable
    Load balance                                  : Not Applicable
    Security Capable                              : Not Applicable
    Security Key Present                          : Not Applicable
    Redundant Path view                           : Not Applicable

    # rpm -qa | grep -i srvadmin
    srvadmin-sysfsutils-6.2.0-2.1.el5
    srvadmin-omilcore-6.2.0-1.9.el5
    srvadmin-isvc-6.2.0-1.16.el5
    srvadmin-itunnelprovider-6.2.0-1.6.el5
    srvadmin-racsvc-6.2.0-677
    srvadmin-base-6.2.0-1.5.el5
    srvadmin-racdrsc5-6.2.0-677
    srvadmin-storelib-sysfs-6.2.0-1.1.el5
    srvadmin-webserver-6.2.0-1.5.el5
    srvadmin-hapi-6.2.0-1.17.el5
    srvadmin-megalib-6.2.0-1.6.el3
    srvadmin-xmlsup-6.2.0-1.17.el5
    srvadmin-smcommon-6.2.0-1.29.el5
    srvadmin-omcommon-6.2.0-1.19.el5
    srvadmin-omacore-6.2.0-1.18.el5
    srvadmin-storage-populator-6.2.0-1.25.el3
    srvadmin-idrac-components-6.2.0-677
    srvadmin-racadm4-6.2.0-677
    srvadmin-rac4-components-6.2.0-677
    srvadmin-rac4-6.2.0-1.5.el5
    srvadmin-standardAgent-6.2.0-1.5.el5
    srvadmin-rac5-components-6.2.0-677
    srvadmin-smweb-6.2.0-1.29.el5
    srvadmin-rac5-6.2.0-1.5.el5
    srvadmin-idracadm-6.2.0-677
    srvadmin-fsa-6.2.0-1.6.el3
    srvadmin-storelib-libpci-6.2.0-1.1.el5
    srvadmin-iws-6.2.0-1.18.el5
    srvadmin-storelib-6.2.0-1.11.el3
    srvadmin-deng-6.2.0-1.6.el5
    srvadmin-storage-6.2.0-1.29.el5
    srvadmin-cm-6.2.0-677
    srvadmin-racdrsc4-6.2.0-677
    srvadmin-idracdrsc-6.2.0-677
    srvadmin-racadm5-6.2.0-677
    srvadmin-idrac-6.2.0-1.5.el5
    srvadmin-storageservices-6.2.0-1.5.el5
    srvadmin-jre-6.2.0-1.17.el5

     

    # srvadmin-services.sh enable
    instsvcdrv      0:on    1:off   2:on    3:on    4:on    5:on    6:on
    dataeng         0:off   1:off   2:on    3:on    4:on    5:on    6:off
    dsm_om_shrsvc   0:off   1:off   2:on    3:on    4:on    5:on    6:off
    dsm_om_connsvc  0:off   1:off   2:on    3:on    4:on    5:on    6:off

     

     

     

     

  • bretticus

    4 Posts

    3195

    0

    Posted January 12th, 2010 23:00

    After installing some ipmi modules into the kernel, omreport storage controller works:

    # /sbin/modprobe ipmi_si
    # /sbin/modprobe ipmi_devintf

    # dmesg

    IPMI System Interface driver.
    ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
    ipmi: Found new BMC (man_id: 0x0002a2,  prod_id: 0x0000, dev_id: 0x20)
     IPMI kcs interface initialized
    ipmi device interface

    # srvadmin-services.sh restart

    I added these modules to rc.modules to start them on boot:

    # echo /sbin/modprobe ipmi_si  >> /etc/rc.modules
    # echo /sbin/modprobe ipmi_devintf  >> /etc/rc.modules
    # chmod +x /etc/rc.modules

    Now if I could just get the web server to work. https://{ip_address}:1311 simply times out on me.

    It appears to be listening on all interfaces:

    # netstat -an | grep 1311
    tcp        0      0 :::1311                     :::*                        LISTEN

    Not sure what is going on. Any ideas to debug this?

     

  • ArTourter

    3 Posts

    3195

    0

    Posted January 13th, 2010 09:00

    Hi Breticus,

    Thanks for the report.

    I tried loading the ipmi modules but that didn't have any effects here :-(

    ipmi_si loaded fine but trying to load ipmi_devintf kind of hangs as it seems that it got loaded by ipmi_si beforehand.

    once loaded, srvadmin_services.sh restart still takes ages with dsm_sa_datamgrd failing after 10 minutes. and omreport anything returning nothing.

    However, my web interface works fine, just doesn't have anything to show anyway:-)

    Could anyone from Dell comment?

    Cheers

     

  • JPBuckley

    1 Message

    3195

    0

    Posted April 18th, 2011 08:00

    I am having a very similar problem as well. I am using the same version of hardware, and am running VMware's ESXi 4.1 as the host platform. However, I am using Ubuntu 10.04.2 (64-bit) opposed to CentOS. I have the respository setup and every thing for Dell OM installed, but when I attempt to start the services:

    $ sudo service dataeng start

    I recieve the following error message:

    Starting Systems Management Data Engine:
    Failed to start because system is not supported

    Any help would be much appreciated!

  • Ankit_Test

    1 Rookie

    •

    72 Posts

    3195

    0

    Posted July 29th, 2011 03:00

    Gy