UNSOLVED

TheShortman

updated

13 years ago

T

TheShortman

1 Rookie

•

2 Posts

0

69130

January 11th, 2014 10:00

MD3000i seems to have corrupted firmware and does not function. Help!

See the below capture from Hyper-terminal using a console cable.  Any ideas on how to reload the firmware OR default the box would be greatly appreciated!

Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
     01 Data lines                                                  Passed
     02 Address lines                                               Passed
3300 NVSRAM
     01 Data lines                                                  Passed
5900 Ethernet 91c111 #1
     01 Register read                                               Passed
     02 Register test                                               Passed
3A00 NAND Flash
     06 Bad Blocks Test                                             Passed
2310 Application Accelerator Unit
     01 AAU Register Test                                           Passed
6D00 LSI SAS 1068 IOC--Base Board
     01 IOC Register Read Test                                      Passed
     02 IOC Register Address Lines Test                             Passed
     03 IOC Register Data Lines Test                                Passed
6F01 QLOGIC EP4032 CHIP 0
     01 Register Read Test                                          Passed
     02 Register Address Lines Test                                 Passed
     03 Register Data Lines Test                                    Passed
3900 Real-Time Clock
     01 RT Clock Tick                                               Passed
Diagnostic Manager exited normally.


Current date: 01/11/14  time: 09:28:29

Send for Service Interface or baud rate change
01/11/14-17:41:53 (GMT) (tRAID): NOTE:  Set Powerup State
01/11/14-17:41:53 (GMT) (tRAID): NOTE:  SOD Sequence is Normal, 0
01/11/14-17:41:53 (GMT) (tRAID): NOTE:  SOD: removed SAS host from index 0
01/11/14-17:41:53 (GMT) (tRAID): NOTE:  In iscsiIOQLIscsiInitDq.  iscsiIoFstrBas
e = 0x0
01/11/14-17:41:53 (GMT) (tRAID): NOTE:  Turning on tray summary fault LED
01/11/14-17:41:55 (GMT) (tRAID): NOTE:  SYMBOL: SYMbolAPI registered.
01/11/14-17:41:55 (GMT) (tRAID): NOTE:  lost persistent dq data because buffer w
as modified or size changed.
esmc0: LinkUp event
01/11/14-17:41:56 (GMT) (tNetCfgInit): NOTE:  Network Ready
01/11/14-17:41:59 (GMT) (tRAID): NOTE:  Initiating Drive channel: ioc:0 bringup
01/11/14-17:42:01 (GMT) (tRAID): NOTE:  IOC Firmware Version: 00-24-62-00
01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:0 phy:0 prevNumAc
tivePhys:2 numActivePhys:2
01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE:  sasIocPhyUp: chan:0 phy:1 prevNumAc
tivePhys:2 numActivePhys:2
01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE:  New Initiator:  1 - channel: 0,devH
andle: x10, SAS Address: 50024e8080618604
01/11/14-17:42:19 (GMT) (tRAID): NOTE:  IonMgr: Drive Interface Enabled
01/11/14-17:42:19 (GMT) (tRAID): NOTE:  SOD: Instantiation Phase Complete
01/11/14-17:42:19 (GMT) (tRAID): WARN:  No attempt made to open Inter-Controller
 Communication Channels
01/11/14-17:42:19 (GMT) (tRAID): NOTE:  LockMgr Role is Master
01/11/14-17:42:19 (GMT) (tRAID): WARN:  FBM:validateSubModel: Exception - Alt co
ntroller not ready
01/11/14-17:42:19 (GMT) (tSasDiscCom): NOTE:  SAS Discovery complete task spawne
d
01/11/14-17:42:20 (GMT) (tRAID): NOTE:  spmEarlyData: No data available
01/11/14-17:42:20 (GMT) (sasCheckExpanderSet): NOTE:  Expander Firmware Version:
 0116-e05c
01/11/14-17:42:20 (GMT) (sasCheckExpanderSet): NOTE:  Expander SAS address: Hi =
 x50026b94 Low = x3a76a310
01/11/14-17:42:41 (GMT) (tSasDiscCom): WARN:  SAS: Initial Discovery Complete Ti
me: 46 seconds
01/11/14-17:42:41 (GMT) (tRAID): NOTE:  WWN baseName 00020026-b93a76a3 (valid==>
NoPrevAlt)
01/11/14-17:42:41 (GMT) (tRAID): NOTE:  IonMgr: Host Interface Enabled
01/11/14-17:42:41 (GMT) (tRAID): NOTE:  SOD: Pre-Initialization Phase Complete
01/11/14-17:42:42 (GMT) (tRAID): WARN:  BID: initialize(): Power latched!
01/11/14-17:42:50 (GMT) (tRAID): NOTE:  ACS: Icon ping to alternate failed: -2,
resp: 0
01/11/14-17:42:50 (GMT) (tRAID): NOTE:  ACS: autoCodeSync(): Process start. Comm
 Mode: 0, Status: 0
01/11/14-17:42:50 (GMT) (tRAID): WARN:  ACS: autoCodeSync(): Skipped since alt n
ot communicating.
01/11/14-17:42:50 (GMT) (tRAID): NOTE:  SOD: Code Synchronization Initialization
 Phase Complete
01/11/14-17:42:51 (GMT) (tRAID): NOTE:  Caught IconSendInfeasibleException Error
 in iop::requestAltIopDelay
01/11/14-17:42:51 (GMT) (tRAID): NOTE:  CheckInMonitor: Check-in failed (IconSen
dInfeasibleException Error)
01/11/14-17:42:51 (GMT) (NvpsPersistentSyncM): NOTE:  NVSRAM Persistent Storage
updated successfully
01/11/14-17:42:51 (GMT) (tRAID): NOTE:  USM Mgr initialization complete with 0 r
ecords.
01/11/14-17:42:51 (GMT) (tRAID): WARN:  Received IconSendInfeasibleException Err
or adding small edr records from alt controller
01/11/14-17:42:52 (GMT) (tRAID): WARN:  spm: unable to exchange features, assumi
ng none
01/11/14-17:42:52 (GMT) (tRAID): NOTE:  SPM acquireObjects exception: IconSendIn
feasibleException Error
01/11/14-17:42:52 (GMT) (tRAID): NOTE:  DBRead               0.054 secs
01/11/14-17:42:52 (GMT) (tRAID): NOTE:  sas: Peering Disabled (Alt Unavailable)
01/11/14-17:42:53 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image
 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0
01/11/14-17:43:20 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion
 timeout
01/11/14-17:43:20 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion tim
eout, cmd = 0x69
01/11/14-17:43:21 (GMT) (tRAID): NOTE:  Qlogic coredump file written to 'DWPF0L1
:/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50
01/11/14-17:43:21 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose
 returned -1

01/11/14-17:43:21 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC
01/11/14-17:43:21 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail
ed.  Stat f000
01/11/14-17:43:21 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State
 4543
01/11/14-17:43:21 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed
01/11/14-17:43:22 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image
 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0
01/11/14-17:43:49 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion
 timeout
01/11/14-17:43:49 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion tim
eout, cmd = 0x69
01/11/14-17:43:50 (GMT) (tRAID): NOTE:  Qlogic coredump file written to 'DWPF0L1
:/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50
01/11/14-17:43:50 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose
 returned -1

01/11/14-17:43:50 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC
01/11/14-17:43:50 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail
ed.  Stat f000
01/11/14-17:43:50 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State
 4543
01/11/14-17:43:50 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed
01/11/14-17:43:51 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image
 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0
01/11/14-17:44:18 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion
 timeout
01/11/14-17:44:18 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion tim
eout, cmd = 0x69
01/11/14-17:44:19 (GMT) (tRAID): NOTE:  Qlogic coredump file written to 'DWPF0L1
:/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50
01/11/14-17:44:19 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose
 returned -1

01/11/14-17:44:19 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC
01/11/14-17:44:19 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail
ed.  Stat f000
01/11/14-17:44:19 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State
 4543
01/11/14-17:44:19 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed
01/11/14-17:44:21 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image
 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0
01/11/14-17:44:48 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion
 timeout
01/11/14-17:44:48 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion tim
eout, cmd = 0x69
01/11/14-17:44:49 (GMT) (tRAID): NOTE:  Qlogic coredump file written to 'DWPF0L1
:/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50
01/11/14-17:44:49 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose
 returned -1

01/11/14-17:44:49 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC
01/11/14-17:44:49 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail
ed.  Stat f000
01/11/14-17:44:49 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State
 4543
01/11/14-17:44:49 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed
01/11/14-17:44:50 (GMT) (tRAID): NOTE:  QLStartFw: Downloading Driver's FW image
 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0
01/11/14-17:45:17 (GMT) (tRAID): WARN:  QLMailboxCommand: Cmd = 0069, completion
 timeout
01/11/14-17:45:17 (GMT) (tRAID): WARN:  QLMailboxCommand: command completion tim
eout, cmd = 0x69
01/11/14-17:45:18 (GMT) (tRAID): NOTE:  Qlogic coredump file written to 'DWPF0L1
:/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50
01/11/14-17:45:18 (GMT) (tRAID): WARN:  Qlogic coredump file write failed.fclose
 returned -1

01/11/14-17:45:18 (GMT) (tRAID): NOTE:  QLProcessSystemError: Restart RISC
01/11/14-17:45:18 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail
ed.  Stat f000
01/11/14-17:45:18 (GMT) (tRAID): NOTE:  QLRebootTimer: Status after Get FW State
 4543
01/11/14-17:45:18 (GMT) (tRAID): NOTE:  QLRebootTimer: QLGetFwState failed
01/11/14-17:45:19 (GMT) (tRAID): WARN:  QLStartAdapter: ControllerErrorCount exc
eeds threshold.
01/11/14-17:45:19 (GMT) (tRAID): ERROR: QLInitializeDevice: QLStartAdapter faile
d
01/11/14-17:45:19 (GMT) (tRAID): ERROR: QLAddDevice: controller/device/chip init
ialization failed.
01/11/14-17:45:19 (GMT) (tRAID): ERROR: qlgEnableHostInterface: QLInitializeDevi
ce failed.
01/11/14-17:45:19 (GMT) (tRAID): NOTE:  ****************************************
****************************************
01/11/14-17:45:19 (GMT) (tRAID): NOTE:    QLogic Target Application, Version 2.0
1.08 6-13-2005 (W2K)
01/11/14-17:45:19 (GMT) (tRAID): NOTE:          iSCSI Target Application
01/11/14-17:45:19 (GMT) (tRAID): NOTE:   ***************************************
*****************************************

Exception: Data Abort
cpsr:  60000013   pc:  0x0020eb94   semTake

  • DELL-Kenny K

    1 Rookie

    •

    685 Posts

    1319

    0

    Posted January 13th, 2014 11:00

    Most of the time we have seen this issue occurs due to power outages. You can try the following troubleshooting steps that may bring some controllers out of bad state:

    1. Turn off array and pull all the HDD’s from array.

    2. Pull one controller out and leave one controller in any slot.

    3. Wait until controller LEDs are off, Power ON the array with single controller (that had the issue). You may be able to break that controller to get to shell prompt.

    4. Flash controller firmware with the same FW (if step 3 was successful).

    5. If “sodMain complete”, install other controller HOT till it sync up and does the “sodMain complete”.  Reboot both controllers.

    6. If both controller has completed SODMAIN, turn off the array and install all the HDD’s. Turn on the array.

    Note: We recommend, when the array is optimal, update the controllers to the latest MD3000i code level and capture the Support Bundle

  • TheShortman

    1 Rookie

    •

    2 Posts

    1319

    0

    Posted January 14th, 2014 05:00

    The above will not work....  This MD only has one controller and using the console cable it will not allow me into the service interface.  If I then press it does nothing....  I have decided to just replace the EMM all together in hopes this will fix this issue.

  • DELL-Kenny K

    1 Rookie

    •

    685 Posts

    1319

    0

    Posted January 14th, 2014 09:00

     Okay, well let me know how that goes when you get the new Controller and if you are still having issues we can look into this further.

  • 1319

    0

    Posted January 25th, 2014 06:00

    OK..  I have replaced the EMM and now the CLI is responding but I do not know the password to get into the service interface...  Can one of you help me with that??  Mt EMM is running 7.35.22.61 firmware

  • 1319

    0

    Posted January 25th, 2014 06:00

    BTW...   Both users "TheShortman" and "shaneshirley" belong to me

  • DELL-Sam L

    Community Manager

    •

    8149 Posts

    •

    34712 Points

    1319

    0

    Posted January 27th, 2014 06:00

    Hello shaneshirley,

    I will send you a private message on how to reset that password.  

    Please let us know if you have any other questions.

  • 1319

    0

    Posted January 27th, 2014 08:00

    Thanks for the reply!  That was exactly what I needed...

  • kkji09

    1 Message

    1319

    0

    Posted September 21st, 2016 20:00

    We currently had the same situation with our MD3000 controllers.  They just keep rebooting when they get to this stage......

    ********************************************************************************

    09/22/16-02:01:59 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC

    09/22/16-02:01:59 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE failed. Stat f000
    09/22/16-02:01:59 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543
    09/22/16-02:01:59 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed
    09/22/16-02:02:00 (GMT) (tRAID): WARN: QLStartAdapter: ControllerErrorCount exceeds threshold.
    09/22/16-02:02:00 (GMT) (tRAID): ERROR: QLInitializeDevice: QLStartAdapter failed
    09/22/16-02:02:00 (GMT) (tRAID): ERROR: QLAddDevice: controller/device/chip initialization failed.
    09/22/16-02:02:00 (GMT) (tRAID): ERROR: qlgEnableHostInterface: QLInitializeDevice failed.
    09/22/16-02:02:00 (GMT) (tRAID): NOTE: ********************************************************************************

    While going through your steps above, how can I flash the controller firmware as stated in #4?   I am able to Telnet into the controller and I do have the admin logins, but I can't find the procedures?

    Also, MD Storage Manager will not connect to it.

    Thanks

  • DELL-Sam L

    Community Manager

    •

    8149 Posts

    •

    34712 Points

    1319

    0

    Posted September 28th, 2016 06:00

    Hello kkji09,

    So based on the errors that you are getting from your controllers they will need to be replaced. Since you are not able to access the secure shell on the controllers and clear the configuration the controllers are in a boot loop as they are not able to complete post.

    Please let us know if you have any other questions.