MD3000i seems to have corrupted firmware and does not function. Help!
See the below capture from Hyper-terminal using a console cable. Any ideas on how to reload the firmware OR default the box would be greatly appreciated!
Reset, Power-Up Diagnostics - Loop 1 of 1 3600 Processor DRAM 01 Data lines Passed 02 Address lines Passed 3300 NVSRAM 01 Data lines Passed 5900 Ethernet 91c111 #1 01 Register read Passed 02 Register test Passed 3A00 NAND Flash 06 Bad Blocks Test Passed 2310 Application Accelerator Unit 01 AAU Register Test Passed 6D00 LSI SAS 1068 IOC--Base Board 01 IOC Register Read Test Passed 02 IOC Register Address Lines Test Passed 03 IOC Register Data Lines Test Passed 6F01 QLOGIC EP4032 CHIP 0 01 Register Read Test Passed 02 Register Address Lines Test Passed 03 Register Data Lines Test Passed 3900 Real-Time Clock 01 RT Clock Tick Passed Diagnostic Manager exited normally.
Current date: 01/11/14 time: 09:28:29
Send for Service Interface or baud rate change 01/11/14-17:41:53 (GMT) (tRAID): NOTE: Set Powerup State 01/11/14-17:41:53 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0 01/11/14-17:41:53 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0 01/11/14-17:41:53 (GMT) (tRAID): NOTE: In iscsiIOQLIscsiInitDq. iscsiIoFstrBas e = 0x0 01/11/14-17:41:53 (GMT) (tRAID): NOTE: Turning on tray summary fault LED 01/11/14-17:41:55 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered. 01/11/14-17:41:55 (GMT) (tRAID): NOTE: lost persistent dq data because buffer w as modified or size changed. esmc0: LinkUp event 01/11/14-17:41:56 (GMT) (tNetCfgInit): NOTE: Network Ready 01/11/14-17:41:59 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup 01/11/14-17:42:01 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-62-00 01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:0 prevNumAc tivePhys:2 numActivePhys:2 01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:1 prevNumAc tivePhys:2 numActivePhys:2 01/11/14-17:42:09 (GMT) (tSasEvtWkr): NOTE: New Initiator: 1 - channel: 0,devH andle: x10, SAS Address: 50024e8080618604 01/11/14-17:42:19 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled 01/11/14-17:42:19 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete 01/11/14-17:42:19 (GMT) (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels 01/11/14-17:42:19 (GMT) (tRAID): NOTE: LockMgr Role is Master 01/11/14-17:42:19 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt co ntroller not ready 01/11/14-17:42:19 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawne d 01/11/14-17:42:20 (GMT) (tRAID): NOTE: spmEarlyData: No data available 01/11/14-17:42:20 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c 01/11/14-17:42:20 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x50026b94 Low = x3a76a310 01/11/14-17:42:41 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Ti me: 46 seconds 01/11/14-17:42:41 (GMT) (tRAID): NOTE: WWN baseName 00020026-b93a76a3 (valid==> NoPrevAlt) 01/11/14-17:42:41 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled 01/11/14-17:42:41 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete 01/11/14-17:42:42 (GMT) (tRAID): WARN: BID: initialize(): Power latched! 01/11/14-17:42:50 (GMT) (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0 01/11/14-17:42:50 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0 01/11/14-17:42:50 (GMT) (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt n ot communicating. 01/11/14-17:42:50 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete 01/11/14-17:42:51 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleException Error in iop::requestAltIopDelay 01/11/14-17:42:51 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSen dInfeasibleException Error) 01/11/14-17:42:51 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully 01/11/14-17:42:51 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 r ecords. 01/11/14-17:42:51 (GMT) (tRAID): WARN: Received IconSendInfeasibleException Err or adding small edr records from alt controller 01/11/14-17:42:52 (GMT) (tRAID): WARN: spm: unable to exchange features, assumi ng none 01/11/14-17:42:52 (GMT) (tRAID): NOTE: SPM acquireObjects exception: IconSendIn feasibleException Error 01/11/14-17:42:52 (GMT) (tRAID): NOTE: DBRead 0.054 secs 01/11/14-17:42:52 (GMT) (tRAID): NOTE: sas: Peering Disabled (Alt Unavailable) 01/11/14-17:42:53 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0 01/11/14-17:43:20 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout 01/11/14-17:43:20 (GMT) (tRAID): WARN: QLMailboxCommand: command completion tim eout, cmd = 0x69 01/11/14-17:43:21 (GMT) (tRAID): NOTE: Qlogic coredump file written to 'DWPF0L1 :/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50 01/11/14-17:43:21 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1
01/11/14-17:43:21 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC 01/11/14-17:43:21 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail ed. Stat f000 01/11/14-17:43:21 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543 01/11/14-17:43:21 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed 01/11/14-17:43:22 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0 01/11/14-17:43:49 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout 01/11/14-17:43:49 (GMT) (tRAID): WARN: QLMailboxCommand: command completion tim eout, cmd = 0x69 01/11/14-17:43:50 (GMT) (tRAID): NOTE: Qlogic coredump file written to 'DWPF0L1 :/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50 01/11/14-17:43:50 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1
01/11/14-17:43:50 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC 01/11/14-17:43:50 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail ed. Stat f000 01/11/14-17:43:50 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543 01/11/14-17:43:50 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed 01/11/14-17:43:51 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0 01/11/14-17:44:18 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout 01/11/14-17:44:18 (GMT) (tRAID): WARN: QLMailboxCommand: command completion tim eout, cmd = 0x69 01/11/14-17:44:19 (GMT) (tRAID): NOTE: Qlogic coredump file written to 'DWPF0L1 :/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50 01/11/14-17:44:19 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1
01/11/14-17:44:19 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC 01/11/14-17:44:19 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail ed. Stat f000 01/11/14-17:44:19 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543 01/11/14-17:44:19 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed 01/11/14-17:44:21 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0 01/11/14-17:44:48 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout 01/11/14-17:44:48 (GMT) (tRAID): WARN: QLMailboxCommand: command completion tim eout, cmd = 0x69 01/11/14-17:44:49 (GMT) (tRAID): NOTE: Qlogic coredump file written to 'DWPF0L1 :/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50 01/11/14-17:44:49 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1
01/11/14-17:44:49 (GMT) (tRAID): NOTE: QLProcessSystemError: Restart RISC 01/11/14-17:44:49 (GMT) (tRAID): ERROR: QLGetFwState: MBOX_CMD_GET_FW_STATE fail ed. Stat f000 01/11/14-17:44:49 (GMT) (tRAID): NOTE: QLRebootTimer: Status after Get FW State 4543 01/11/14-17:44:49 (GMT) (tRAID): NOTE: QLRebootTimer: QLGetFwState failed 01/11/14-17:44:50 (GMT) (tRAID): NOTE: QLStartFw: Downloading Driver's FW image 03.00.01.47 from 0058bfa0 4c0c8 bytes , result 0 01/11/14-17:45:17 (GMT) (tRAID): WARN: QLMailboxCommand: Cmd = 0069, completion timeout 01/11/14-17:45:17 (GMT) (tRAID): WARN: QLMailboxCommand: command completion tim eout, cmd = 0x69 01/11/14-17:45:18 (GMT) (tRAID): NOTE: Qlogic coredump file written to 'DWPF0L1 :/tmp/QLogic_Coredump_port_0_DWPF0L1',rc 204E50, expected 204E50 01/11/14-17:45:18 (GMT) (tRAID): WARN: Qlogic coredump file write failed.fclose returned -1
Most of the time we have seen this issue occurs due to power outages. You can try the following troubleshooting steps that may bring some controllers out of bad state:
1. Turn off array and pull all the HDD’s from array.
2. Pull one controller out and leave one controller in any slot.
3. Wait until controller LEDs are off, Power ON the array with single controller (that had the issue). You may be able to break that controller to get to shell prompt.
4. Flash controller firmware with the same FW (if step 3 was successful).
5. If “sodMain complete”, install other controller HOT till it sync up and does the “sodMain complete”. Reboot both controllers.
6. If both controller has completed SODMAIN, turn off the array and install all the HDD’s. Turn on the array.
Note: We recommend, when the array is optimal, update the controllers to the latest MD3000i code level and capture the Support Bundle
The above will not work.... This MD only has one controller and using the console cable it will not allow me into the service interface. If I then press it does nothing.... I have decided to just replace the EMM all together in hopes this will fix this issue.
OK.. I have replaced the EMM and now the CLI is responding but I do not know the password to get into the service interface... Can one of you help me with that?? Mt EMM is running 7.35.22.61 firmware
While going through your steps above, how can I flash the controller firmware as stated in #4? I am able to Telnet into the controller and I do have the admin logins, but I can't find the procedures?
So based on the errors that you are getting from your controllers they will need to be replaced. Since you are not able to access the secure shell on the controllers and clear the configuration the controllers are in a boot loop as they are not able to complete post.
Please let us know if you have any other questions.
DELL-Kenny K
1 Rookie
•
685 Posts
1319
0
Posted January 13th, 2014 11:00
Most of the time we have seen this issue occurs due to power outages. You can try the following troubleshooting steps that may bring some controllers out of bad state:
1. Turn off array and pull all the HDD’s from array.
2. Pull one controller out and leave one controller in any slot.
3. Wait until controller LEDs are off, Power ON the array with single controller (that had the issue). You may be able to break that controller to get to shell prompt.
4. Flash controller firmware with the same FW (if step 3 was successful).
5. If “sodMain complete”, install other controller HOT till it sync up and does the “sodMain complete”. Reboot both controllers.
6. If both controller has completed SODMAIN, turn off the array and install all the HDD’s. Turn on the array.
Note: We recommend, when the array is optimal, update the controllers to the latest MD3000i code level and capture the Support Bundle