UNSOLVED
mega_HAM_Timeout : ERROR : timeout on cloneMbox
Hi all,
I have a PE 1600SC that began reporting a failed drive in the PERC4 RAID array today. Upon reviewing the console log (NW 6.5 SP3-CPR), I am finding the following errors:
mega_HAM_Timeout : ERROR : timeout on cloneMbox 0x926460, hacb 0xB6E1B520 for in
stance [0]
mega_HAM_ISR : ERROR : cloneMbox with cmd 6F is serviced already.
Aug 26, 2005 7:49:05 PM org.apache.jk.server.JkCoyoteHandler action
SEVERE: Error in action code
java.io.IOException: Operation would block
stance [0]
mega_HAM_ISR : ERROR : cloneMbox with cmd 6F is serviced already.
Aug 26, 2005 7:49:05 PM org.apache.jk.server.JkCoyoteHandler action
SEVERE: Error in action code
java.io.IOException: Operation would block
and
mega_HAM_Timeout : ERROR : timeout on cloneMbox 0x938960, hacb 0xB6E1BE20 for in
stance [0]
mega_HAM_ISR : ERROR : cloneMbox with cmd 16 is serviced already.
stance [0]
mega_HAM_ISR : ERROR : cloneMbox with cmd 16 is serviced already.
This appeared in the log Friday evening (during tape backups), and the drive failure apparently occured early this morning (Sunday).
I see from other posts about this message, that it may mean the PERC controller may be failing and needs to be replaced. Since this error occured 2 days before the drive failure, could this indicate that the drive hasn't actually failed, but rather the controller believes it has? Any recommendations?
Also, how do I disable the PERC's beeping alarm?
By the way, the server and PERC have the latest firmware, and the PERC's HAM is ver 7.02.03 (C)
Thanks much!
If you loaded the Dell PEDGE3.HAM, it should have copied DELLMGR.NLM into SYS:SYSTEM. When you run this module from the console, it looks like the CTRL+M PERC BIOS. You can disable the alarm there.
As for the mega_HAM_Timeout error, this is going to take some troubleshooting. It is possible that it is a PERC failure, but I think at this point it is unlikely.
I would start by making sure all software has the latest patches installed. Check to see you an ABEND.LOG exists in SYS:SYSTEM.
Also, you can pull a log from the PERC card itself. It can pulled using Array Manager if you have it installed and configured or Dell technical support can provide you with a tool called TTY that will allow you to pull the PERC log from DOS.
Please post your service tag number and let me know if your server is abending and if you have Array Manager installed.
Carrie
Looks like your server has a PERC4dc. Check to see what firmware you have installed. You should be able to find it in DELLMGR in Object - Adapter - Other Adapter information.
Also, all the third party software needs to be patched, which includes your backup software.
As for Open Manage, take a look in the autoexec.ncf towards the bottom and see if you see any comments for Dell software. If you do, copy the lines here and I'll tell you what you have installed. I would also like to know if you have Symantec Antivirus installed.
Carrie
Message Edited by DELL-Carrie on 08-29-2005 09:14 AM
- I believe all 3rd party software has been patched, including my tape backup software (Arcserve 9.1). The PERC firmware was patched to:351SThere is nothing dell specific loading. I had to manually load dellmon and dellmgr after the drive failure.Thanks for your help.
- Carrie,I have not received an e-mail yet, but I have tried to use the nwttylog.nlm to extract the PERC's log. I was able to see entries in the log that started from my last reboot on 08/08, but unfortunately, this tool did not extract any of the recent log entries. For whatever reason, the log seems to be fine until it gets to PR log entries on 08/16, prints a "€" character, and then the log duplicates again from the beginning multiple times.I did find the following Novell TID: http://support.novell.com/cgi-bin/search/searchtid.cgi?10096720.htm that was last updated this month, and indicates that the controller's cache memory module is defective.Even though one of the drives was deactivated, it did not show any media errors, and I was able to reactivate and rebuild the offline drive.What do you think?-ThanksExcerpt from NWTTYLOG.NLM Output:TTY History for HA(0) -Bus 0x01 Device 0x02
€T0: LSI Logic MegaRAID i960 firmware loadedT0: Firmware version 351S build on May 31 2005 at 21:05:31T0: Board is type 1000/1960/1028/0520T0: Can_flush=0T0: DRAM SIZE=64 MBT0: DRAM_ALT sig invalid from previous bootT0: FLUSH_ON_SYSTEM_RESET=0T0: (C) LSI Logic Corporation 2002T0: MegaRAID Series 520 firmware version 351ST0: WAIT FOR BIOS....T0: BIOS UP!T0: Enabling data cacheT0: EepromInit: Family=14, SN=d50a1d010000T0: EepromInit: Board SN not programmedT0: Environment data:T0: VALIDATION=NoneT0: MFC data:T0: Vendor/DeviceID=1000/1960T0: SubVendor/SubDevice=1028/0520T0: OEM=Dell, clusterDisable=0, flexRaidDisable=0T0: rebuildRate=30, stripeSize=128, flushTime=4T0: cachedIo=1, writeBack=1, readAhead=0T0: channelBase='0', smartMode=6, alarmDisable=0T0: fastInitDisable=0, coercion=128M, disablePredictiveFail=0T0: disableWebBios=1, disableCtrlM=0, writeThroughWhenBatteryBad=1T0: zcrConfig=Undefined, keepSafteFailedStatus=0, autoHotSpareRestore=0T0: variableChkConRate=0, enableNvramDiskMgmtChange=0T0: dirtyLedShowsDriveActivity=0, disableConsChkRestoration=0T0: biosContinueOnError=0, biosAutoConfig=PromptT0: disableRandomDriveDeletion=0T0: DISK_CACHE_ADDR=d0d2c800T0: MEM_END_ADDR=d3fffff0T0: Found MPT LVD 1030(1/5/0) at fbff0004/0, mapped to 8bff0000T0: Total LSI MPT Chips found 1T0: LSI_InitMPT : start_index 0 totalLSIMPTChips 1T0: Verifying Image Signatures...VERIFIEDT0: Verifying image check sum... VERIFIEDT0: The FW version being loaded is MPTFW-01.03.35.00-ITT0: NextImageHeaderOffset=9c70, ExtImageSize=818T0: FW download complete... Expecting LSI FW to start excute and come to ready stateT2: MISM CHN_STATE_MPT_GET_FW_FEAT chip 0T2: PRESENT SCSI_ID = 0T2: Changing scsiId to 7T2: Check IOC FACTS chip 0T2: MISM CHN_STATE_MPT_OPERATIONAL chip 0T2: MISM: Reply frame size 60 start addr d0525320T2: fe reply free frames postedT2: MISM CHN_STATE_MPT_INIT_BUS_RST chip 0T3: MPT_Poll: chip 0 CHN_STATE_MPT_INIT_BUS_RSTT5: DISM: Queued!T5: MPT_ProcessIo Reply Fr 2 EVENT_NOTIFICATIONT5: MPI_EVENT_EVENT_CHANGET5: MPT_SetIocPageParameters: After Write CoalescingDepth=1, Timeout=0, Flags=1T8: DISM_ProcessPprState: DomainVal done on all disksT9: DISM: Complete!!!T9: Physical device info:T9: ID NVRState Vendor Product Rev 6 7 56T9: -- -------- -------- ---------------- ---- -- -- --T9: 00 Online SEAGATE ST336607LW DS08 01 3e 0fT9: 01 Online SEAGATE ST336607LW DS08 01 3e 0fT9: 02 Online SEAGATE ST336607LW DS08 01 3e 0fT9: battery init: battery backup circuit is not mountedT9: TBBU: No TBBU h/wT9: Verifying config struct at Addr e0001400T9: NVRAM checksum OK - reading configurationT9: DISK_CACHE_ADDR=d0d2c800T9: MEM_END_ADDR=d3fffff0T9: Memory End d3fffff0T9: Total memory available for disk cache: 32d33f0T9: Total Number of Cache Lines 811T9: SS 128: mrs=3 lc=811 ldc=1 ps=1 cm=ff ba=0 LDs: 0T9: LD 0: L=5 SS=128 Size=8776000 NL=811 Status=2 DT=251 BT=512T9: span 0: sBlk=00000000, nBlk=043bb000, dev=00-01-02T9: can_flush = 0T9: No Reconst:Checking drive infoT9: MIGRATE: 40LD or 8ld new drive ch 0 tgt 0T9: REF drive found at ch 0 tgt 0T9: Attempting to perform drive roamingT9: NOT Flushing CacheT9: RMW: NVRAM structure valid - checking for active RMWsT9: Inside SymRequestPoolInitT9: Memory Pages USed=100T9: Memory Pages USed=164T9: RequestQ ADDR=d0210000T9: exit SymRequestPoolInitT9: inside SymMsgFifoInitT9: InboundFreeQEnd=d0200000T9: exit SymMsgFifoInitT10: inside SymMsgFifoInitT10: exit SymMsgFifoInitT10: Memory Pages USed=34aT10: Memory Pages USed=34eT10: Memory Pages USed=34fT10: Memory Pages USed=374T10: found 1 logical drivesT10: logDrv=0 Size=877600008/08 19:22:31: Time established at T6808/08 19:22:31: BIOS CALL FOR DRV ROAMING : 5508/08 19:22:31: drive roaming not done08/08 20:00:00: prDiskStart: starting Patrol Read on PD=0008/08 20:00:00: prDiskStart: starting Patrol Read on PD=0108/08 20:00:00: prDiskStart: starting Patrol Read on PD=0208/08 20:00:00: Next PR scheduled to start at 08/09 0:00:0008/08 21:16:56: prCallback: PR completed for pd=0108/08 21:16:56: prDiskDone: finishing PR on PD=0108/08 21:17:16: prCallback: PR completed for pd=0208/08 21:17:16: prDiskDone: finishing PR on PD=0208/08 21:17:24: prCallback: PR completed for pd=0008/08 21:17:24: prDiskDone: finishing PR on PD=0008/08 21:17:24: PR cycle complete08/09 0:00:00: prDiskStart: starting Patrol Read on PD=0008/09 0:00:00: prDiskStart: starting Patrol Read on PD=0108/09 0:00:00: prDiskStart: starting Patrol Read on PD=0208/09 0:00:00: Next PR scheduled to start at 08/09 4:00:0008/09 1:35:37: prCallback: PR completed for pd=0108/09 1:35:37: prDiskDone: finishing PR on PD=0108/09 1:35:49: prCallback: PR completed for pd=0008/09 1:35:49: prDiskDone: finishing PR on PD=0008/09 1:36:44: prCallback: PR completed for pd=0208/09 1:36:44: prDiskDone: finishing PR on PD=0208/09 1:36:44: PR cycle complete08/09 4:00:00: prDiskStart: starting Patrol Read on PD=0008/09 4:00:00: prDiskStart: starting Patrol Read on PD=0108/09 4:00:00: prDiskStart: starting Patrol Read on PD=0208/09 4:00:00: Next PR scheduled to start at 08/09 8:00:00(REMOVED FOR BEREVITY)08/16 20:00:00: prDiskStart: starting Patrol Read on PD=0008/16 20:00:00: prDiskStart: starting Patrol Read on PD=0108/16 20:00:00: prDiskStart: starting Patrol Read on PD=0208/16 20:00:00: Next PR scheduled to start at 08/17 0:00:0008/16 22:44:16: prCallback: PR completed for pd=0108/16 22:44:16: prDiskDone: finishing PR on PD=0108/16 22:49:32: prCallback: PR completed for pd=0208/16 22:49:32: prDiskDone: finishing PR on PD=0208/16 23:31:04: prCallback: PR completed for pd=0008/16 23:31:04: prDiskDone: €AND LOG REPEATST0: LSI Logic MegaRAID i960 firmware loadedT0: Firmware version 351S build on May 31 2005 at 21:05:31T0: Board is type 1000/1960/1028/0520 ...... The tty log looks fine. I hadn't seen the Novell TID you posted.
Please call into Dell technical support and ask them to replace the raid card. Give the Dell tech a link to this forum post. Please let me know if replacing the PERC resolves the error.
Carrie
I have. After 3 hrs of telephone time, they tell me this error is a software error, not a hardware problem. After much wrangling, they agreed to send me a card, which I just received. I'll let you know the outcome.
Any idea why the log didn't have any of the more recent entries in it?
Thanks again for your assistance. I really do appreciate it.
- If you look a little further down in this forum you will see that I posted the identical error message. I wrangled with Dell for a couple of days, lost a ton of data, and then they replaced the card. It's amazing how the error went away after the card was replaced, note the software is the same, but Dell always blames the software.
Gman05
7 Posts
1511
0
Posted August 29th, 2005 12:00
Thanks,
I've been into Dellmgr and disabled the alarm. The system ID is B4JT931. The server has had all the recent Novell patches installed as well as firmware updates from Dell. The server had been abending and/or hanging until the installation of NW6.5 SP3 and related patches about 30 days ago, so there is a lengthy abend.log.
I'm not sure about array manager. I only recently began working with this server, and am unfamiliar with dell specific tools, as I primarily have used HP in the past. I do see a dellmon.nlm. What is array mgr, and can I download it if it's not installed?
Thanks