UNSOLVED

jhboricuamn

updated

15 years ago

J

jhboricuamn

26 Posts

0

224881

May 3rd, 2012 11:00

Having tons of issues updating DRAC firmware on 710s

I have several R710s that are giving me a lot of headaches with the iDRAC6 firmware update. They currently are at (v1.41)Out of 5 hosts, only 1 updated to the latest version (1.85). The remaining 4 not only failed but are stuck in update hell. Meaning if I SSH to the drac and issue a 'racadm fwupdate -s' command, it still outputs 'Preparing for firmware update'. Because of this, I can't reset the DRAC. The only option to get the DRAC out of this state is to completely power off the server and unplug the power cords. Since these are production servers, that becomes a scheduling issue.

In any case, I was able to power down and unplug the power cords to 1 server out of the 4, which allowed the DRAC to get out of that 'updating' funk state. When tried the update the first time I used the Dell DRAC tool, I decided not to go that route. So now I'm doing by uploading the image file directly to the DRAC via the web interface. It still fails.

I've tried v 1.70, 1.80 and 1.85. They all fail on this host at the same spot, shortly after showing 30% progress on the Updating webpage. When I ssh to the drac and issue a 'racadm gettracelog', this is what it shows:

-------------------------------------------------------------------------------
Record: 147
Date/Time: May 3 11:32:00
Source: os[407]
Description: 2012 FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 148
Date/Time: May 3 11:32:00
Source: os[407]
Description: 2012 RAC:FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 149
Date/Time: May 3 11:32:00
Source: os[407]
Description: firmwareUpdate() - set the firmware destination device : 2
-------------------------------------------------------------------------------
Record: 150
Date/Time: May 3 11:32:00
Source: os[407]
Description: firmwareUpdate() - get the firmware destination device : 2
-------------------------------------------------------------------------------
Record: 151
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 FWupdate initialization complete
-------------------------------------------------------------------------------
Record: 152
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 RAC:FWupdate initialization complete
-------------------------------------------------------------------------------
Record: 153
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readMBR() - MBR PARTITION 1 STATUS: 0x80
-------------------------------------------------------------------------------
Record: 154
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readMBR() - partition 1 is currently active, partition 3 is currently inactive
-------------------------------------------------------------------------------
Record: 155
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 156
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 RAC:FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 157
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 158
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 RAC:FWupdate checksumming image
-------------------------------------------------------------------------------
Record: 159
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 FWupdate verify image headers
-------------------------------------------------------------------------------
Record: 160
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 RAC:FWupdate verify image headers
-------------------------------------------------------------------------------
Record: 161
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readHeader()
-------------------------------------------------------------------------------
Record: 162
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readHeader() - header information: uHeaderCRC:0x7A97E3E7 uHeaderVersion:0x1 uImageType:0x1 uNumPartitions:0x3 uFWVersion:0x115001 uFWTotalImageSize:0x3477E00 uBootVerMajor:0x333B0 uBootVerMinor:0x333B4
-------------------------------------------------------------------------------
Record: 163
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readHeader() - partition 1 info: uOffset:0x0200 uSize:0x445BE0 uCRC:0x853463B1
-------------------------------------------------------------------------------
Record: 164
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readHeader() - partition 2 info: uOffset:0x445E00 uSize:0x300F000 uCRC:0x8D219434
-------------------------------------------------------------------------------
Record: 165
Date/Time: May 3 11:32:07
Source: os[18933]
Description: readHeader() - partition 3 info: uOffset:0x3454E00 uSize:0x22E38 uCRC:0x918B375A
-------------------------------------------------------------------------------
Record: 166
Date/Time: May 3 11:32:07
Source: os[18933]
Description: isCorrectBoardType() - file and board have the same board type
-------------------------------------------------------------------------------
Record: 167
Date/Time: May 3 11:32:07
Source: os[18933]
Description: isCorrectBoardType() - board type is Dell Whoville, board type is correct
-------------------------------------------------------------------------------
Record: 168
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 FWupdate programming flash
-------------------------------------------------------------------------------
Record: 169
Date/Time: May 3 11:32:07
Source: os[18933]
Description: 2012 RAC:FWupdate programming flash
-------------------------------------------------------------------------------
Record: 170
Date/Time: May 3 11:32:07
Source: os[18933]
Description: copyFileTransfer() - szSrcPath:/mnt/scratchpad/idrac_fw.bin szDestPath:/dev/mmcblk0p3 nBytesToTransfer:4479968 nSrcOffset:512
-------------------------------------------------------------------------------
Record: 171
Date/Time: May 3 11:32:14
Source: os[18933]
Description: copyFirmware() - copyFileTransfer of /dev/mmcblk0p3 succeeded
-------------------------------------------------------------------------------
Record: 172
Date/Time: May 3 11:32:14
Source: os[18933]
Description: copyFileTransfer() - szSrcPath:/mnt/scratchpad/idrac_fw.bin szDestPath:/dev/mmcblk0p5 nBytesToTransfer:50393088 nSrcOffset:4480512
-------------------------------------------------------------------------------
Record: 173
Date/Time: May 3 11:32:23
Source: kernel
Description: mmcblk0: error 1 transferring data
-------------------------------------------------------------------------------
Record: 174
Date/Time: May 3 11:32:24
Source: logger
Description: last message repeated 2 times
-------------------------------------------------------------------------------
Record: 175
Date/Time: May 3 11:32:24
Source: kernel
Description: end_request: I/O error, dev mmcblk0, sector 162866
-------------------------------------------------------------------------------
Record: 176
Date/Time: May 3 11:32:24
Source: os[18933]
Description: copyFileTransfer() - number of bytes written to dest: -1 not equal to number read from src: 2048, copy failed
-------------------------------------------------------------------------------
Record: 177
Date/Time: May 3 11:32:26
Source: os[18933]
Description: copyFirmware() - copyFileTransfer of /dev/mmcblk0p5 failed
-------------------------------------------------------------------------------
Record: 178
Date/Time: May 3 11:32:26
Source: os[18933]
Description: threadFirmwareUpdate() - copyFirmware failed
-------------------------------------------------------------------------------
Record: 179
Date/Time: May 3 11:32:26
Source: os[18933]
Description: 2012 FWupdate failed
-------------------------------------------------------------------------------
Record: 180
Date/Time: May 3 11:32:26
Source: os[18933]
Description: 2012 RAC:FWupdate failed
-------------------------------------------------------------------------------
 
I get the same results regardless of which image I upload (1.70, 1.80 or 1.85). I think I have a bad iDRAC. Any ideas?
  • DELL-Rey G

    4 Apprentice

    •

    1068 Posts

    3091

    0

    Posted May 3rd, 2012 11:00

    I hate to say this, but it appears your idrac is off in the weeds. there are 2 possible solutions, one you already know, and thats to pull the power cords to let it reset the idrac. The other is to try  the USC repair package, steps below.

    I would also be extremely curious to know if the system is in a state where even flashing the bios would work. To that end, might I ask if  you would attempt a bios update on the system and let me know if it completes? You would not have to actually reboot the system (unless you want to), I just need to know if the process will go until it gets to the point where it says Finished. if you cant, I understand, if you can, here is the link to the bios file for Windows

    www.dell.com/.../DriverDetails

    idrac fix:

    Download idrac  img firmware update date file from www.dell.com/.../DriverDetails

    File name will be idrac6_1.85_a00_fw_img.exe

    Double click file to extract contents to a folder. Navigate to that folder and note the file named firming.d6

    Login to web gui of idrac that is to be updated. This is done by opening a browser and then entering the IP Address of the idrac.

    In the idrac gui, on the left hand side tree, choose iDRAC Settings, then choose the “update” tab at the top

    In the main body of the gui, browse to the location of the file called firming.d6, then in the main idrac gui, click the upload button.

    The file will begin the upload process.

    The upload process can take anywhere from 1 minute to 5 minutes depending on the speed of the network. If the upload fails, please try at least 2 more times before failing.

    Once the upload is complete,  you will be returned to the idrac gui and will be shown the old version and the new version. The check box that says “Preserve Configuration” is normally checked.

    Then click Next. A popup will ask you if you are sure you want to update, click yes. The update process begins and the progress is shown. If the process fails, please try 2 more times before failing.

    The actual update will take 3 to 5 minutes. After the update is complete, you may close the browser. The idrac will be unavailable for the next 1-2 minutes as it reboots.

  • jhboricuamn

    26 Posts

    3091

    0

    Posted May 3rd, 2012 12:00

    Rey,

    The DRAC upgrade process you are describing is exactly what I'm doing and what fails. I've used image files for v 1.70, 1.80 and 1.85.

    As far as the upgrade of the firmware for the rest of the system components (BIOS, RAID, etc), this 710 is having problems too. I tried to do them via OME and the task never progressed beyond copying the files to the host. Then I created a SUU DVD disk via Repository Manager that has updates for both R710 & R810 servers so I can do the updates via the Lifecycle controller. On the same 710 that I'm having this DRAC issue, the Lifecycle controller doesn't see any updates available on the DVD. On the R810s, the Lifecycle controller works fine with the DVD. My next step is to create a linux bootable update DVD with Repository Manager and attempt the updates there. The whole process on teh 710s has been kinda frustrating though. Have no issues with updating the R810s.

    I haven't tried another 710 for system updates yet as I'm trying to figure the problems with the first one. If the bootable linux DVD fails too, I'll move to another 710 and see what happens.

  • DELL-Rey G

    4 Apprentice

    •

    1068 Posts

    3091

    0

    Posted May 3rd, 2012 14:00

    Understood. i would still be interested in the response the system would give if you ran that bios update manually if you get a chance :emotion-5:

  • jhboricuamn

    26 Posts

    3091

    0

    Posted May 10th, 2012 12:00

    I just attempted this via console. It fails with:

    Failed to update to standby image. Which is pretty much the behavior I'm seeing. The update first uploads the image file and then attempts to flash the idrac with the uploaded image. It is at this 2nd step that it fails.

    I can confidently say that I have a borked iDrac as I've attempted every possible way to do this update:

    1. OME

    2. Direct image upload to iDrac

    3. racadm via tftp

    4. Update via lifecycle controller

    5. manual update via console

    I guess I'll have to get support involved for a part replacement.

  • DELL-Rey G

    4 Apprentice

    •

    1068 Posts

    3091

    0

    Posted May 10th, 2012 13:00

    Did removing power help or still borked?

  • DELL-Rey G

    4 Apprentice

    •

    1068 Posts

    3091

    0

    Posted May 10th, 2012 13:00

    assuming you are in the USA, please instruct tech support to "capture" the failing unit to me when they replace it. "Rey G in IPS"

  • jhboricuamn

    26 Posts

    3091

    0

    Posted May 10th, 2012 13:00

    I physically removed power from the server before each update attempt, so no, it didn't help. Drac is not borked though, meaning I can use it. I just can't get it to update from the firmware it shipped with, v1.40, which is several revisions behind (current is v1.85).

  • DELL-Jeff M

    2 Intern

    •

    793 Posts

    3091

    0

    Posted May 10th, 2012 17:00

    Something I've seen work when trying to update those DRACs is to first update to v1.50, then update to 1.85 or the latest.  The 1.50 revision made quite a few changes, and taking that intermediate step has helped.

    Bear in mind that I've seen the 1.50 and later updates take 30-40 minutes in some cases, so be patient.  I suggest giving them an hour before you consider the update failed.

  • jhboricuamn

    26 Posts

    3100

    0

    Posted May 14th, 2012 11:00

    So I received a new iDRAC module for the server and performed the swap a few minutes ago. I didn't have to reenter IP settings, they were retained. The firmware still shows unchanged as 1.41 after the swap, which makes me believe that those (firmware and settings) are not stored in the iDRAC daughterboard but on the server systemboard? If so, my gut feeling at the moment was that this was not going to solve the problem.

    True enough, tried once again to update the firmware and it failed in exactly the same manner as before. So unless I replace the actual component that is being reprogrammed, this issue won't go away. Can any of the Dell guys here confirm where is the firmware flash stored for the iDRAC? is it on the systemboard? If so, I'm looking at a systemboard replacement and I need to confirm that.

    Jeff - I tried various levels of firmware on this particular box. It's pretty obvious to me by now that this is a hardware issue for 2 reasons:

    1. I upgraded all my other 710s straight from 1.41 to 1.85 without issues.

    2. Even after replacing the daughterboard, the issue persist (see info above on this post).

  • DELL-Rey G

    4 Apprentice

    •

    1068 Posts

    1285

    0

    Posted May 14th, 2012 11:00

    here is what you are running into. On the R610 and above, the system board comes with idrac express, which is a daughter board on the system board. So is they replaced the idrac enterprise module, the idrac express portion is still there. My understanding is that the idrac express is not a field replaceable item, so you would need a system board replacement in order to replace the idrac express. Check with the tech to be sure, but that is my understanding.