Having tons of issues updating DRAC firmware on 710s
I have several R710s that are giving me a lot of headaches with the iDRAC6 firmware update. They currently are at (v1.41)Out of 5 hosts, only 1 updated to the latest version (1.85). The remaining 4 not only failed but are stuck in update hell. Meaning if I SSH to the drac and issue a 'racadm fwupdate -s' command, it still outputs 'Preparing for firmware update'. Because of this, I can't reset the DRAC. The only option to get the DRAC out of this state is to completely power off the server and unplug the power cords. Since these are production servers, that becomes a scheduling issue.
In any case, I was able to power down and unplug the power cords to 1 server out of the 4, which allowed the DRAC to get out of that 'updating' funk state. When tried the update the first time I used the Dell DRAC tool, I decided not to go that route. So now I'm doing by uploading the image file directly to the DRAC via the web interface. It still fails.
I've tried v 1.70, 1.80 and 1.85. They all fail on this host at the same spot, shortly after showing 30% progress on the Updating webpage. When I ssh to the drac and issue a 'racadm gettracelog', this is what it shows:
------------------------------------------------------------------------------- Record: 147 Date/Time: May 3 11:32:00 Source: os[407] Description: 2012 FWupdate checksumming image ------------------------------------------------------------------------------- Record: 148 Date/Time: May 3 11:32:00 Source: os[407] Description: 2012 RAC:FWupdate checksumming image ------------------------------------------------------------------------------- Record: 149 Date/Time: May 3 11:32:00 Source: os[407] Description: firmwareUpdate() - set the firmware destination device : 2 ------------------------------------------------------------------------------- Record: 150 Date/Time: May 3 11:32:00 Source: os[407] Description: firmwareUpdate() - get the firmware destination device : 2 ------------------------------------------------------------------------------- Record: 151 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 FWupdate initialization complete ------------------------------------------------------------------------------- Record: 152 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 RAC:FWupdate initialization complete ------------------------------------------------------------------------------- Record: 153 Date/Time: May 3 11:32:07 Source: os[18933] Description: readMBR() - MBR PARTITION 1 STATUS: 0x80 ------------------------------------------------------------------------------- Record: 154 Date/Time: May 3 11:32:07 Source: os[18933] Description: readMBR() - partition 1 is currently active, partition 3 is currently inactive ------------------------------------------------------------------------------- Record: 155 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 FWupdate checksumming image ------------------------------------------------------------------------------- Record: 156 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 RAC:FWupdate checksumming image ------------------------------------------------------------------------------- Record: 157 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 FWupdate checksumming image ------------------------------------------------------------------------------- Record: 158 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 RAC:FWupdate checksumming image ------------------------------------------------------------------------------- Record: 159 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 FWupdate verify image headers ------------------------------------------------------------------------------- Record: 160 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 RAC:FWupdate verify image headers ------------------------------------------------------------------------------- Record: 161 Date/Time: May 3 11:32:07 Source: os[18933] Description: readHeader() ------------------------------------------------------------------------------- Record: 162 Date/Time: May 3 11:32:07 Source: os[18933] Description: readHeader() - header information: uHeaderCRC:0x7A97E3E7 uHeaderVersion:0x1 uImageType:0x1 uNumPartitions:0x3 uFWVersion:0x115001 uFWTotalImageSize:0x3477E00 uBootVerMajor:0x333B0 uBootVerMinor:0x333B4 ------------------------------------------------------------------------------- Record: 163 Date/Time: May 3 11:32:07 Source: os[18933] Description: readHeader() - partition 1 info: uOffset:0x0200 uSize:0x445BE0 uCRC:0x853463B1 ------------------------------------------------------------------------------- Record: 164 Date/Time: May 3 11:32:07 Source: os[18933] Description: readHeader() - partition 2 info: uOffset:0x445E00 uSize:0x300F000 uCRC:0x8D219434 ------------------------------------------------------------------------------- Record: 165 Date/Time: May 3 11:32:07 Source: os[18933] Description: readHeader() - partition 3 info: uOffset:0x3454E00 uSize:0x22E38 uCRC:0x918B375A ------------------------------------------------------------------------------- Record: 166 Date/Time: May 3 11:32:07 Source: os[18933] Description: isCorrectBoardType() - file and board have the same board type ------------------------------------------------------------------------------- Record: 167 Date/Time: May 3 11:32:07 Source: os[18933] Description: isCorrectBoardType() - board type is Dell Whoville, board type is correct ------------------------------------------------------------------------------- Record: 168 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 FWupdate programming flash ------------------------------------------------------------------------------- Record: 169 Date/Time: May 3 11:32:07 Source: os[18933] Description: 2012 RAC:FWupdate programming flash ------------------------------------------------------------------------------- Record: 170 Date/Time: May 3 11:32:07 Source: os[18933] Description: copyFileTransfer() - szSrcPath:/mnt/scratchpad/idrac_fw.bin szDestPath:/dev/mmcblk0p3 nBytesToTransfer:4479968 nSrcOffset:512 ------------------------------------------------------------------------------- Record: 171 Date/Time: May 3 11:32:14 Source: os[18933] Description: copyFirmware() - copyFileTransfer of /dev/mmcblk0p3 succeeded ------------------------------------------------------------------------------- Record: 172 Date/Time: May 3 11:32:14 Source: os[18933] Description: copyFileTransfer() - szSrcPath:/mnt/scratchpad/idrac_fw.bin szDestPath:/dev/mmcblk0p5 nBytesToTransfer:50393088 nSrcOffset:4480512 ------------------------------------------------------------------------------- Record: 173 Date/Time: May 3 11:32:23 Source: kernel Description: mmcblk0: error 1 transferring data ------------------------------------------------------------------------------- Record: 174 Date/Time: May 3 11:32:24 Source: logger Description: last message repeated 2 times ------------------------------------------------------------------------------- Record: 175 Date/Time: May 3 11:32:24 Source: kernel Description: end_request: I/O error, dev mmcblk0, sector 162866 ------------------------------------------------------------------------------- Record: 176 Date/Time: May 3 11:32:24 Source: os[18933] Description: copyFileTransfer() - number of bytes written to dest: -1 not equal to number read from src: 2048, copy failed ------------------------------------------------------------------------------- Record: 177 Date/Time: May 3 11:32:26 Source: os[18933] Description: copyFirmware() - copyFileTransfer of /dev/mmcblk0p5 failed ------------------------------------------------------------------------------- Record: 178 Date/Time: May 3 11:32:26 Source: os[18933] Description: threadFirmwareUpdate() - copyFirmware failed ------------------------------------------------------------------------------- Record: 179 Date/Time: May 3 11:32:26 Source: os[18933] Description: 2012 FWupdate failed ------------------------------------------------------------------------------- Record: 180 Date/Time: May 3 11:32:26 Source: os[18933] Description: 2012 RAC:FWupdate failed -------------------------------------------------------------------------------
I get the same results regardless of which image I upload (1.70, 1.80 or 1.85). I think I have a bad iDRAC. Any ideas?
I hate to say this, but it appears your idrac is off in the weeds. there are 2 possible solutions, one you already know, and thats to pull the power cords to let it reset the idrac. The other is to try the USC repair package, steps below.
I would also be extremely curious to know if the system is in a state where even flashing the bios would work. To that end, might I ask if you would attempt a bios update on the system and let me know if it completes? You would not have to actually reboot the system (unless you want to), I just need to know if the process will go until it gets to the point where it says Finished. if you cant, I understand, if you can, here is the link to the bios file for Windows
Double click file to extract contents to a folder. Navigate to that folder and note the file named firming.d6
Login to web gui of idrac that is to be updated. This is done by opening a browser and then entering the IP Address of the idrac.
In the idrac gui, on the left hand side tree, choose iDRAC Settings, then choose the “update” tab at the top
In the main body of the gui, browse to the location of the file called firming.d6, then in the main idrac gui, click the upload button.
The file will begin the upload process.
The upload process can take anywhere from 1 minute to 5 minutes depending on the speed of the network. If the upload fails, please try at least 2 more times before failing.
Once the upload is complete, you will be returned to the idrac gui and will be shown the old version and the new version. The check box that says “Preserve Configuration” is normally checked.
Then click Next. A popup will ask you if you are sure you want to update, click yes. The update process begins and the progress is shown. If the process fails, please try 2 more times before failing.
The actual update will take 3 to 5 minutes. After the update is complete, you may close the browser. The idrac will be unavailable for the next 1-2 minutes as it reboots.
The DRAC upgrade process you are describing is exactly what I'm doing and what fails. I've used image files for v 1.70, 1.80 and 1.85.
As far as the upgrade of the firmware for the rest of the system components (BIOS, RAID, etc), this 710 is having problems too. I tried to do them via OME and the task never progressed beyond copying the files to the host. Then I created a SUU DVD disk via Repository Manager that has updates for both R710 & R810 servers so I can do the updates via the Lifecycle controller. On the same 710 that I'm having this DRAC issue, the Lifecycle controller doesn't see any updates available on the DVD. On the R810s, the Lifecycle controller works fine with the DVD. My next step is to create a linux bootable update DVD with Repository Manager and attempt the updates there. The whole process on teh 710s has been kinda frustrating though. Have no issues with updating the R810s.
I haven't tried another 710 for system updates yet as I'm trying to figure the problems with the first one. If the bootable linux DVD fails too, I'll move to another 710 and see what happens.
Failed to update to standby image. Which is pretty much the behavior I'm seeing. The update first uploads the image file and then attempts to flash the idrac with the uploaded image. It is at this 2nd step that it fails.
I can confidently say that I have a borked iDrac as I've attempted every possible way to do this update:
1. OME
2. Direct image upload to iDrac
3. racadm via tftp
4. Update via lifecycle controller
5. manual update via console
I guess I'll have to get support involved for a part replacement.
I physically removed power from the server before each update attempt, so no, it didn't help. Drac is not borked though, meaning I can use it. I just can't get it to update from the firmware it shipped with, v1.40, which is several revisions behind (current is v1.85).
Something I've seen work when trying to update those DRACs is to first update to v1.50, then update to 1.85 or the latest. The 1.50 revision made quite a few changes, and taking that intermediate step has helped.
Bear in mind that I've seen the 1.50 and later updates take 30-40 minutes in some cases, so be patient. I suggest giving them an hour before you consider the update failed.
So I received a new iDRAC module for the server and performed the swap a few minutes ago. I didn't have to reenter IP settings, they were retained. The firmware still shows unchanged as 1.41 after the swap, which makes me believe that those (firmware and settings) are not stored in the iDRAC daughterboard but on the server systemboard? If so, my gut feeling at the moment was that this was not going to solve the problem.
True enough, tried once again to update the firmware and it failed in exactly the same manner as before. So unless I replace the actual component that is being reprogrammed, this issue won't go away. Can any of the Dell guys here confirm where is the firmware flash stored for the iDRAC? is it on the systemboard? If so, I'm looking at a systemboard replacement and I need to confirm that.
Jeff - I tried various levels of firmware on this particular box. It's pretty obvious to me by now that this is a hardware issue for 2 reasons:
1. I upgraded all my other 710s straight from 1.41 to 1.85 without issues.
2. Even after replacing the daughterboard, the issue persist (see info above on this post).
here is what you are running into. On the R610 and above, the system board comes with idrac express, which is a daughter board on the system board. So is they replaced the idrac enterprise module, the idrac express portion is still there. My understanding is that the idrac express is not a field replaceable item, so you would need a system board replacement in order to replace the idrac express. Check with the tech to be sure, but that is my understanding.
DELL-Rey G
4 Apprentice
•
1068 Posts
3091
0
Posted May 3rd, 2012 11:00
I hate to say this, but it appears your idrac is off in the weeds. there are 2 possible solutions, one you already know, and thats to pull the power cords to let it reset the idrac. The other is to try the USC repair package, steps below.
I would also be extremely curious to know if the system is in a state where even flashing the bios would work. To that end, might I ask if you would attempt a bios update on the system and let me know if it completes? You would not have to actually reboot the system (unless you want to), I just need to know if the process will go until it gets to the point where it says Finished. if you cant, I understand, if you can, here is the link to the bios file for Windows
www.dell.com/.../DriverDetails
idrac fix:
Download idrac img firmware update date file from www.dell.com/.../DriverDetails
File name will be idrac6_1.85_a00_fw_img.exe
Double click file to extract contents to a folder. Navigate to that folder and note the file named firming.d6
Login to web gui of idrac that is to be updated. This is done by opening a browser and then entering the IP Address of the idrac.
In the idrac gui, on the left hand side tree, choose iDRAC Settings, then choose the “update” tab at the top
In the main body of the gui, browse to the location of the file called firming.d6, then in the main idrac gui, click the upload button.
The file will begin the upload process.
The upload process can take anywhere from 1 minute to 5 minutes depending on the speed of the network. If the upload fails, please try at least 2 more times before failing.
Once the upload is complete, you will be returned to the idrac gui and will be shown the old version and the new version. The check box that says “Preserve Configuration” is normally checked.
Then click Next. A popup will ask you if you are sure you want to update, click yes. The update process begins and the progress is shown. If the process fails, please try 2 more times before failing.
The actual update will take 3 to 5 minutes. After the update is complete, you may close the browser. The idrac will be unavailable for the next 1-2 minutes as it reboots.