R740xd fans stuck high: MFSMinimumLimit jumps 43→82 at OS boot. Persists after
LC purge, BIOS/iDRAC reflash, NIC rollback
System: PowerEdge R740xd, <To protect you, your private information was removed from public view. All private data was saved to your private Case. DELL-Admin>. Dual Xeon Gold 5218, 2x DIMM, HBA330 (NonRAID.Slot.3), Intel X540/I350 rNDC (integrated), no third-party PCIe cards. iDRAC9 Enterprise 7.00.00.183, BIOS 2.27.0, LC 7.00.00.183. OS: Proxmox VE (Debian trixie based).
Symptom: Since a firmware update session on 2026-07-04 (BIOS 2.25.0→2.27.0, HBA330 16.17.00.03→16.17.01.00, Intel rNDC 13.5.7→23.0.8), the calculated minimum fan speed floor (#MFSMinimumLimit in system.thermalsettings) is stuck high. Reproducible pattern on every boot: during POST and before OS load, MFSMinimumLimit=43. Within ~1-2 minutes of the OS booting, it recalculates to 82-86 and stays there. Fans at ~19k RPM constantly. All temperatures are low (inlet 28-29C, CPUs 46-52C, exhaust 36-41C), so this is not a thermal response — it's the calculated floor.
During the July 4 update session there were transient CPU machine check events (CPU0704 both sockets, UEFI0078) and a burst of PWR2270 Intel ME health events, coinciding with a power disturbance at the site. ePSA 4301.74 full diagnostics: clean. rasdaemon on the host since then: zero new MCEs. No PWR2270 since July 4.
Physical flea power drain (both PSUs unplugged, power button 60s, 5 min wait) — floor briefly recalculated to 43, returned to 82+ once OS booted
FullPowerCycle via BIOS.MiscSettings.PowerCycleRequest + config job — same result
BIOS 2.27.0 reflashed (same version, to rewrite ME region) — no change, no new PWR2270
iDRAC reflashed — no change
Intel rNDC firmware rolled back to 13.5.7 (the pre-July-4 version, confirmed in hwinventory Family Version) — no change
systemerase lcdata — LC database fully purged, inventory rebuilt from scratch, LC log empty — floor still jumps 43→82 on OS boot with a virgin inventory
iDRAC Service Module: not installed in the OS (no dcism package, no service)
IPMI raw fan control: correctly blocked on this firmware (expected)
No third-party PCIe present; ThirdPartyPCIFanResponse object not applicable
Question: What does iDRAC re-evaluate at the OS handoff (drivers loading? NIC ports going active? PCIe link state changes?) that raises the calculated MFSMinimumLimit from 43 to 82 on this platform, given a completely clean LC inventory and logs? Is this a known behavior/regression of BIOS 2.27.0 or iDRAC 7.00.00.18x on 14G? Prior to July 4 (BIOS 2.25.0 / rNDC 13.5.7 / older iDRAC) the same physical configuration idled far quieter. Any way to see the per-component LFM/airflow demands feeding the MFS calculation (Enterprise license, not Datacenter)?
Roll Back the HBA330: Revert the HBA330 back to 16.17.00.03. Given the timing of the update and the known behavior of mpt3sas probing, this is the most likely firmware culprit.
Check the TSR: Let the OS boot, wait for the fans to hit 19k RPM, and generate a TSR to catch the algorithm red-handed.
If you generate that TSR or run racadm getsensorinfo while the fans are pinned at 19k RPM, do either the HBA330 or the rNDC show their temperature as "Unknown" or missing?
with kind regards Martin/ Liebe Grüße Martin
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!
Ran racadm getsensorinfo with the fans pinned (78-80% PWM, ~16,200 RPM). Result: neither the HBA330 nor the rNDC expose any temperature sensor at all. The only temperature entries on the system are: CPU1 Temp 45C (Ok), CPU2 Temp 46C (Ok), System Board Inlet 27C (Ok), System Board Exhaust 35C (Ok). No NDC/NIC or HBA temperature entries — not "Unknown", simply absent from the list. The "System Board NDC PG" voltage rail reads Good.
Question: on the R740xd, is the absence of NDC/HBA temperature entries in getsensorinfo expected (read internally via shared memory only), or does their absence itself indicate the thermal algorithm has lost those readings?
I'm generating a TSR now while the fans are pinned, and will proceed with the HBA330 rollback to 16.17.00.03 via OS DUP (the LC blocks HBA downgrades and the rollback image was lost to a systemerase lcdata during troubleshooting). Will report back.
I attatch the console output for the command you asked.
The complete absence of discrete temperature entries for the HBA330 and the rNDC in getsensorinfo is completely normal and expected on the PowerEdge R740xd. Their absence does not indicate that the thermal algorithm has lost its readings.
Unlike CPUs or the system board inlet, the iDRAC thermal algorithm does not look at real-time, active temperature sensors for these specific expansion components. Instead, it calculates the fan floor using a predictive lookup table based on the system configuration
Once your Technical Support Report (TSR) is ready, unzip it and search the log files (sysconfig_inventory.xml or raw logs) for these exact anchors:
Search for Cooling Request or LFM: Find whether Slot 3 (HBA) or the Integrated NDC is explicitly demanding the high percentage.
Search for ThirdParty or Unknown Card: See if the corrupted state has caused iDRAC to misidentify either piece of native hardware.
with kind regards Martin/ Liebe Grüße Martin
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!
Rollback done and tested. HBA330 downgraded to 16.17.00.05 (16.17.00.03 is not published on the support site; 16.17.00.05 is the closest available build from the same 2019-era 16.17.00.x family). Confirmed in swinventory: NonRAID.Slot.3-1 Current Version = 16.17.00.05. Host rebooted, 5+ minutes with the OS fully up. Result: #MFSMinimumLimit is still 86. No change. The HBA330 firmware is ruled out as the cause.
Full investigation summary, since this narrows it down:
Confirmed NOT the cause (each tested and reverted/reproduced):
iDRAC thermal algorithm — downgraded to 6.10.80.00, no change (and the system was already on 7.x when it ran quiet)
rNDC firmware — rolled back from 23.0.8 (swinventory reports 20.5.13; hwinventory FamilyVersion 13.5.7), no change
HBA330 firmware — now on 16.17.00.05, no change
LC database corruption — systemerase lcdata, inventory rebuilt from scratch, LC log empty, no change
MCE / thermal latch — ePSA clean, zero MCEs since July 4 (rasdaemon monitoring on the host), logs empty
System Profile and thermal config — all at defaults and persistent (ThermalProfile=Minimum Power, MinimumFanSpeed=255/Default, FanSpeedOffset=Off, SysProfile=PerfPerWattOptimizedOs)
No third-party PCIe present; no card misidentification (Slot 3: 3rdPartyCard=No, CardType=NonRAID, LFMMode=Automatic, TargetLFM=Temperature Controlled)
TSR internal findings:
The internal D4D telemetry tracks the rNDC (NIC.Integrated.1-1-1) at a steady 55–57 °C straight through the OS handoff — the floor jumps 43→82 with no corresponding change on any temperature sensor (CPUs, DIMMs, PSUs, inlet, exhaust all steady). It behaves like the predictive lookup table applying a higher requirement in the OS-active device state, not a closed-loop thermal response.
The unencrypted debug logs (dumplogs, idraclogs, CM_Logs, sidebandlog, pm_debug, iDRACStateInfo) contain no thermal-algorithm decision entries at all — the thermal daemon (thermald/thermpop) is running, but its floor calculation is not written anywhere accessible.
The only place that data can live is hardware/PlatformData/thermaldata.zip, which is password-protected (Dell-internal) and contains: ThermalTable.txt (1.1MB), pbt_conf.txt (3.8MB), thmtestgaoutput.log (352KB), and throttle_run.txt.
The only variable still changed vs. the quiet pre-July-4 configuration is BIOS 2.25.0 → 2.27.0. Same physical hardware, same iDRAC generation, ran far quieter before that update. (System CPLD is at 1.0.1 from 2018 — never reflashed during any of this.)
Two questions:
Is there a documented thermal-table / fan-curve change between R740xd BIOS 2.25.0 and 2.27.0?
Since the floor is set by the predictive table, the protected thermaldata archive should show exactly which component/config entry demands ~82% in the OS-active state. Can it be inspected through my case? Happy to provide the full TSR via DM.
@DellMartinS One more finding from digging through the TSR's raw iDRAC logs (dbglog/logs/idraclogs), which may be exactly the "red-handed" evidence:
During the thermal daemon's boot-time parse of the platform thermal configuration, it logs validation failures on precisely the structures that govern the fan floor:
252 lines of thParser_ValidateVariableSize:472 key[energy_balance_table.N:eb_table_lfm_pci_slot_entries / eb_table_lfm_pci_slotnum.0-7] - output variable size is 0 bytes, across at least 28 table indexes (energy_balance_table.0–.26, .40). If the energy balance table is the predictive lookup table you described, all of its PCIe-slot LFM entries are parsing as empty.
thParser_GetVariableString:829 key[global_fan_behaviors:global_abl_fan_speed_min_pwr_snd_enabled] string value (3 bytes) too large — and the same for global_abl_fan_speed_max_perf_enabled. Note the active System Profile on this box IS Minimum Power.
Followed by thPlatform_I2CInit / thCPU_Init / thDIMM_Init: I2CTopology.bin parsing error, read_config_info() failed! status=2.
No successful thParser load lines anywhere in the log.
I can't tell from one system whether these are normal boot verbosity or evidence that the platform thermal data (populated by the BIOS) is malformed on this machine — can you compare against a healthy R740xd, or confirm whether these parse failures would cause the daemon to fall back to a conservative floor? Combined with the earlier findings, this is consistent with BIOS 2.25.0→2.27.0 being the remaining changed variable.
No, there is no officially documented change in the public release notes for BIOS 2.27.0 regarding fan curves or energy balance modifications. However, the release payload for 2.27.0 includes critical security mitigation microcode updates targeting the Intel Management Engine (ME) and updated structural tables handed down to the unified system ACPI/SMBIOS layers.
The Prescribed Path to Restore Table Parsing
To force a complete re-allocation of the damaged platform binary mappings, you must execute a structural platform downgrade followed by a sequential step-up, accompanied by a CPLD payload force.
Step 1: Execute a Complete Multi-Subsystem Version Downgrade
You must step the system firmware block backward to force the platform to write old, known-good structural shapes back to the SPI regions.
Flash BIOS back to version 2.25.0 via the iDRAC interface or the OS update payload.
Flash iDRAC back to version 6.10.80.00 (or any 6.00.x branch release).
Shut the system down completely.
Step 2: Clear the Volatile CPLD Registers
Because the CPLD holds the physical configuration registers for the chassis fan arrays and I2C lines:
Pull both AC power cords.
Hold the physical front panel power button down for a minimum of 60 seconds.
Leave the unit completely disconnected from AC power for 30 full minutes. This allows the internal capacitance on the standby power planes to bleed entirely, clearing the volatile hardware states of the 1.0.1 CPLD logic loops.
Step 3: Flash the Modern CPLD Update Payload
Dell released critical stability fixes for the 14G CPLD framework to prevent data link corruption between the BIOS and the thermal managers.
Re-seat power cords, boot the server, and immediately update the System CPLD firmware from 1.0.1 to the latest available release (typically 1.0.8 or newer depending on motherboard revision) using the Dell Update Package.
Once the CPLD flash completes, the system will perform an automatic cold reset where the fans will scream at 100% for approximately 3 minutes while the CPLD writes its fresh registers.
Step 4: Step Forward and Re-Verify Logs
Once the system stabilizes under BIOS 2.25.0 / CPLD Update / iDRAC 6.x, pull a fresh TSR and check dbglog/logs/idraclogs.
Confirm that the thParser lines now read out successful initialization of the energy_balance_table and that the I2CTopology.bin parsing error has completely vanished.
Only after confirming a clean log parse should you attempt to cleanly upgrade back to BIOS 2.27.0 and iDRAC 7.00.00.183.
If you proceed with this structural re-flash matrix, let me know:
Did forcing the CPLD firmware update clear the I2CTopology.bin status=2 error?
Does the energy_balance_table now log positive byte sizing allocations upon system initialization?
with kind regards Martin/ Liebe Grüße Martin
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!
@DellMartinS Will do this this afternoon. BTW a side question for the future. Is it possible to use non certified GPUs on the R740XD without fans going crazy?
Yes, it is entirely possible to run non-certified or consumer GPUs (such as NVIDIA GeForce RTX or workstation cards) in a PowerEdge R740xd without the fans running at maximum speed, but difficult and not really supported.
By default, inserting an uncertified PCIe card triggers a safety mechanism called the Third-Party PCIe Card Cooling Response, which sets a high baseline fan floor (~60–100%) because iDRAC cannot read the card's internal thermal sensors. You can easily disable this behavior using raw RACADM commands.
Disable the third-party fan response:
racadm set system.thermalsettings.ThirdPartyPCICardInflowVentMitigation Disabled
Check the current status:
racadm get system.thermalsettings.ThirdPartyPCICardInflowVentMitigation
with kind regards Martin/ Liebe Grüße Martin
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!
@DellMartinS CPLD updated from 1.0.1 to 1.1.4 (latest published for R740/R740XD; note there is no "1.0.8" release). Result: no change — #MFSMinimumLimit still 82-91 with the OS up.
More significantly, comparing the thermal daemon's boot parse before and after (TSR debug logs from July 8 vs July 15): bit-for-bit identical. Same 252 energy_balance_table.N:eb_table_lfm_pci_slot* empty-value validation failures, same 3 I2CTopology.bin parsing error status=2 lines (thPlatform/thCPU/thDIMM init), same global_abl_fan_speed_min_pwr_snd_enabled / max_perf_enabled "string value too large" errors. Additionally, the protected ThermalTable.txt (1,130,240 bytes) and pbt_conf.txt (3,835,126 bytes) are byte-identical in size across both collections — static content unaffected by iDRAC, HBA, NIC or CPLD changes.
The elimination matrix is now complete: these parse errors and the raised floor are invariant across iDRAC 7.00.00.18x/6.10.80.00, CPLD 1.0.1/1.1.4, rNDC 23.0.8/20.5.13, HBA330 16.17.01.00/16.17.00.05, and a fully purged LC database. The only July-4 change still in place is BIOS 2.25.0 → 2.27.0 — which is also the component that generates the platform thermal tables the daemon is failing to parse. Proceeding with the BIOS 2.25.0 downgrade as the final isolation test. Also — the racadm attribute suggested for the GPU question (ThirdPartyPCICardInflowVentMitigation) doesn't appear to exist in any iDRAC attribute registry; the documented one is ThirdPartyPCIFanResponse. Could you confirm whether these thParser validation failures appear on healthy R740xd systems, and whether the thermaldata archive can be decrypted through my case?
DM Me if you want and I send you full TSR zip...
DellMartinS
Moderator
•
3883 Publicaciones
•
25810 Puntos
0
0
Publicado July 8th, 2026 05:45
Roll Back the HBA330: Revert the HBA330 back to 16.17.00.03. Given the timing of the update and the known behavior of mpt3sas probing, this is the most likely firmware culprit.
Check the TSR: Let the OS boot, wait for the fans to hit 19k RPM, and generate a TSR to catch the algorithm red-handed.
If you generate that TSR or run racadm getsensorinfo while the fans are pinned at 19k RPM, do either the HBA330 or the rNDC show their temperature as "Unknown" or missing?
with kind regards Martin/ Liebe Grüße Martin
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!
DELL-Martin S
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell