SIN RESOLVER

lilpandemio

actualizó

4 months ago

L

lilpandemio

1 Rookie

•

6 Publicaciones

•

19 Puntos

0

173

July 5th, 2026 23:41

R740xd fans stuck high: MFSMinimumLimit jumps 43→82 at OS boot. Persists after LC purge, BIOS/iDRAC reflash, NIC rollback

System: PowerEdge R740xd, <To protect you, your private information was removed from public view. All private data was saved to your private Case. DELL-Admin>. Dual Xeon Gold 5218, 2x DIMM, HBA330 (NonRAID.Slot.3), Intel X540/I350 rNDC (integrated), no third-party PCIe cards. iDRAC9 Enterprise 7.00.00.183, BIOS 2.27.0, LC 7.00.00.183. OS: Proxmox VE (Debian trixie based).

Symptom: Since a firmware update session on 2026-07-04 (BIOS 2.25.0→2.27.0, HBA330 16.17.00.03→16.17.01.00, Intel rNDC 13.5.7→23.0.8), the calculated minimum fan speed floor (#MFSMinimumLimit in system.thermalsettings) is stuck high. Reproducible pattern on every boot: during POST and before OS load, MFSMinimumLimit=43. Within ~1-2 minutes of the OS booting, it recalculates to 82-86 and stays there. Fans at ~19k RPM constantly. All temperatures are low (inlet 28-29C, CPUs 46-52C, exhaust 36-41C), so this is not a thermal response — it's the calculated floor.

During the July 4 update session there were transient CPU machine check events (CPU0704 both sockets, UEFI0078) and a burst of PWR2270 Intel ME health events, coinciding with a power disturbance at the site. ePSA 4301.74 full diagnostics: clean. rasdaemon on the host since then: zero new MCEs. No PWR2270 since July 4.

Everything already tried (all verified, none fixed it):

  • ThermalProfile=Minimum Power, MinimumFanSpeed=255 (Default), FanSpeedOffset=Off — all applied and persistent
  • BIOS SysProfile changed to PerfPerWattOptimizedOs via config job — applied and persists across reboots (confirmed via BIOS.SysProfileSettings)
  • clrsel (SEL cleared)
  • iDRAC factory reset (racresetcfg, preserving network/user)
  • Physical flea power drain (both PSUs unplugged, power button 60s, 5 min wait) — floor briefly recalculated to 43, returned to 82+ once OS booted
  • FullPowerCycle via BIOS.MiscSettings.PowerCycleRequest + config job — same result
  • BIOS 2.27.0 reflashed (same version, to rewrite ME region) — no change, no new PWR2270
  • iDRAC reflashed — no change
  • Intel rNDC firmware rolled back to 13.5.7 (the pre-July-4 version, confirmed in hwinventory Family Version) — no change
  • systemerase lcdata — LC database fully purged, inventory rebuilt from scratch, LC log empty — floor still jumps 43→82 on OS boot with a virgin inventory
  • iDRAC Service Module: not installed in the OS (no dcism package, no service)
  • IPMI raw fan control: correctly blocked on this firmware (expected)
  • No third-party PCIe present; ThirdPartyPCIFanResponse object not applicable

Question: What does iDRAC re-evaluate at the OS handoff (drivers loading? NIC ports going active? PCIe link state changes?) that raises the calculated MFSMinimumLimit from 43 to 82 on this platform, given a completely clean LC inventory and logs? Is this a known behavior/regression of BIOS 2.27.0 or iDRAC 7.00.00.18x on 14G? Prior to July 4 (BIOS 2.25.0 / rNDC 13.5.7 / older iDRAC) the same physical configuration idled far quieter. Any way to see the per-component LFM/airflow demands feeding the MFS calculation (Enterprise license, not Datacenter)?

(editado)

  • DellMartinS

    Moderator

    •

    3883 Publicaciones

    •

    25810 Puntos

    0

    0

    Publicado July 8th, 2026 05:45

    Roll Back the HBA330: Revert the HBA330 back to 16.17.00.03. Given the timing of the update and the known behavior of mpt3sas probing, this is the most likely firmware culprit.

    Check the TSR: Let the OS boot, wait for the fans to hit 19k RPM, and generate a TSR to catch the algorithm red-handed.

    If you generate that TSR or run racadm getsensorinfo while the fans are pinned at 19k RPM, do either the HBA330 or the rNDC show their temperature as "Unknown" or missing?

    with kind regards Martin/ Liebe Grüße Martin

     

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

     

    DELL-Martin S

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell


  • lilpandemio

    1 Rookie

    •

    6 Publicaciones

    •

    19 Puntos

    0

    0

    Publicado July 8th, 2026 13:33

    @DellMartinS​ 

    Hi Martin, thanks for the pointer.

    Ran racadm getsensorinfo with the fans pinned (78-80% PWM, ~16,200 RPM). Result: neither the HBA330 nor the rNDC expose any temperature sensor at all. The only temperature entries on the system are: CPU1 Temp 45C (Ok), CPU2 Temp 46C (Ok), System Board Inlet 27C (Ok), System Board Exhaust 35C (Ok). No NDC/NIC or HBA temperature entries — not "Unknown", simply absent from the list. The "System Board NDC PG" voltage rail reads Good.

    Question: on the R740xd, is the absence of NDC/HBA temperature entries in getsensorinfo expected (read internally via shared memory only), or does their absence itself indicate the thermal algorithm has lost those readings?

    I'm generating a TSR now while the fans are pinned, and will proceed with the HBA330 rollback to 16.17.00.03 via OS DUP (the LC blocks HBA downgrades and the rollback image was lost to a systemerase lcdata during troubleshooting). Will report back.

    I attatch the console output for the command you asked.

    (editado)

    1 archivo adjunto

  • DellMartinS

    Moderator

    •

    3883 Publicaciones

    •

    25810 Puntos

    0

    0

    Publicado July 8th, 2026 13:54

    OK

    The complete absence of discrete temperature entries for the HBA330 and the rNDC in getsensorinfo is completely normal and expected on the PowerEdge R740xd. Their absence does not indicate that the thermal algorithm has lost its readings.
    Unlike CPUs or the system board inlet, the iDRAC thermal algorithm does not look at real-time, active temperature sensors for these specific expansion components. Instead, it calculates the fan floor using a predictive lookup table based on the system configuration

    Once your Technical Support Report (TSR) is ready, unzip it and search the log files (sysconfig_inventory.xml or raw logs) for these exact anchors:
    1. Search for Cooling Request or LFM: Find whether Slot 3 (HBA) or the Integrated NDC is explicitly demanding the high percentage.
    2. Search for ThirdParty or Unknown Card: See if the corrupted state has caused iDRAC to misidentify either piece of native hardware.
     

    with kind regards Martin/ Liebe Grüße Martin

     

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

     

    DELL-Martin S

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell


  • lilpandemio

    1 Rookie

    •

    6 Publicaciones

    •

    19 Puntos

    0

    0

    Publicado July 8th, 2026 15:08

    @Dell-Martin S

    Rollback done and tested. HBA330 downgraded to 16.17.00.05 (16.17.00.03 is not published on the support site; 16.17.00.05 is the closest available build from the same 2019-era 16.17.00.x family). Confirmed in swinventory: NonRAID.Slot.3-1 Current Version = 16.17.00.05. Host rebooted, 5+ minutes with the OS fully up. Result: #MFSMinimumLimit is still 86. No change. The HBA330 firmware is ruled out as the cause.

    Full investigation summary, since this narrows it down:

    Confirmed NOT the cause (each tested and reverted/reproduced):

    • iDRAC thermal algorithm — downgraded to 6.10.80.00, no change (and the system was already on 7.x when it ran quiet)
    • rNDC firmware — rolled back from 23.0.8 (swinventory reports 20.5.13; hwinventory FamilyVersion 13.5.7), no change
    • HBA330 firmware — now on 16.17.00.05, no change
    • LC database corruption — systemerase lcdata, inventory rebuilt from scratch, LC log empty, no change
    • MCE / thermal latch — ePSA clean, zero MCEs since July 4 (rasdaemon monitoring on the host), logs empty
    • System Profile and thermal config — all at defaults and persistent (ThermalProfile=Minimum Power, MinimumFanSpeed=255/Default, FanSpeedOffset=Off, SysProfile=PerfPerWattOptimizedOs)
    • No third-party PCIe present; no card misidentification (Slot 3: 3rdPartyCard=No, CardType=NonRAID, LFMMode=Automatic, TargetLFM=Temperature Controlled)

    TSR internal findings:

    • The internal D4D telemetry tracks the rNDC (NIC.Integrated.1-1-1) at a steady 55–57 °C straight through the OS handoff — the floor jumps 43→82 with no corresponding change on any temperature sensor (CPUs, DIMMs, PSUs, inlet, exhaust all steady). It behaves like the predictive lookup table applying a higher requirement in the OS-active device state, not a closed-loop thermal response.
    • The unencrypted debug logs (dumplogs, idraclogs, CM_Logs, sidebandlog, pm_debug, iDRACStateInfo) contain no thermal-algorithm decision entries at all — the thermal daemon (thermald/thermpop) is running, but its floor calculation is not written anywhere accessible.
    • The only place that data can live is hardware/PlatformData/thermaldata.zip, which is password-protected (Dell-internal) and contains: ThermalTable.txt (1.1MB), pbt_conf.txt (3.8MB), thmtestgaoutput.log (352KB), and throttle_run.txt.

    The only variable still changed vs. the quiet pre-July-4 configuration is BIOS 2.25.0 → 2.27.0. Same physical hardware, same iDRAC generation, ran far quieter before that update. (System CPLD is at 1.0.1 from 2018 — never reflashed during any of this.)

    Two questions:

    1. Is there a documented thermal-table / fan-curve change between R740xd BIOS 2.25.0 and 2.27.0?
    2. Since the floor is set by the predictive table, the protected thermaldata archive should show exactly which component/config entry demands ~82% in the OS-active state. Can it be inspected through my case? Happy to provide the full TSR via DM.
  • lilpandemio

    1 Rookie

    •

    6 Publicaciones

    •

    19 Puntos

    0

    0

    Publicado July 8th, 2026 15:21

    @DellMartinS  One more finding from digging through the TSR's raw iDRAC logs (dbglog/logs/idraclogs), which may be exactly the "red-handed" evidence:

    During the thermal daemon's boot-time parse of the platform thermal configuration, it logs validation failures on precisely the structures that govern the fan floor:

    • 252 lines of thParser_ValidateVariableSize:472 key[energy_balance_table.N:eb_table_lfm_pci_slot_entries / eb_table_lfm_pci_slotnum.0-7] - output variable size is 0 bytes, across at least 28 table indexes (energy_balance_table.0–.26, .40). If the energy balance table is the predictive lookup table you described, all of its PCIe-slot LFM entries are parsing as empty.
    • thParser_GetVariableString:829 key[global_fan_behaviors:global_abl_fan_speed_min_pwr_snd_enabled] string value (3 bytes) too large — and the same for global_abl_fan_speed_max_perf_enabled. Note the active System Profile on this box IS Minimum Power.
    • Followed by thPlatform_I2CInit / thCPU_Init / thDIMM_Init: I2CTopology.bin parsing error, read_config_info() failed! status=2.
    • No successful thParser load lines anywhere in the log.

    I can't tell from one system whether these are normal boot verbosity or evidence that the platform thermal data (populated by the BIOS) is malformed on this machine — can you compare against a healthy R740xd, or confirm whether these parse failures would cause the daemon to fall back to a conservative floor? Combined with the earlier findings, this is consistent with BIOS 2.25.0→2.27.0 being the remaining changed variable.

    (editado)

  • DellMartinS

    Moderator

    •

    3883 Publicaciones

    •

    25810 Puntos

    0

    0

    Publicado July 9th, 2026 06:54

    Hi,

    No, there is no officially documented change in the public release notes for BIOS 2.27.0 regarding fan curves or energy balance modifications. However, the release payload for 2.27.0 includes critical security mitigation microcode updates targeting the Intel Management Engine (ME) and updated structural tables handed down to the unified system ACPI/SMBIOS layers.

     

     

    The Prescribed Path to Restore Table Parsing
    To force a complete re-allocation of the damaged platform binary mappings, you must execute a structural platform downgrade followed by a sequential step-up, accompanied by a CPLD payload force.
     
    Step 1: Execute a Complete Multi-Subsystem Version Downgrade
    You must step the system firmware block backward to force the platform to write old, known-good structural shapes back to the SPI regions.
    1. Flash BIOS back to version 2.25.0 via the iDRAC interface or the OS update payload.
    2. Flash iDRAC back to version 6.10.80.00 (or any 6.00.x branch release).
    3. Shut the system down completely.
     
    Step 2: Clear the Volatile CPLD Registers
    Because the CPLD holds the physical configuration registers for the chassis fan arrays and I2C lines:
    1. Pull both AC power cords.
    2. Hold the physical front panel power button down for a minimum of 60 seconds.
    3. Leave the unit completely disconnected from AC power for 30 full minutes. This allows the internal capacitance on the standby power planes to bleed entirely, clearing the volatile hardware states of the 1.0.1 CPLD logic loops.
     
    Step 3: Flash the Modern CPLD Update Payload
    Dell released critical stability fixes for the 14G CPLD framework to prevent data link corruption between the BIOS and the thermal managers.
    1. Re-seat power cords, boot the server, and immediately update the System CPLD firmware from 1.0.1 to the latest available release (typically 1.0.8 or newer depending on motherboard revision) using the Dell Update Package.
    2. Once the CPLD flash completes, the system will perform an automatic cold reset where the fans will scream at 100% for approximately 3 minutes while the CPLD writes its fresh registers.
     
    Step 4: Step Forward and Re-Verify Logs
    1. Once the system stabilizes under BIOS 2.25.0 / CPLD Update / iDRAC 6.x, pull a fresh TSR and check dbglog/logs/idraclogs.
    2. Confirm that the thParser lines now read out successful initialization of the energy_balance_table and that the I2CTopology.bin parsing error has completely vanished.
    3. Only after confirming a clean log parse should you attempt to cleanly upgrade back to BIOS 2.27.0 and iDRAC 7.00.00.183.
    If you proceed with this structural re-flash matrix, let me know:
    • Did forcing the CPLD firmware update clear the I2CTopology.bin status=2 error?
    • Does the energy_balance_table now log positive byte sizing allocations upon system initialization?

    with kind regards Martin/ Liebe Grüße Martin

     

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

     

    DELL-Martin S

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell


  • lilpandemio

    1 Rookie

    •

    6 Publicaciones

    •

    19 Puntos

    0

    0

    Publicado July 9th, 2026 08:36

    @DellMartinS​ Will do this this afternoon. BTW a side question for the future. Is it possible to use non certified GPUs on the R740XD without fans going crazy?

    (editado)

  • DellMartinS

    Moderator

    •

    3883 Publicaciones

    •

    25810 Puntos

    0

    0

    Publicado July 9th, 2026 09:23

    Yes, it is entirely possible to run non-certified or consumer GPUs (such as NVIDIA GeForce RTX or workstation cards) in a PowerEdge R740xd without the fans running at maximum speed, but difficult and not really supported.
     
    By default, inserting an uncertified PCIe card triggers a safety mechanism called the Third-Party PCIe Card Cooling Response, which sets a high baseline fan floor (~60–100%) because iDRAC cannot read the card's internal thermal sensors. You can easily disable this behavior using raw RACADM commands.
     
     


    Disable the third-party fan response:

    racadm set system.thermalsettings.ThirdPartyPCICardInflowVentMitigation Disabled

     
    Check the current status:
     
    racadm get system.thermalsettings.ThirdPartyPCICardInflowVentMitigation

    with kind regards Martin/ Liebe Grüße Martin

     

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

     

    DELL-Martin S

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell


  • lilpandemio

    1 Rookie

    •

    6 Publicaciones

    •

    19 Puntos

    0

    0

    Publicado July 15th, 2026 15:55

    @DellMartinS​ CPLD updated from 1.0.1 to 1.1.4 (latest published for R740/R740XD; note there is no "1.0.8" release). Result: no change — #MFSMinimumLimit still 82-91 with the OS up.
    More significantly, comparing the thermal daemon's boot parse before and after (TSR debug logs from July 8 vs July 15): bit-for-bit identical. Same 252 energy_balance_table.N:eb_table_lfm_pci_slot* empty-value validation failures, same 3 I2CTopology.bin parsing error status=2 lines (thPlatform/thCPU/thDIMM init), same global_abl_fan_speed_min_pwr_snd_enabled / max_perf_enabled "string value too large" errors. Additionally, the protected ThermalTable.txt (1,130,240 bytes) and pbt_conf.txt (3,835,126 bytes) are byte-identical in size across both collections — static content unaffected by iDRAC, HBA, NIC or CPLD changes.
    The elimination matrix is now complete: these parse errors and the raised floor are invariant across iDRAC 7.00.00.18x/6.10.80.00, CPLD 1.0.1/1.1.4, rNDC 23.0.8/20.5.13, HBA330 16.17.01.00/16.17.00.05, and a fully purged LC database. The only July-4 change still in place is BIOS 2.25.0 → 2.27.0 — which is also the component that generates the platform thermal tables the daemon is failing to parse. Proceeding with the BIOS 2.25.0 downgrade as the final isolation test. Also — the racadm attribute suggested for the GPU question (ThirdPartyPCICardInflowVentMitigation) doesn't appear to exist in any iDRAC attribute registry; the documented one is ThirdPartyPCIFanResponse. Could you confirm whether these thParser validation failures appear on healthy R740xd systems, and whether the thermaldata archive can be decrypted through my case?
    DM Me if you want and I send you full TSR zip...

    (editado)