UNSOLVED

Blackie1337

updated

2 years ago

B

Blackie1337

1 Rookie

•

12 Posts

•

2 Points

0

770

October 21st, 2024 14:59

r740xd with 24x nvme backplane troubles.

I have a server that was shipping damaged, Riser2 had the connectors broken. I have replaced the riser and it seems to be working fine. I am however having huge problems getting all the nvme slots functioning. 

First of all, I am getting this error on bootup: HWC2003 the storage BP2 power cable is not connected.

Now, the server has no midplane, which as far as I can google is where the BP2 power connector is supposed to go, (I dont have one) so I am looking for an explanation for this error.

And no matter what I try I can not get the 2? backplanes in the front detected, Backplane 1 is listed as having 12 slots, so I am guessing there should be a Backplane 2 with the second half of the slots?

The extender connected to the second half of the backplane is also not recognized properly, it is listed in the inventory, but it does not show up under Storage in iDRAC. Only the one connected to the first half of the backplane show up here. 

I have tried switching places of the extenders, but same result. Tried several different 16x pcie slots and the same symptom follows. 

At the moment I am leaning towards one of the switch chips on the backplane it defective. But I haven't been able to find a definitive proof that it is dead. Mostly because of the following:

Some of the nvme drives in the second half of the slots are detected in TrueNAS, but NOT in idrac storage. And some of them shows up in inventory as nvme drive connected to Backplane 17 or 19 or some such nonsense.

Also, the drives connected to the "working" part of the backplane are listed as "Ready" in idrac storage, (not online), and they are NOT detected in TrueNAS. 

I'm highly confused about this so I'm trying to reach out to someone with more experience with these servers.

  • DELL-Charles R

    Moderator

    •

    4730 Posts

    •

    25476 Points

    0

    0

    Posted October 21st, 2024 19:37

    Hello,

     

    I'm sorry to see that. Fixing a system that was damaged in shipping can be trying.

    Does the replacement Riser2 have the same part number as the original Riser 2 in the system?

     

    Check these places for BP2 cable and see if they are the ones you are looking for.

     

    Page 182 connection #19

    Backplane 2 signal connector

    #19 J_BP_SIG2 Backplane 2 signal connector

       Check #6 on page 155. Figure 153. for the Cable routing – 24 x 2.5 inch NVMe drive backplane

    https://dell.to/3UhXd7w

     

     

    Page 182 connection #22

    BP2 appears to be Riser 2 to system board

    #22 J_BP2 (RSR2_225W) Backplane 2 power connector (Riser 2 PCIe 225 W power)

    https://dell.to/3UhXd7w

     

    Along with damaged connectors did you also verify any cable damage?

     

     

    For the hard drives; did you have a Foreign configuration on the PERC controller that you could import?

    Page 265 Importing foreign configuration using iDRAC web interface

    https://dell.to/3UhXd7w

     

    You may need to get the backplane issue resolved to see all the drives.

    DELL-Charles R

    Social Media and Communities Professional
    Dell Technologies | Enterprise Support Services
    #IWork4Dell

    Did I answer your query? Please click on ‘Accept as Solution’. ‘Thumbs up’ the posts you like!

  • Blackie1337

    1 Rookie

    •

    12 Posts

    •

    2 Points

    0

    0

    Posted October 22nd, 2024 14:26

    @DELL-Charles R​ 

    Does the replacement Riser2 have the same part number as the original Riser 2 in the system?

     

    Yes I verified that it has the exact same part number.

    Page 182 connection #19

    Backplane 2 signal connector

    #19 J_BP_SIG2 Backplane 2 signal connector

       Check #6 on page 155. Figure 153. for the Cable routing – 24 x 2.5 inch NVMe drive backplane

    https://dell.to/3UhXd7w

     

    This I have checked and verified continuity with a multimeter. And also the server will not power on with either of the signal wires disconnected or either of the power cables disconnected.

     

    Page 182 connection #22

    BP2 appears to be Riser 2 to system board

    #22 J_BP2 (RSR2_225W) Backplane 2 power connector (Riser 2 PCIe 225 W power)

    https://dell.to/3UhXd7w

     

    I read that same thing, but its very unspecific.. Does this mean that the BP2 power socket is powered by the same traces that powers the Riser2? (I'm guessing "system board" is their version of motherboard)

    Along with damaged connectors did you also verify any cable damage?

     

    No cable damage as far as I can tell, I have not tried to test continuity of all the pcie extenders though. 

    For the hard drives; did you have a Foreign configuration on the PERC controller that you could import?

    Page 265 Importing foreign configuration using iDRAC web interface

    https://dell.to/3UhXd7w

    I have no PERC controller at all, just pcie extenders. But also, I have no prior configs.

    I have screenshots from the seller of the server identifying drives 1-24 and both pcie extenders in iDRAC, now it only shows the extender connected to the first half of the front backplane.

    This remains the case if I switch the extenders and the pcie slots. Which leads me towards the second half of the front backplane having an issue. 

    HOWEVER. I did make a discovery last night, there were a minor break in the rear most connector for Riser 2 on the motherboard. We're taking so small that I had to have a magnifying glass to notice, and that means that there might have been 2-3 pins that missed their intended connections on Riser 2. 

    This shouldn't really matter since I have no extenders plugged into Riser 2 atm. Unless, this riser gets power from the same place as the BP2 socket on the motherboard. Which MIGHT have made a short or something that makes the motherboard think that there is a middle backplane connected, which MAYBE? disables the front second half of the nvme backplane?

    This I hope you can cast some light on. Does the second half of the front nvme backplane gets disabled (downgraded to sas?) if the mid backplane is detected?

    (edited)

  • DELL-Charles R

    Moderator

    •

    4730 Posts

    •

    25476 Points

    0

    0

    Posted October 22nd, 2024 15:08

    Hello,

     

    If you didn't have BP2 to Riser 2 originally I wouldn't think you need it ,but, the damage to the socket and especially the system board (mother board) could be causing issue in the multilayer PCB I can't say what all could be damaged.

     

    I don't know why the signal wires disconnected would cause the system not to power on.

     

    You could try bringing the system down to minimum-to-POST configuration for troubleshooting.

     

    Minimum to post (this is minimum components to POST. If you get successful POST, put things back a little at a time until find faulting component)

     

    The minimum components to allow the PowerEdge R740xd to complete POST are:

     

    * One processor (CPU) in socket processor 1

    * One memory module (DIMM) in socket A1

    * One power supply unit

    * Right control panel and cable (for power button functionality)

    * System board

     

     

    Remove anything not on that list: DVD, Hard drives, PERC controller, backplane, network card, NIC cable, any pcie cards, keyboard, mouse, USB devices, …. anything not on the list remove.

     

    If it does not post then the issue is with one of those minimum to post components.

    If you get successful POST, put things back a little at a time until you find the faulting component.

     

     

    You may see if you can get replacement under the shipping company insurance.

    DELL-Charles R

    Social Media and Communities Professional
    Dell Technologies | Enterprise Support Services
    #IWork4Dell

    Did I answer your query? Please click on ‘Accept as Solution’. ‘Thumbs up’ the posts you like!

  • Blackie1337

    1 Rookie

    •

    12 Posts

    •

    2 Points

    0

    0

    Posted October 22nd, 2024 16:06

    @DELL-Charles R​ 

    I've already been down the minimum viable product route a few times.

    There are no cable from the motherboard to Riser 2 (or any of the risers).

    You need both cpus plugged in to access both backplanes since they are (need to be because of length limitations of the extender cables) connected to different cpus. I dont really have any problems posting, except for that nagging BP2 power connector fault.

    The problem is that there are differing info in the different dell documentation as to what BP2 is.. Is it the second half of the front backplane? Or is it the mid backplane? Or certain places in Lifecycle I believe I found them listed as Backplane 0, 1 and 2. But the motherboard power connectors for the front backplane are labeled BP1 and BP3.

    So is this BP2 error even related to the front backplane at all? Or is it simply a red herring.

    But again, can you tell me how the behavior of the server would be if I connected a mid backplane to the BP2 power and a PERC? Would it change the front backplane from 24x nvme to 12x nvme + something else (SAS?)

    ex: https://www.reddit.com/r/homelab/comments/15rls28/r740xd_with_24_nvme_drives_working_dell_parts_list/

    This guy had some of the same errors when building his server to r740xd specs. and that is exactly the wiring I have in mine (minus the Perc and SAS cables).

    Notice that Figure 16 from the manual lists: 

    24 x 2.5 inch drive backplane with up to 24 NVMe or up to 8 SAS/SATA drives with NVMe

    Could you tell me what that means? There are only 24 slots for drives, how is the SAS/SATA drives supposed to be incorporated? Could it be that only a few slots of the second half of the backplane is nvme compatible if the motherboards thinks that there is a PERC / midplane connected?

  • DELL-Charles R

    Moderator

    •

    4730 Posts

    •

    25476 Points

    0

    0

    Posted October 22nd, 2024 18:36

    Hello,

     

     

    Has anything changed in the original configuration, particularly in the backplane configuration?

     

     

    I ask that because we don't support field changing of backplane configuration as there are no kits for that.

     

    Looks like #21 and #22 are power for GPU:

    #21 - J_BP0 (RSR3_225W) Backplane 0 power connector (Riser 3 PCIe 225 W power)

    #22 - J_BP2 (RSR2_225W) Backplane 2 power connector (Riser 2 PCIe 225 W power)

    The riser power connecter provides additional power up to 225 W to support PCIe cards that are rated above 75 W - 300 W

     

     

    From page 145 can you tell me which cable configuration your system has?

    And which backplane you have from page 136?

    https://dell.to/3UhXd7w

     

    DELL-Charles R

    Social Media and Communities Professional
    Dell Technologies | Enterprise Support Services
    #IWork4Dell

    Did I answer your query? Please click on ‘Accept as Solution’. ‘Thumbs up’ the posts you like!

  • Blackie1337

    1 Rookie

    •

    12 Posts

    •

    2 Points

    0

    0

    Posted October 23rd, 2024 06:13

    @DELL-Charles R​ 

    Nothing has been changed, its an original 24x nvme server.

    Looks like #21 and #22 are power for GPU:

    #21 - J_BP0 (RSR3_225W) Backplane 0 power connector (Riser 3 PCIe 225 W power)

    #22 - J_BP2 (RSR2_225W) Backplane 2 power connector (Riser 2 PCIe 225 W power)

    The riser power connecter provides additional power up to 225 W to support PCIe cards that are rated above 75 W - 300 W

    That was my suspicion as well, but then why is the server whining about not having this connector "connected"? I do not have any cards at all plugged into Riser 2. (ATM).

    Can I surmise that "Backplane 0" is for the optional rear backplane, and that Backplane 2 is the mid backplane? So the front would be 1 and 3? 

    From page 145 can you tell me which cable configuration your system has?

    And which backplane you have from page 136?

    I have the cables from page 155 Figure 153.

    And I have the backplane from page 136 Figure 125.

    (and notice that it has a SAS connector, hence why I'm asking what the behavior would be if this is plugged in, it is not well explained in the manual as far as I can tell.)

  • DELL-Erman O

    Moderator

    •

    2984 Posts

    •

    14874 Points

    0

    0

    Posted October 23rd, 2024 07:20

    Hi,

    According to the manual, the riser power connectors (#21 and #22) are there to give extra power to PCIe cards that need more than 75W. Since you don’t have any cards in Riser 2, the problem might be with the backplane setup. about the backplane numbers, you’re right! They usually go like this:

    front backplane (closest to the front of the server), mid backplane (Backplane 2, as you mentioned), rear backplane (closest to the back of the server)

    So, if you’re not using the mid backplane, the server might be complaining because it expects it to be connected. It’s hard to say for sure a definitive answer for me. I'm just brainstorming. Let's see what Charles thinking.

    DELL-Erman O

    Social Media and Communities Professional

    Dell Technologies | Enterprise Support Services

    #IWork4Dell

    Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!

  • DELL-Charles R

    Moderator

    •

    4730 Posts

    •

    25476 Points

    0

    0

    Posted October 23rd, 2024 20:38

    Hello,

     

    Try this:   check the backplane mode through the iDRAC GUI: iDRAC > Storage > Overview > Enclosures

     

    See what difference it makes in Split mode and Unified mode

     

    After setting it, you will need to save, exit and shut down ( Power Cycle)  

    A Restart won’t be enough for the server to apply the change.

     

    Configuring backplane mode using web interface

    https://dell.to/40hITQ3

     

    Note an additional reboot of the DRAC and Host power cycle is needed:

    "To avoid inventory issues, in case of any back plane cable connection changes you require additional iDRAC reboot and Host power cycle"

    DELL-Charles R

    Social Media and Communities Professional
    Dell Technologies | Enterprise Support Services
    #IWork4Dell

    Did I answer your query? Please click on ‘Accept as Solution’. ‘Thumbs up’ the posts you like!

  • Blackie1337

    1 Rookie

    •

    12 Posts

    •

    2 Points

    0

    0

    Posted October 24th, 2024 15:12

    @DELL-Erman O​ 

    Thanks for chipping in, but that didn't really answer what I am wondering. Fair enough, the mid plane might be listed as BP2, but what is the rear plane then? (0 or 3)? And are the 2 fronts 0-1 or 1 and 3?

  • Blackie1337

    1 Rookie

    •

    12 Posts

    •

    2 Points

    0

    0

    Posted October 24th, 2024 15:15

    @DELL-Charles R​ 

    Aw, I almost got my hopes up that there was a setting I had missed. But sadly, that option is not there. and I have this info listed in advanced properties:

    Backplane 1 of PCIe Extender in PCIe Slot 8

    ......

    Enclosure Split Mode Capability Not Capable

    And if you're curious here are the full list of properties.

    Advanced Properties
    Device Description Backplane 1 of PCIe Extender in PCIe Slot 8
    Connector 0
    Enclosure Position Not Applicable
    Enclosure Location Right
    Bay ID 1
    Firmware Version 4.35
    SAS Address Not Applicable
    Enclosure Split Mode Capability Not Capable
    PCI Express Generation Gen 3

    Again, the other extender/backplane does not show up :(

    (edited)