Olainen

updated

5 years ago

O

Olainen

1 Rookie

•

37 Posts

2

22123

January 21st, 2019 03:00

MX9116n FCoE-FC configuration?

We just got the new MX infrastructure, MX7000 chassis, MX740c servers, and MX9116n Fabric Switch Engines. While we will be switching to ethernet based storage later, we still have FC and thus, our plan is to use FCoE on the (esxi-)servers and use forward it as normal FC traffic to our Brocade FC switches on switch port 44 (configured for FC mode 16-4x)

Problem - I know little about FCoE configuration. I found this, but it didn't really help me too much: https://www.dell.com/support/home/us/en/04/product-support/product/networking-mx9116n/manuals

Do I need to venture into vFrabric configuration on the MX9116n? what FC feature mode, FPORT? NPG? Do I use FIP? Do I assign the vlan set for FCoE onto the CNA card, or use FIP alone?

Have anyone configured the following "path" and can share some config examples?

MX740c-with-CNA_FCoE <-> MX9116n <-> FC breakout cable 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage

I'm using the switches (two MX9116n) in full-switch mode btw.

  • Olainen

    1 Rookie

    •

    37 Posts

    17666

    2

    Posted January 28th, 2019 05:00

    With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers :BigSmile:

    Environment:

    • MX7000 chassis with 2 x MX9116n Fabric Switch Engines ("FSE")
    • MX740c Servers with QL41262 CNA cards (4 servers so far in my chasses, all 8 will get CNA soon)
    • Brocade FC fabrics wiht HUS-VM SAN behind

    So the FCoE/FC traffic will be like this:

    MX740c w/CNA_FCoE (ports 1-16) <-> MX9116n <-> FC breakout 4x16Gb (port 44) <-> Brocade FC-SW <-> FC-Storage

    For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
    For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.

    My configuration for FCoE is so far like this:

    FSE-A1:

    interface vlan4092
     description "FCoE-01-internal-vlan"
     no shutdown
     exit
    !
    dcbx enable
    feature fc npg
    vfabric 1
     name vFabric-1
     vlan 4092
     fcoe fcmap 0efc01
     exit

    FSE-A2:

    interface vlan4093
     description "FCoE-02-internal-vlan"
     no shutdown
     exit
    !
    dcbx enable
    feature fc npg
    vfabric 2
     name vFabric-2
     vlan 4093
     fcoe fcmap 0efc02
     exit

    FSE (both switches):

    !# Break-out ethernet
    port-group 1/1/13
     mode Eth 10g-4x
    !
    !# Break-out fibrechannel
    port-group 1/1/16
     mode FC 16g-4x
    
    class-map type network-qos class-Dot1p-3-FCoE
     match qos-group 3
    class-map type queuing map-ETSQueue-0
     match queue 0
    class-map type queuing map-ETSQueue-3
     match queue 3
    exit
    !
    trust dot1p-map map-Dot1pToGroups
     qos-group 0 dot1p 0-2,4-7
     qos-group 3 dot1p 3
     exit
    !
    qos-map traffic-class map-GroupsToQueues
     queue 0 qos-group 0
     queue 3 qos-group 3
     exit
    !
    policy-map policy-Input-TrustDot1p
     class class-trust
     trust dot1p
     exit
     exit
    !
    policy-map type network-qos policy-Input-PFC
     class class-Dot1p-3-FCoE
     pause
     pfc-cos 3
     exit
     exit
    !
    policy-map type queuing policy-Output-BandwidthPercent
     class map-ETSQueue-0
     bandwidth percent 68
     class map-ETSQueue-3
     bandwidth percent 32
     exit
     exit
    !
    system qos
     ets mode on
     trust-map dot1p map-Dot1pToGroups 
     qos-map traffic-class map-GroupsToQueues
     exit

    FSE-A1:

    interface range ethernet 1/1/1-1/1/16
    service-policy input type qos policy-Input-TrustDot1p
    service-policy input type network-qos policy-Input-PFC
    service-policy output type queuing policy-Output-BandwidthPercent
    priority-flow-control mode on
    vfabric 1
    exit
    !
    interface range fibrechannel 1/1/44:1-1/1/44:2
    vfabric 1
    exit
    exit

    FSE-A2:

    interface range ethernet 1/1/1-1/1/16
     service-policy input type qos policy-Input-TrustDot1p
     service-policy input type network-qos policy-Input-PFC
     service-policy output type queuing policy-Output-BandwidthPercent
     priority-flow-control mode on
     vfabric 2
     exit
    !
    interface range fibrechannel 1/1/44:1-1/1/44:2
     vfabric 2
     exit

    You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:

    show fc switch
    show vfabric
    show npg devices brief
    show fcoe enode
    show fcoe fcf
    show fcoe sessions  
    show fc statistics interface fibrechannel 1/1/44:1
    show fc statistics interface fibrechannel 1/1/44:2
    show fc statistics vfabric 1

    Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):

    OS10# show fcoe sessions
    Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE MAC FC-ID PORT WWPN PORT WWNN
    ---------------------------------------------------------------------------------------------------------------------------------------------------------
    66:66:66:15:fe:7f Eth 1/1/15 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:02 05:8d:02 66:66:66:66:66:15:fe:7f 66:66:66:66:66:15:fe:7f
    66:66:66:15:ff:e7 Eth 1/1/13 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:01 05:8d:01 66:66:66:66:66:15:ff:e7 66:66:66:66:66:15:ff:e7
    66:66:66:15:ff:0f Eth 1/1/11 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:02 05:5c:02 66:66:66:66:66:15:ff:0f 66:66:66:66:66:15:ff:0f
    66:66:66:0c:25:a7 Eth 1/1/1 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:01 05:5c:01 66:66:66:66:66:0c:25:a7 66:66:66:66:66:0c:25:a7

    Hopefully this can help others get FCoE going on MX-infrastructure..

  • DELL-Josh Cr

    Community Manager

    •

    9790 Posts

    •

    45217 Points

    17822

    0

    Posted January 21st, 2019 10:00

    Hi,

    Page 18 of the config guide may help with providing some of the steps, they are using smartfabrics though. https://downloads.dell.com/manuals/common/mx7000_fc_storage_networks_using_ethernet_ioms.pdf

  • Joffer77

    7 Posts

    17820

    0

    Posted January 21st, 2019 10:00

    Yes I found that. But I'm ot using smart fabric.. My uplinks at the moment is break out cables also for 4x10G ethernet (port 41) and 4x16G for FC (port 44). I will have a new look at it tomorrow anyway.

    (This is @Olainen posting, found out i had two accounts. Will delete this account)

  • Olainen

    1 Rookie

    •

    37 Posts

    17776

    0

    Posted January 23rd, 2019 02:00

    So there are no documentation (with examples) on setting up FCoE on MX9116n in full-switch mode?

    What I've deducted so far is that the "fc domain-id" mode is not the mode I want, as I do want to do all zoning on my main Brocade switches (and I'm not sure if dell/mx <-> brocade is supported in "full fc switch mode" (aka F_PORT mode if I'm not mistaken?).

    So it seems I need to use either "fip-snooping" or "fc mode npg"? 

  • DELL-Josh Cr

    Community Manager

    •

    9790 Posts

    •

    45217 Points

    17768

    0

    Posted January 23rd, 2019 09:00

    There is not any documentation yet, because it is a new product. FIP snooping should work for what you need.

  • DELL-Josh Cr

    Community Manager

    •

    9790 Posts

    •

    45217 Points

    17653

    0

    Posted January 29th, 2019 09:00

    Great information, thanks for sharing.

  • Joffer77

    7 Posts

    17683

    0

    Posted January 29th, 2019 13:00

    Happy to help. I see all my three attempts to post got revived.. I'll see if I can clean it up.. The config is the same.. 

    (Ugh. found out I had two accounts. This is @Olainen posting. Will delete this account, and keep my olainen account)

  • Olainen

    1 Rookie

    •

    37 Posts

    17652

    0

    Posted January 30th, 2019 01:00

    This explains a lot.. I couldn't change some of my posts.. I now see I have two account on here.. ugh.. Joffer77 and Olainen.. 

    Is there a way to consolidate them? 

  • DELL-Josh Cr

    Community Manager

    •

    9790 Posts

    •

    45217 Points

    10592

    0

    Posted January 30th, 2019 08:00

    I was able to remove the duplicate posts, but not combine your accounts.

  • Olainen

    1 Rookie

    •

    37 Posts

    4214

    0

    Posted February 5th, 2019 01:00

    Ok. I'll just stop using the other account :)