We just got the new MX infrastructure, MX7000 chassis, MX740c servers, and MX9116n Fabric Switch Engines. While we will be switching to ethernet based storage later, we still have FC and thus, our plan is to use FCoE on the (esxi-)servers and use forward it as normal FC traffic to our Brocade FC switches on switch port 44 (configured for FC mode 16-4x)
Do I need to venture into vFrabric configuration on the MX9116n? what FC feature mode, FPORT? NPG? Do I use FIP? Do I assign the vlan set for FCoE onto the CNA card, or use FIP alone?
Have anyone configured the following "path" and can share some config examples?
With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers :BigSmile:
Environment:
MX7000 chassis with 2 x MX9116n Fabric Switch Engines ("FSE")
MX740c Servers with QL41262 CNA cards (4 servers so far in my chasses, all 8 will get CNA soon)
For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet. For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.
My configuration for FCoE is so far like this:
FSE-A1:
interface vlan4092
description "FCoE-01-internal-vlan"
no shutdown
exit
!
dcbx enable
feature fc npg
vfabric 1
name vFabric-1
vlan 4092
fcoe fcmap 0efc01
exit
FSE-A2:
interface vlan4093
description "FCoE-02-internal-vlan"
no shutdown
exit
!
dcbx enable
feature fc npg
vfabric 2
name vFabric-2
vlan 4093
fcoe fcmap 0efc02
exit
FSE (both switches):
!# Break-out ethernet
port-group 1/1/13
mode Eth 10g-4x
!
!# Break-out fibrechannel
port-group 1/1/16
mode FC 16g-4x
class-map type network-qos class-Dot1p-3-FCoE
match qos-group 3
class-map type queuing map-ETSQueue-0
match queue 0
class-map type queuing map-ETSQueue-3
match queue 3
exit
!
trust dot1p-map map-Dot1pToGroups
qos-group 0 dot1p 0-2,4-7
qos-group 3 dot1p 3
exit
!
qos-map traffic-class map-GroupsToQueues
queue 0 qos-group 0
queue 3 qos-group 3
exit
!
policy-map policy-Input-TrustDot1p
class class-trust
trust dot1p
exit
exit
!
policy-map type network-qos policy-Input-PFC
class class-Dot1p-3-FCoE
pause
pfc-cos 3
exit
exit
!
policy-map type queuing policy-Output-BandwidthPercent
class map-ETSQueue-0
bandwidth percent 68
class map-ETSQueue-3
bandwidth percent 32
exit
exit
!
system qos
ets mode on
trust-map dot1p map-Dot1pToGroups
qos-map traffic-class map-GroupsToQueues
exit
FSE-A1:
interface range ethernet 1/1/1-1/1/16
service-policy input type qos policy-Input-TrustDot1p
service-policy input type network-qos policy-Input-PFC
service-policy output type queuing policy-Output-BandwidthPercent
priority-flow-control mode on
vfabric 1
exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
vfabric 1
exit
exit
FSE-A2:
interface range ethernet 1/1/1-1/1/16
service-policy input type qos policy-Input-TrustDot1p
service-policy input type network-qos policy-Input-PFC
service-policy output type queuing policy-Output-BandwidthPercent
priority-flow-control mode on
vfabric 2
exit
!
interface range fibrechannel 1/1/44:1-1/1/44:2
vfabric 2
exit
You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:
show fc switch
show vfabric
show npg devices brief
show fcoe enode
show fcoe fcf
show fcoe sessions
show fc statistics interface fibrechannel 1/1/44:1
show fc statistics interface fibrechannel 1/1/44:2
show fc statistics vfabric 1
Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):
OS10# show fcoe sessions Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE MAC FC-ID PORT WWPN PORT WWNN --------------------------------------------------------------------------------------------------------------------------------------------------------- 66:66:66:15:fe:7f Eth 1/1/15 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:02 05:8d:02 66:66:66:66:66:15:fe:7f 66:66:66:66:66:15:fe:7f 66:66:66:15:ff:e7 Eth 1/1/13 66:66:66:1f:f6:7d Fc 1/1/44:1 4093 66:66:66:05:8d:01 05:8d:01 66:66:66:66:66:15:ff:e7 66:66:66:66:66:15:ff:e7 66:66:66:15:ff:0f Eth 1/1/11 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:02 05:5c:02 66:66:66:66:66:15:ff:0f 66:66:66:66:66:15:ff:0f 66:66:66:0c:25:a7 Eth 1/1/1 66:66:66:1f:f6:7e Fc 1/1/44:2 4093 66:66:66:05:5c:01 05:5c:01 66:66:66:66:66:0c:25:a7 66:66:66:66:66:0c:25:a7
Hopefully this can help others get FCoE going on MX-infrastructure..
Yes I found that. But I'm ot using smart fabric.. My uplinks at the moment is break out cables also for 4x10G ethernet (port 41) and 4x16G for FC (port 44). I will have a new look at it tomorrow anyway.
(This is @Olainen posting, found out i had two accounts. Will delete this account)
So there are no documentation (with examples) on setting up FCoE on MX9116n in full-switch mode?
What I've deducted so far is that the "fc domain-id" mode is not the mode I want, as I do want to do all zoning on my main Brocade switches (and I'm not sure if dell/mx <-> brocade is supported in "full fc switch mode" (aka F_PORT mode if I'm not mistaken?).
So it seems I need to use either "fip-snooping" or "fc mode npg"?
Olainen
1 Rookie
•
37 Posts
17666
2
Posted January 28th, 2019 05:00
With some help from Dell ProSupport I've pieced together the following configuration for my environment, and I've successfully established connection with my LUN on my FC SAN and created a VMFS6 datastore for my ESXi servers :BigSmile:
Environment:
So the FCoE/FC traffic will be like this:
For ethernet uplink, we use port 41 on each FSE using a breakout cable for 4x10GbE, where we so far only use two of them from each FSE. Using VLT, and a LACP trunk to our core, we have 4x10GbE total uplink for ethernet.
For FC uplink, using port 44, we also have a break out cable, set to 4x16G, and here too we only use 2 of them from each FSE. Best practice dictates we use one uniqe vlan ID in each FSE so we can be sure we won't have FCoE crosstalk over the VLT interconnect, having each FSE as a separate fabric (vfabric) for FC/FCoE. Each FC uplink connects only to one FC brocade fabric.
My configuration for FCoE is so far like this:
FSE-A1:
FSE-A2:
FSE (both switches):
FSE-A1:
FSE-A2:
You should now (hopefully) be able to get some data in the different 'show' commands for FCoE/FC:
Example (i've masked the MACs/WWNs) - shows my 4 servers with CNA card (FCoE enabled with NPAR-configuration):
Hopefully this can help others get FCoE going on MX-infrastructure..