This post is part of my CCIE Datacenter reference series and will cover all there is to get FCoE up and running on the Cisco Nexus 7k, 5k and 2k switches.
The text assumes you have a understanding knowledge of fundamentals (storage paths, flogis, vPC, etc) and can be used as a configuration reference (I basically wrote this for myself).
Contents
- 1 Installing & Enabling FCoE
- 2 Basic setup
- 3 Connecting a host – Single wire
- 4 Connecting a host – vPC wire
- 5 Connecting a host – FEX Single Homed – Single wire
- 6 Connecting a host – FEX Single Homed – Dual wire
- 7 Connecting a host – FEX Single Homed – vPC wire
- 8 Connecting a host – FEX Dual Homed – vPC wire
- 9 Connecting a switch – Single wire
- 10 Connecting a switch – Port-channel
- 11 Connecting a switch – FCoE-NPV
- 12 Connecting UCS – Port-channel
Installing & Enabling FCoE
Nexus 7K
Installing and enabling FCoE on the Nexus 7k involves getting the storage VDC up and running. This section will cover just that.
QoS
The Nexus 7k comes with a default QoS policy of default-nq-8e-policy, which does not have a no drop CoS. FCoE needs at least one, so change the network-qos policy to one that has a no drop CoS. default-nq-7e-policy is usually good enough. If you forget to change the policy, the creation of the storage VDC will not work.
7K-DEFAULT(config)# system qos 7K-DEFAULT(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
Feature-set and license
7K-DEFAULT(config)# install feature-set fcoe 7K-DEFAULT(config)# license fcoe module 1 7K-DEFAULT(config)# feature lldp
Creating the VDC
We’re going to dedicate VLANs 1000 to 1004 to FCoE.
7K-DEFAULT(config)# vdc 7K-FCOE type storage 7K-DEFAULT(config-vdc)# allow feature-set fcoe 7K-DEFAULT(config-vdc)# allocate fcoe-vlan-range 1000-1004 from vdcs DEFAULT
Dedicated interfaces
The storage VDC can have two types of interfaces; dedicated and shared. Dedicated interfaces are FCoE only and are not used for host connectivity. Usually used for FCoE-only storage controllers and host CNAs dedicated to FCoE or FCoE only trunks to distribution switches (5Ks).
7K-DEFAULT(config-vdc)# allocate interface Ethernet1/1-2
Shared interfaces are typically used to connect hosts to. The interface is shared between 2 VDCs, one being the storage VDC and the other being a data VDC. The storage VDC will handle the FCoE traffic and the data VDC will handle all the rest of the traffic. Prerequisite of shared interfaces is that they are in trunk mode and spanning-tree allows it to come up quickly.
7K-DEFAULT(config)# interface Ethernet1/3-4 7K-DEFAULT(config-if)# switchport mode trunk 7K-DEFAULT(config-if)# spanning-tree port type edge trunk 7K-DEFAULT(config)# vdc 7K-FCOE 7K-DEFAULT(config-vdc)# allocate shared interface Ethernet1/3-4
Nexus 5K
Enabling FCoE on the Nexus 5k is a bit easier. Again, mind the QoS. The Nexus 5000 has no drop enabled by default (you can’t even take it off), but the Nexus 5500 does not.
5K-SW(config)# policy-map type network-qos FCOE 5K-SW(config-pmap-nq)# class type network-qos class-fcoe 5K-SW(config-pmap-nq-c)# pause no-drop 5K-SW(config-pmap-nq-c)# mtu 2158 5K-SW(config-pmap-nq)# class type network-qos class-default 5K-SW(config-pmap-nq-c)# mtu 9216 (don't forget this one if you have a need for jumbo frames on the 5Ks) 5K-SW(config)# system qos 5K-SW(config-sys-qos)# service-policy type network-qos FCOE 5K-SW(config)# feature fcoe
Basic setup
With basic setup, we create a VSAN, link that to a VLAN and do some basic settings like zoning.
FCOE(config)# vsan database FCOE(config-vsan-db)# vsan 1000 FCOE(config)# vlan 1000 FCOE(config-vlan)# fcoe vsan 1000 FCOE(config)# fcdomain domain 10 preferred vsan 1000 (make domain IDs predictable) FCOE(config)# fcdomain priority 10 vsan 1000 (make switch domain root) FCOE(config)# fcdomain restart disruptive vsan 1000 (apply changes) FCOE(config)# zone default-zone permit vsan 1000 (yes, this is generally bad. In production use proper zoning)
Connecting a host – Single wire
A simple host connecting into a VSAN with a single connection for data and storage.
5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk native vlan 10 5K-SW(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW(config-if)# spanning-tree port type edge trunk 5K-SW(config)# int vfc1 5K-SW(config-if)# bind interface Ethernet1/1 5K-SW(config)# vsan database 5K-SW(config-vsan-db)# vsan 1000 interface vfc1
Connecting a host – vPC wire
vPC is an excellent way to connect hosts to two Nexus switches to create a redundant setup. We’ll use the vPC for normal data traffic and bind the vfc interfaces to the physical interfaces.
Nexus 5K-1 5K-SW1(config)# interface Ethernet1/1 5K-SW1(config-if)# channel-group 1 mode active 5K-SW1(config)# interface port-channel1 5K-SW1(config-if)# switchport mode trunk 5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW1(config-if)# switchport trunk native vlan 10 5K-SW1(config-if)# spanning-tree port type edge trunk 5K-SW1(config-if)# vpc 1 5K-SW1(config)# interface vfc1 5K-SW1(config-if)# bind interface Ethernet1/1 5K-SW1(config)# vsan database 5K-SW1(config-vsan-db)# vsan 1000 interface vfc1 Nexus 5K-2 5K-SW1(config)# interface Ethernet1/1 5K-SW2(config-if)# channel-group 1 mode active 5K-SW2(config)# interface port-channel1 5K-SW2(config-if)# switchport mode trunk 5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000 5K-SW2(config-if)# switchport trunk native vlan 10 5K-SW2(config-if)# spanning-tree port type edge trunk 5K-SW2(config-if)# vpc 1 5K-SW2(config)# interface vfc1 5K-SW2(config-if)# bind interface Ethernet1/1 5K-SW2(config)# vsan database 5K-SW2(config-vsan-db)# vsan 2000 interface vfc1
Connecting a host – FEX Single Homed – Single wire
5K-SW(config)# fex 100 5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# switchport mode fex-fabric 5K-SW(config-if)# fex associate 100 5K-SW(config)# interface Ethernet100/1/1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk native vlan 10 5K-SW(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW(config-if)# spanning-tree port type edge trunk 5K-SW(config)# int vfc1 5K-SW(config-if)# bind interface Ethernet100/1/1 5K-SW(config)# vsan database 5K-SW(config-vsan-db)# vsan 1000 interface vfc1
Connecting a host – FEX Single Homed – Dual wire
Nexus 5K-1 - Get the FEXes up and running first 5K-SW1(config)# fex 100 5K-SW1(config-fex)# fcoe 5K-SW1(config)# interface Ethernet1/1 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 100 5K-SW1(config)# interface Ethernet100/1/1 5K-SW1(config-if)# switchport mode trunk 5K-SW1(config-if)# switchport trunk native vlan 10 5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW1(config-if)# spanning-tree port type edge trunk 5K-SW1(config)# int vfc1 5K-SW1(config-if)# bind interface Ethernet100/1/1 5K-SW1(config)# vsan database 5K-SW1(config-vsan-db)# vsan 1000 interface vfc1 Nexus 5K-2 - Get the FEXes up and running first 5K-SW2(config)# fex 200 5K-SW2(config-fex)# fcoe 5K-SW2(config)# interface Ethernet1/2 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 200 5K-SW2(config)# interface Ethernet200/1/1 5K-SW2(config-if)# switchport mode trunk 5K-SW2(config-if)# switchport trunk native vlan 10 5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000 5K-SW2(config-if)# spanning-tree port type edge trunk 5K-SW2(config)# int vfc1 5K-SW2(config-if)# bind interface Ethernet200/1/1 5K-SW2(config)# vsan database 5K-SW2(config-vsan-db)# vsan 2000 interface vfc1
Connecting a host – FEX Single Homed – vPC wire
Nexus 5K-1 - Get the FEXes up and running first 5K-SW1(config)# fex 100 5K-SW1(config-fex)# fcoe 5K-SW1(config)# interface Ethernet1/1 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 100 5K-SW1(config)# interface Ethernet100/1/1 5K-SW1(config-if)# channel-group 101 5K-SW1(config)# interface port-channel 101 5K-SW1(config-if)# switchport mode trunk 5K-SW1(config-if)# switchport trunk native vlan 10 5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW1(config-if)# spanning-tree port type edge trunk 5K-SW1(config-if)# vpc 101 5K-SW1(config)# int vfc1 5K-SW1(config-if)# bind interface Ethernet100/1/1 5K-SW1(config)# vsan database 5K-SW1(config-vsan-db)# vsan 1000 interface vfc1 Nexus 5K-2 - Get the FEXes up and running first 5K-SW2(config)# fex 200 5K-SW2(config-fex)# fcoe 5K-SW2(config)# interface Ethernet1/2 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 200 5K-SW2(config)# interface Ethernet200/1/1 5K-SW2(config-if)# channel-group 201 5K-SW2(config)# interface port-channel 201 5K-SW2(config-if)# switchport mode trunk 5K-SW2(config-if)# switchport trunk native vlan 10 5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000 5K-SW2(config-if)# spanning-tree port type edge trunk 5K-SW2(config-if)# vpc 201 5K-SW2(config)# int vfc1 5K-SW2(config-if)# bind interface Ethernet200/1/1 5K-SW2(config)# vsan database 5K-SW2(config-vsan-db)# vsan 2000 interface vfc1
Connecting a host – FEX Dual Homed – vPC wire
Nexus 5K-1 - Get the FEXes up and running first 5K-SW1(config)# fex 100 5K-SW1(config-fex)# fcoe 5K-SW1(config)# fex 200 5K-SW1(config)# interface Ethernet1/1 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 100 5K-SW1(config-if)# channel-group 100 5K-SW1(config)# interface Ethernet1/2 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 200 5K-SW1(config-if)# channel-group 200 5K-SW1(config)# interface port-channel 100 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 100 5K-SW1(config-if)# vpc 100 5K-SW1(config)# interface port-channel 200 5K-SW1(config-if)# switchport mode fex-fabric 5K-SW1(config-if)# fex associate 200 5K-SW1(config-if)# vpc 200 - Create the host port-channels and mark one side for FCoE 5K-SW1(config)# interface Ethernet100/1/1 5K-SW1(config-if)# channel-group 101 5K-SW1(config)# interface port-channel 101 5K-SW1(config-if)# switchport mode trunk 5K-SW1(config-if)# switchport trunk native vlan 10 5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000 5K-SW1(config-if)# spanning-tree port type edge trunk 5K-SW1(config)# interface Ethernet200/1/1 5K-SW1(config-if)# channel-group 201 5K-SW1(config)# interface port-channel 201 5K-SW1(config-if)# switchport mode trunk 5K-SW1(config-if)# switchport trunk native vlan 10 5K-SW1(config-if)# switchport trunk allowed vlan 10 5K-SW1(config-if)# spanning-tree port type edge trunk - Create the virtual FC interface 5K-SW1(config)# int vfc1 5K-SW1(config-if)# bind interface Ethernet100/1/1 5K-SW1(config)# vsan database 5K-SW1(config-vsan-db)# vsan 1000 interface vfc1 Nexus 5K-2 - Get the FEXes up and running first 5K-SW2(config)# fex 100 5K-SW2(config)# fex 200 5K-SW2(config-fex)# fcoe 5K-SW2(config)# interface Ethernet1/1 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 100 5K-SW2(config-if)# channel-group 100 5K-SW2(config)# interface Ethernet1/2 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 200 5K-SW2(config-if)# channel-group 200 5K-SW2(config)# interface port-channel 100 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 100 5K-SW2(config-if)# vpc 100 5K-SW2(config)# interface port-channel 200 5K-SW2(config-if)# switchport mode fex-fabric 5K-SW2(config-if)# fex associate 200 5K-SW2(config-if)# vpc 200 - Create the host port-channels and mark one side for FCoE 5K-SW2(config)# interface Ethernet100/1/1 5K-SW2(config-if)# channel-group 101 5K-SW2(config)# interface port-channel 101 5K-SW2(config-if)# switchport mode trunk 5K-SW2(config-if)# switchport trunk native vlan 10 5K-SW2(config-if)# switchport trunk allowed vlan 10 5K-SW2(config-if)# spanning-tree port type edge trunk 5K-SW2(config)# interface Ethernet200/1/1 5K-SW2(config-if)# channel-group 201 5K-SW2(config)# interface port-channel 201 5K-SW2(config-if)# switchport mode trunk 5K-SW2(config-if)# switchport trunk native vlan 10 5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000 5K-SW2(config-if)# spanning-tree port type edge trunk - Create the virtual FC interface 5K-SW2(config)# int vfc1 5K-SW2(config-if)# bind interface Ethernet200/1/1 5K-SW2(config)# vsan database 5K-SW2(config-vsan-db)# vsan 2000 interface vfc1
Connecting a switch – Single wire
Switch to switch, allowing multiple VSAN and data traffic at the same time.
5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001 5K-SW(config)# int vfc1 5K-SW(config-if)# bind interface Ethernet1/1 5K-SW(config-if)# switchport mode E 5K-SW(config-if)# switchport trunk allowed vsan 1000-1001
Connecting a switch – Port-channel
Switch to switch with multiple wires forming a channel, allowing multiple VSAN and data traffic at the same time.
5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# channel-group 1 mode active 5K-SW(config)# interface port-channel 1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001 5K-SW(config)# int vfc1 5K-SW(config-if)# bind interface port-channel1 5K-SW(config-if)# switchport mode E 5K-SW(config-if)# switchport trunk allowed vsan 1000-1001
There’s an interface called a vfc-port-channel, which is only available on the Nexus 7k. I haven’t been able to find a significant difference between creating a vfc-port-channel and just binding a vfc interface to a port-channel, but it’s available if you want it.
5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# channel-group 1 mode active 5K-SW(config)# interface port-channel 1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001 5K-SW(config)# int vfc-port-channel 1 5K-SW(config-if)# switchport mode E 5K-SW(config-if)# switchport trunk allowed vsan 1000-1001
Connecting a switch – FCoE-NPV
7K-FCOE(config)# feature npiv 7K-FCOE(config)# interface Ethernet1/1 7K-FCOE(config-if)# switchport mode trunk 7K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000 7K-FCOE(config)# int vfc 1 7K-FCOE(config-if)# bind interface Ethernet1/1 7K-FCOE(config)# vsan database 7K-FCOE(config-vsan-db)# vsan 1000 interface vfc1 5K-SW(config)# feature fcoe-npv 5K-SW(config)# interface Ethernet1/1 5K-SW(config-if)# switchport mode trunk 5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000 5K-SW(config)# int vfc 1 5K-SW(config-if)# bind interface Ethernet1/1 5K-SW(config-if)# switchport mode NP 5K-SW(config)# vsan database 5K-SW(config-vsan-db)# vsan 1000 interface vfc1
Connecting UCS – Port-channel
5K-FCOE(config)# feature npiv 5K-FCOE(config)# feature fport-channel-trunk 5K-FCOE(config)# interface Ethernet1/1-2 5K-FCOE(config-if)# channel-group 12 mode active 5K-FCOE(config)# interface port-channel 12 5K-FCOE(config-if)# switchport mode trunk 5K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000-1001 5K-FCOE(config-if)# spanning-tree port type edge trunk 5K-FCOE(config)# int vfc 1 5K-FCOE(config-if)# bind interface port-channel 12 5K-FCOE(config-if)# switchport trunk allowed vsan 1000-1001
July 16, 2014 at 22:45
Dear Martijn,
thanks for the reference guide. However, I have a question regarding to your storage vdc config. Are you sure you need to use feature fport-channel-trunk in storage vdc? The fact is even this feature is not enabled, vfc interfaces can still come up as TF port.
Cheers,
James
July 17, 2014 at 15:48
Hi James,
You’re right, it does seem to come up without the feature being enabled. Even weirder, it seems that you cannot turn it off – “switchport trunk mode off” gives an error.
Unfortunately, I cannot find anything in the documentation that explains this behaviour, so I’ll update the fport-channel-trunk bit to a N5K and keep looking for the N7K behaviour.
January 6, 2015 at 14:05
connecting between two switchs, i see you configure “spanning-tree port type edge trunk” on Nexus 5k side, what ‘s the detail meaning about the command .
thanks
January 6, 2015 at 22:20
Hi Xiao,
You usually just see that config option on host interfaces, to quickly get the interface online and even prevent the FCoE handshake to fail when the port comes online too slow. I use it on switch to switch links as well for the first reason.
January 7, 2015 at 04:25
Thanks Martijn quickly respond.
Cisco document point out edge ports, which are connected to hosts, can be either an access port or a trunk port.
In other ways, with this command the interface should not receive BPDU, so you suggestion to configure it between two switches ?
For the section “Connecting a host” ,from cisco best practise document :
You must not configure an FCoE VLAN as the native VLAN of the trunk port.
The native VLAN is the default VLAN on a trunk. Any untagged frames transit the trunk as native VLAN traffic.
You should use an FCoE VLAN only for FCoE.
Do not use the default VLAN, VLAN1, as an FCoE VLAN.
———it’s wrong ?
5K-SW2(config-if)# switchport trunk native vlan 1000
5K-SW2(config-if)# switchport trunk allowed vlan 1000
5K-SW2(config-if)# spanning-tree port type edge trunk
but the vfc port can be up trunking with the wrong configure
what about you opinions ?
January 7, 2015 at 09:17
FCoE topology can not be meshed, like you would do with regular ethernet. This means there are architecturally no loops present inside the FCoE data path. So if your design is correct, there are no objections for disabling spanning-tree on those interfaces.
And you’re right about FCoE not being the native VLAN on a trunk; the data VLAN of the host should be the native VLAN. The FCoE VLAN is discovered by DCBX/LLDP and then used as tagged by the host. But in my examples I’m using VLANs 1000 and 2000 as FCoE VLANs and 10 and 20 as the data VLANs.
Also, the vFC interface will not come up if it cannot negotiate with the host properly, so if you’ve got a misconfiguration, you’ll notice by the vFC interface not coming up. 🙂
January 7, 2015 at 10:04
For Nexus 5K connecting to Host situation:
If the C serial UCS no vlan tag for the vHBA interface , how to configure the ethernet and vfc . the customer also want to running vsan/vlan 15 .
interface e 4/1
sw mode trunk
sw trunk allow vlan 15
sw trunk na vlan 15
spanning-tree port ty ed tr
no shu
inter vfc 15
bind inter e 4/1
sw mode f
sw trunk all vsan 15
no shu
vsan da
vsan 15 interface vfc 15
or
inter e 4/1
sw mode trunk
spann port ty ed tr
—-default native vlan =1
—-trunk link default allow vlans all
vsan da
vsan 15 interface vfc 15
inter vfc 15
bind inter e 4/1
no shu
which solution can solve the problem. thanks
January 7, 2015 at 10:15
Second config, as the first uses the FCoE VLAN as the native VLAN, which is not supported. Don’t forget to create VLAN 15 and mark it as “fcoe vsan 15” as well and mind the order of operations (create VLAN, VSAN, ethernet interface, vfc interface, no shutdown)