Ultimate Cisco Nexus FCoE Configuration Guide

This post is part of my CCIE Datacenter reference series and will cover all there is to get FCoE up and running on the Cisco Nexus 7k, 5k and 2k switches.

The text assumes you have a understanding knowledge of fundamentals (storage paths, flogis, vPC, etc) and can be used as a configuration reference (I basically wrote this for myself).

Installing & Enabling FCoE

Nexus 7K

Installing and enabling FCoE on the Nexus 7k involves getting the storage VDC up and running. This section will cover just that.

QoS

The Nexus 7k comes with a default QoS policy of default-nq-8e-policy, which does not have a no drop CoS. FCoE needs at least one, so change the network-qos policy to one that has a no drop CoS. default-nq-7e-policy is usually good enough. If you forget to change the policy, the creation of the storage VDC will not work.

7K-DEFAULT(config)# system qos 
7K-DEFAULT(config-sys-qos)# service-policy type network-qos default-nq-7e-policy 

Feature-set and license

7K-DEFAULT(config)# install feature-set fcoe 
7K-DEFAULT(config)# license fcoe module 1
7K-DEFAULT(config)# feature lldp

Creating the VDC

We’re going to dedicate VLANs 1000 to 1004 to FCoE.

7K-DEFAULT(config)# vdc 7K-FCOE type storage
7K-DEFAULT(config-vdc)# allow feature-set fcoe
7K-DEFAULT(config-vdc)# allocate fcoe-vlan-range 1000-1004 from vdcs DEFAULT

Dedicated interfaces

The storage VDC can have two types of interfaces; dedicated and shared. Dedicated interfaces are FCoE only and are not used for host connectivity. Usually used for FCoE-only storage controllers and host CNAs dedicated to FCoE or FCoE only trunks to distribution switches (5Ks).

7K-DEFAULT(config-vdc)# allocate interface Ethernet1/1-2

Shared interfaces

Shared interfaces are typically used to connect hosts to. The interface is shared between 2 VDCs, one being the storage VDC and the other being a data VDC. The storage VDC will handle the FCoE traffic and the data VDC will handle all the rest of the traffic. Prerequisite of shared interfaces is that they are in trunk mode and spanning-tree allows it to come up quickly.

7K-DEFAULT(config)# interface Ethernet1/3-4
7K-DEFAULT(config-if)# switchport mode trunk
7K-DEFAULT(config-if)# spanning-tree port type edge trunk
7K-DEFAULT(config)# vdc 7K-FCOE 
7K-DEFAULT(config-vdc)# allocate shared interface Ethernet1/3-4

Nexus 5K

Enabling FCoE on the Nexus 5k is a bit easier. Again, mind the QoS. The Nexus 5000 has no drop enabled by default (you can’t even take it off), but the Nexus 5500 does not.

5K-SW(config)# policy-map type network-qos FCOE
5K-SW(config-pmap-nq)# class type network-qos class-fcoe
5K-SW(config-pmap-nq-c)# pause no-drop 
5K-SW(config-pmap-nq-c)# mtu 2158
5K-SW(config-pmap-nq)# class type network-qos class-default
5K-SW(config-pmap-nq-c)# mtu 9216 (don't forget this one if you have a need for jumbo frames on the 5Ks)
5K-SW(config)# system qos
5K-SW(config-sys-qos)# service-policy type network-qos FCOE

5K-SW(config)# feature fcoe

Basic setup

With basic setup, we create a VSAN, link that to a VLAN and do some basic settings like zoning.

FCOE(config)# vsan database
FCOE(config-vsan-db)# vsan 1000
FCOE(config)# vlan 1000
FCOE(config-vlan)# fcoe vsan 1000
FCOE(config)# fcdomain domain 10 preferred vsan 1000 (make domain IDs predictable) 
FCOE(config)# fcdomain priority 10 vsan 1000 (make switch domain root)
FCOE(config)# fcdomain restart disruptive vsan 1000 (apply changes)
FCOE(config)# zone default-zone permit vsan 1000 (yes, this is generally bad. In production use proper zoning)

Connecting a host – Single wire

A simple host connecting into a VSAN with a single connection for data and storage.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk native vlan 10
5K-SW(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW(config-if)# spanning-tree port type edge trunk

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet1/1

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting a host – vPC wire

vPC is an excellent way to connect hosts to two Nexus switches to create a redundant setup. We’ll use the vPC for normal data traffic and bind the vfc interfaces to the physical interfaces.

Nexus 5K-1

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# channel-group 1 mode active
5K-SW1(config)# interface port-channel1
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# spanning-tree port type edge trunk
5K-SW1(config-if)# vpc 1

5K-SW1(config)# interface vfc1
5K-SW1(config-if)# bind interface Ethernet1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2

5K-SW1(config)# interface Ethernet1/1
5K-SW2(config-if)# channel-group 1 mode active
5K-SW2(config)# interface port-channel1
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# spanning-tree port type edge trunk
5K-SW2(config-if)# vpc 1

5K-SW2(config)# interface vfc1
5K-SW2(config-if)# bind interface Ethernet1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host – FEX Single Homed – Single wire

5K-SW(config)# fex 100
5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode fex-fabric
5K-SW(config-if)# fex associate 100

5K-SW(config)# interface Ethernet100/1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk native vlan 10
5K-SW(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW(config-if)# spanning-tree port type edge trunk

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet100/1/1

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting a host – FEX Single Homed – Dual wire

fcoe-fex-single-homed-dual-wire

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100

5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk

5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk

5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host – FEX Single Homed – vPC wire

fcoe-single-homed-fex-vpc

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100

5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# channel-group 101
5K-SW1(config)# interface port-channel 101
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk
5K-SW1(config-if)# vpc 101

5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# channel-group 201
5K-SW2(config)# interface port-channel 201
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk
5K-SW2(config-if)# vpc 201

5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host – FEX Dual Homed – vPC wire

fcoe-dual-homed-fex

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe
5K-SW1(config)# fex 200

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100
5K-SW1(config-if)# channel-group 100
5K-SW1(config)# interface Ethernet1/2
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 200
5K-SW1(config-if)# channel-group 200

5K-SW1(config)# interface port-channel 100
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100
5K-SW1(config-if)# vpc 100
5K-SW1(config)# interface port-channel 200
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 200
5K-SW1(config-if)# vpc 200

- Create the host port-channels and mark one side for FCoE
5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# channel-group 101
5K-SW1(config)# interface port-channel 101
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk

5K-SW1(config)# interface Ethernet200/1/1
5K-SW1(config-if)# channel-group 201
5K-SW1(config)# interface port-channel 201
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10
5K-SW1(config-if)# spanning-tree port type edge trunk

- Create the virtual FC interface
5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 100
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/1
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 100
5K-SW2(config-if)# channel-group 100
5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200
5K-SW2(config-if)# channel-group 200

5K-SW2(config)# interface port-channel 100
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 100
5K-SW2(config-if)# vpc 100
5K-SW2(config)# interface port-channel 200
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200
5K-SW2(config-if)# vpc 200

- Create the host port-channels and mark one side for FCoE
5K-SW2(config)# interface Ethernet100/1/1
5K-SW2(config-if)# channel-group 101
5K-SW2(config)# interface port-channel 101
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10
5K-SW2(config-if)# spanning-tree port type edge trunk

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# channel-group 201
5K-SW2(config)# interface port-channel 201
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk

- Create the virtual FC interface
5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a switch – Single wire

Switch to switch, allowing multiple VSAN and data traffic at the same time.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet1/1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

Connecting a switch – Port-channel

Switch to switch with multiple wires forming a channel, allowing multiple VSAN and data traffic at the same time.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# channel-group 1 mode active
5K-SW(config)# interface port-channel 1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface port-channel1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

There’s an interface called a vfc-port-channel, which is only available on the Nexus 7k. I haven’t been able to find a significant difference between creating a vfc-port-channel and just binding a vfc interface to a port-channel, but it’s available if you want it.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# channel-group 1 mode active
5K-SW(config)# interface port-channel 1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc-port-channel 1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

Connecting a switch – FCoE-NPV

7K-FCOE(config)# feature npiv
7K-FCOE(config)# interface Ethernet1/1
7K-FCOE(config-if)# switchport mode trunk
7K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000

7K-FCOE(config)# int vfc 1
7K-FCOE(config-if)# bind interface Ethernet1/1

7K-FCOE(config)# vsan database
7K-FCOE(config-vsan-db)# vsan 1000 interface vfc1

5K-SW(config)# feature fcoe-npv
5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000

5K-SW(config)# int vfc 1
5K-SW(config-if)# bind interface Ethernet1/1
5K-SW(config-if)# switchport mode NP

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting UCS – Port-channel

fcoe-ucs-channel

5K-FCOE(config)# feature npiv
5K-FCOE(config)# feature fport-channel-trunk
5K-FCOE(config)# interface Ethernet1/1-2
5K-FCOE(config-if)# channel-group 12 mode active
5K-FCOE(config)# interface port-channel 12
5K-FCOE(config-if)# switchport mode trunk
5K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000-1001
5K-FCOE(config-if)# spanning-tree port type edge trunk

5K-FCOE(config)# int vfc 1
5K-FCOE(config-if)# bind interface port-channel 12
5K-FCOE(config-if)# switchport trunk allowed vsan 1000-1001


Share the wealth!

8 Comments

  1. Dear Martijn,

    thanks for the reference guide. However, I have a question regarding to your storage vdc config. Are you sure you need to use feature fport-channel-trunk in storage vdc? The fact is even this feature is not enabled, vfc interfaces can still come up as TF port.

    Cheers,
    James

    • Martijn

      July 17, 2014 at 15:48

      Hi James,

      You’re right, it does seem to come up without the feature being enabled. Even weirder, it seems that you cannot turn it off – “switchport trunk mode off” gives an error.

      Unfortunately, I cannot find anything in the documentation that explains this behaviour, so I’ll update the fport-channel-trunk bit to a N5K and keep looking for the N7K behaviour.

  2. connecting between two switchs, i see you configure “spanning-tree port type edge trunk” on Nexus 5k side, what ‘s the detail meaning about the command .
    thanks

    • Martijn

      January 6, 2015 at 22:20

      Hi Xiao,

      You usually just see that config option on host interfaces, to quickly get the interface online and even prevent the FCoE handshake to fail when the port comes online too slow. I use it on switch to switch links as well for the first reason.

      • Thanks Martijn quickly respond.
        Cisco document point out edge ports, which are connected to hosts, can be either an access port or a trunk port.
        In other ways, with this command the interface should not receive BPDU, so you suggestion to configure it between two switches ?
        For the section “Connecting a host” ,from cisco best practise document :

        You must not configure an FCoE VLAN as the native VLAN of the trunk port.
        The native VLAN is the default VLAN on a trunk. Any untagged frames transit the trunk as native VLAN traffic.

        You should use an FCoE VLAN only for FCoE.
        Do not use the default VLAN, VLAN1, as an FCoE VLAN.

        ———it’s wrong ?
        5K-SW2(config-if)# switchport trunk native vlan 1000
        5K-SW2(config-if)# switchport trunk allowed vlan 1000
        5K-SW2(config-if)# spanning-tree port type edge trunk

        but the vfc port can be up trunking with the wrong configure
        what about you opinions ?

        • Martijn

          January 7, 2015 at 09:17

          FCoE topology can not be meshed, like you would do with regular ethernet. This means there are architecturally no loops present inside the FCoE data path. So if your design is correct, there are no objections for disabling spanning-tree on those interfaces.

          And you’re right about FCoE not being the native VLAN on a trunk; the data VLAN of the host should be the native VLAN. The FCoE VLAN is discovered by DCBX/LLDP and then used as tagged by the host. But in my examples I’m using VLANs 1000 and 2000 as FCoE VLANs and 10 and 20 as the data VLANs.

          Also, the vFC interface will not come up if it cannot negotiate with the host properly, so if you’ve got a misconfiguration, you’ll notice by the vFC interface not coming up. 🙂

          • For Nexus 5K connecting to Host situation:

            If the C serial UCS no vlan tag for the vHBA interface , how to configure the ethernet and vfc . the customer also want to running vsan/vlan 15 .

            interface e 4/1
            sw mode trunk
            sw trunk allow vlan 15
            sw trunk na vlan 15
            spanning-tree port ty ed tr
            no shu
            inter vfc 15
            bind inter e 4/1
            sw mode f
            sw trunk all vsan 15
            no shu
            vsan da
            vsan 15 interface vfc 15

            or

            inter e 4/1
            sw mode trunk
            spann port ty ed tr
            —-default native vlan =1
            —-trunk link default allow vlans all
            vsan da
            vsan 15 interface vfc 15
            inter vfc 15
            bind inter e 4/1
            no shu

            which solution can solve the problem. thanks

          • Martijn

            January 7, 2015 at 10:15

            Second config, as the first uses the FCoE VLAN as the native VLAN, which is not supported. Don’t forget to create VLAN 15 and mark it as “fcoe vsan 15” as well and mind the order of operations (create VLAN, VSAN, ethernet interface, vfc interface, no shutdown)

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2024 Lostdomain

Theme by Anders NorénUp ↑