This post is part of my CCIE Datacenter reference series and will cover all there is to get FCoE up and running on the Cisco Nexus 7k, 5k and 2k switches.

The text assumes you have a understanding knowledge of fundamentals (storage paths, flogis, vPC, etc) and can be used as a configuration reference (I basically wrote this for myself).

Installing & Enabling FCoE

Nexus 7K

Installing and enabling FCoE on the Nexus 7k involves getting the storage VDC up and running. This section will cover just that.

QoS

The Nexus 7k comes with a default QoS policy of default-nq-8e-policy, which does not have a no drop CoS. FCoE needs at least one, so change the network-qos policy to one that has a no drop CoS. default-nq-7e-policy is usually good enough. If you forget to change the policy, the creation of the storage VDC will not work.

Feature-set and license

Creating the VDC

We’re going to dedicate VLANs 1000 to 1004 to FCoE.

Dedicated interfaces

The storage VDC can have two types of interfaces; dedicated and shared. Dedicated interfaces are FCoE only and are not used for host connectivity. Usually used for FCoE-only storage controllers and host CNAs dedicated to FCoE or FCoE only trunks to distribution switches (5Ks).

Shared interfaces

Shared interfaces are typically used to connect hosts to. The interface is shared between 2 VDCs, one being the storage VDC and the other being a data VDC. The storage VDC will handle the FCoE traffic and the data VDC will handle all the rest of the traffic. Prerequisite of shared interfaces is that they are in trunk mode and spanning-tree allows it to come up quickly.

Nexus 5K

Enabling FCoE on the Nexus 5k is a bit easier. Again, mind the QoS. The Nexus 5000 has no drop enabled by default (you can’t even take it off), but the Nexus 5500 does not.

Basic setup

With basic setup, we create a VSAN, link that to a VLAN and do some basic settings like zoning.

Connecting a host – Single wire

A simple host connecting into a VSAN with a single connection for data and storage.

Connecting a host – vPC wire

vPC is an excellent way to connect hosts to two Nexus switches to create a redundant setup. We’ll use the vPC for normal data traffic and bind the vfc interfaces to the physical interfaces.

Connecting a host – FEX Single Homed – Single wire

Connecting a host – FEX Single Homed – Dual wire

fcoe-fex-single-homed-dual-wire

Connecting a host – FEX Single Homed – vPC wire

fcoe-single-homed-fex-vpc

Connecting a host – FEX Dual Homed – vPC wire

fcoe-dual-homed-fex

Connecting a switch – Single wire

Switch to switch, allowing multiple VSAN and data traffic at the same time.

Connecting a switch – Port-channel

Switch to switch with multiple wires forming a channel, allowing multiple VSAN and data traffic at the same time.

There’s an interface called a vfc-port-channel, which is only available on the Nexus 7k. I haven’t been able to find a significant difference between creating a vfc-port-channel and just binding a vfc interface to a port-channel, but it’s available if you want it.

Connecting a switch – FCoE-NPV

Connecting UCS – Port-channel

fcoe-ucs-channel



Share the wealth!

8 comments on “Ultimate Cisco Nexus FCoE Configuration Guide

  • Dear Martijn,

    thanks for the reference guide. However, I have a question regarding to your storage vdc config. Are you sure you need to use feature fport-channel-trunk in storage vdc? The fact is even this feature is not enabled, vfc interfaces can still come up as TF port.

    Cheers,
    James

    • Hi James,

      You’re right, it does seem to come up without the feature being enabled. Even weirder, it seems that you cannot turn it off – “switchport trunk mode off” gives an error.

      Unfortunately, I cannot find anything in the documentation that explains this behaviour, so I’ll update the fport-channel-trunk bit to a N5K and keep looking for the N7K behaviour.

  • connecting between two switchs, i see you configure “spanning-tree port type edge trunk” on Nexus 5k side, what ‘s the detail meaning about the command .
    thanks

    • Hi Xiao,

      You usually just see that config option on host interfaces, to quickly get the interface online and even prevent the FCoE handshake to fail when the port comes online too slow. I use it on switch to switch links as well for the first reason.

      • Thanks Martijn quickly respond.
        Cisco document point out edge ports, which are connected to hosts, can be either an access port or a trunk port.
        In other ways, with this command the interface should not receive BPDU, so you suggestion to configure it between two switches ?
        For the section “Connecting a host” ,from cisco best practise document :

        You must not configure an FCoE VLAN as the native VLAN of the trunk port.
        The native VLAN is the default VLAN on a trunk. Any untagged frames transit the trunk as native VLAN traffic.

        You should use an FCoE VLAN only for FCoE.
        Do not use the default VLAN, VLAN1, as an FCoE VLAN.

        ———it’s wrong ?
        5K-SW2(config-if)# switchport trunk native vlan 1000
        5K-SW2(config-if)# switchport trunk allowed vlan 1000
        5K-SW2(config-if)# spanning-tree port type edge trunk

        but the vfc port can be up trunking with the wrong configure
        what about you opinions ?

        • FCoE topology can not be meshed, like you would do with regular ethernet. This means there are architecturally no loops present inside the FCoE data path. So if your design is correct, there are no objections for disabling spanning-tree on those interfaces.

          And you’re right about FCoE not being the native VLAN on a trunk; the data VLAN of the host should be the native VLAN. The FCoE VLAN is discovered by DCBX/LLDP and then used as tagged by the host. But in my examples I’m using VLANs 1000 and 2000 as FCoE VLANs and 10 and 20 as the data VLANs.

          Also, the vFC interface will not come up if it cannot negotiate with the host properly, so if you’ve got a misconfiguration, you’ll notice by the vFC interface not coming up. 🙂

          • For Nexus 5K connecting to Host situation:

            If the C serial UCS no vlan tag for the vHBA interface , how to configure the ethernet and vfc . the customer also want to running vsan/vlan 15 .

            interface e 4/1
            sw mode trunk
            sw trunk allow vlan 15
            sw trunk na vlan 15
            spanning-tree port ty ed tr
            no shu
            inter vfc 15
            bind inter e 4/1
            sw mode f
            sw trunk all vsan 15
            no shu
            vsan da
            vsan 15 interface vfc 15

            or

            inter e 4/1
            sw mode trunk
            spann port ty ed tr
            —-default native vlan =1
            —-trunk link default allow vlans all
            vsan da
            vsan 15 interface vfc 15
            inter vfc 15
            bind inter e 4/1
            no shu

            which solution can solve the problem. thanks

          • Second config, as the first uses the FCoE VLAN as the native VLAN, which is not supported. Don’t forget to create VLAN 15 and mark it as “fcoe vsan 15” as well and mind the order of operations (create VLAN, VSAN, ethernet interface, vfc interface, no shutdown)

Leave a Reply

Your email address will not be published. Required fields are marked *