Lostdomain

Ultimate Cisco Nexus FCoE Configuration Guide

Date

Mon Jun 09

Author

Martijn

Tags

  1. cciedc
  2. cisco
  3. fcoe
  4. vdc

This post is part of my CCIE Datacenter reference series and will cover all there is to get FCoE up and running on the Cisco Nexus 7k, 5k and 2k switches.

The text assumes you have a understanding knowledge of fundamentals (storage paths, flogis, vPC, etc) and can be used as a configuration reference (I basically wrote this for myself).

[toc]

Installing & Enabling FCoE

Nexus 7K

Installing and enabling FCoE on the Nexus 7k involves getting the storage VDC up and running. This section will cover just that.

QoS

The Nexus 7k comes with a default QoS policy of default-nq-8e-policy, which does not have a no drop CoS. FCoE needs at least one, so change the network-qos policy to one that has a no drop CoS. default-nq-7e-policy is usually good enough. If you forget to change the policy, the creation of the storage VDC will not work.

7K-DEFAULT(config)# system qos 
7K-DEFAULT(config-sys-qos)# service-policy type network-qos default-nq-7e-policy

Feature-set and license

7K-DEFAULT(config)# install feature-set fcoe 
7K-DEFAULT(config)# license fcoe module 1
7K-DEFAULT(config)# feature lldp

Creating the VDC

We’re going to dedicate VLANs 1000 to 1004 to FCoE.

7K-DEFAULT(config)# vdc 7K-FCOE type storage
7K-DEFAULT(config-vdc)# allow feature-set fcoe
7K-DEFAULT(config-vdc)# allocate fcoe-vlan-range 1000-1004 from vdcs DEFAULT

Dedicated interfaces

The storage VDC can have two types of interfaces; dedicated and shared. Dedicated interfaces are FCoE only and are not used for host connectivity. Usually used for FCoE-only storage controllers and host CNAs dedicated to FCoE or FCoE only trunks to distribution switches (5Ks).

7K-DEFAULT(config-vdc)# allocate interface Ethernet1/1-2

Shared interfaces

Shared interfaces are typically used to connect hosts to. The interface is shared between 2 VDCs, one being the storage VDC and the other being a data VDC. The storage VDC will handle the FCoE traffic and the data VDC will handle all the rest of the traffic. Prerequisite of shared interfaces is that they are in trunk mode and spanning-tree allows it to come up quickly.

7K-DEFAULT(config)# interface Ethernet1/3-4
7K-DEFAULT(config-if)# switchport mode trunk
7K-DEFAULT(config-if)# spanning-tree port type edge trunk
7K-DEFAULT(config)# vdc 7K-FCOE 
7K-DEFAULT(config-vdc)# allocate shared interface Ethernet1/3-4

Nexus 5K

Enabling FCoE on the Nexus 5k is a bit easier. Again, mind the QoS. The Nexus 5000 has no drop enabled by default (you can’t even take it off), but the Nexus 5500 does not.

5K-SW(config)# policy-map type network-qos FCOE
5K-SW(config-pmap-nq)# class type network-qos class-fcoe
5K-SW(config-pmap-nq-c)# pause no-drop 
5K-SW(config-pmap-nq-c)# mtu 2158
5K-SW(config-pmap-nq)# class type network-qos class-default
5K-SW(config-pmap-nq-c)# mtu 9216 (don't forget this one if you have a need for jumbo frames on the 5Ks)
5K-SW(config)# system qos
5K-SW(config-sys-qos)# service-policy type network-qos FCOE

5K-SW(config)# feature fcoe

Basic setup

With basic setup, we create a VSAN, link that to a VLAN and do some basic settings like zoning.

FCOE(config)# vsan database
FCOE(config-vsan-db)# vsan 1000
FCOE(config)# vlan 1000
FCOE(config-vlan)# fcoe vsan 1000
FCOE(config)# fcdomain domain 10 preferred vsan 1000 (make domain IDs predictable) 
FCOE(config)# fcdomain priority 10 vsan 1000 (make switch domain root)
FCOE(config)# fcdomain restart disruptive vsan 1000 (apply changes)
FCOE(config)# zone default-zone permit vsan 1000 (yes, this is generally bad. In production use proper zoning)

Connecting a host - Single wire

A simple host connecting into a VSAN with a single connection for data and storage.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk native vlan 10
5K-SW(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW(config-if)# spanning-tree port type edge trunk

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet1/1

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting a host - vPC wire

vPC is an excellent way to connect hosts to two Nexus switches to create a redundant setup. We’ll use the vPC for normal data traffic and bind the vfc interfaces to the physical interfaces.

Nexus 5K-1

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# channel-group 1 mode active
5K-SW1(config)# interface port-channel1
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# spanning-tree port type edge trunk
5K-SW1(config-if)# vpc 1

5K-SW1(config)# interface vfc1
5K-SW1(config-if)# bind interface Ethernet1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2

5K-SW1(config)# interface Ethernet1/1
5K-SW2(config-if)# channel-group 1 mode active
5K-SW2(config)# interface port-channel1
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# spanning-tree port type edge trunk
5K-SW2(config-if)# vpc 1

5K-SW2(config)# interface vfc1
5K-SW2(config-if)# bind interface Ethernet1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host - FEX Single Homed - Single wire

5K-SW(config)# fex 100
5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode fex-fabric
5K-SW(config-if)# fex associate 100

5K-SW(config)# interface Ethernet100/1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk native vlan 10
5K-SW(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW(config-if)# spanning-tree port type edge trunk

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet100/1/1

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting a host - FEX Single Homed - Dual wire

fcoe-fex-single-homed-dual-wire

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100

5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk

5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk

5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host - FEX Single Homed - vPC wire

fcoe-single-homed-fex-vpc

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100

5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# channel-group 101
5K-SW1(config)# interface port-channel 101
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk
5K-SW1(config-if)# vpc 101

5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# channel-group 201
5K-SW2(config)# interface port-channel 201
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk
5K-SW2(config-if)# vpc 201

5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a host - FEX Dual Homed - vPC wire

fcoe-dual-homed-fex

Nexus 5K-1
- Get the FEXes up and running first
5K-SW1(config)# fex 100
5K-SW1(config-fex)# fcoe
5K-SW1(config)# fex 200

5K-SW1(config)# interface Ethernet1/1
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100
5K-SW1(config-if)# channel-group 100
5K-SW1(config)# interface Ethernet1/2
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 200
5K-SW1(config-if)# channel-group 200

5K-SW1(config)# interface port-channel 100
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 100
5K-SW1(config-if)# vpc 100
5K-SW1(config)# interface port-channel 200
5K-SW1(config-if)# switchport mode fex-fabric
5K-SW1(config-if)# fex associate 200
5K-SW1(config-if)# vpc 200

- Create the host port-channels and mark one side for FCoE
5K-SW1(config)# interface Ethernet100/1/1
5K-SW1(config-if)# channel-group 101
5K-SW1(config)# interface port-channel 101
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10, 1000
5K-SW1(config-if)# spanning-tree port type edge trunk

5K-SW1(config)# interface Ethernet200/1/1
5K-SW1(config-if)# channel-group 201
5K-SW1(config)# interface port-channel 201
5K-SW1(config-if)# switchport mode trunk
5K-SW1(config-if)# switchport trunk native vlan 10
5K-SW1(config-if)# switchport trunk allowed vlan 10
5K-SW1(config-if)# spanning-tree port type edge trunk

- Create the virtual FC interface
5K-SW1(config)# int vfc1
5K-SW1(config-if)# bind interface Ethernet100/1/1

5K-SW1(config)# vsan database
5K-SW1(config-vsan-db)# vsan 1000 interface vfc1

Nexus 5K-2
- Get the FEXes up and running first
5K-SW2(config)# fex 100
5K-SW2(config)# fex 200
5K-SW2(config-fex)# fcoe

5K-SW2(config)# interface Ethernet1/1
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 100
5K-SW2(config-if)# channel-group 100
5K-SW2(config)# interface Ethernet1/2
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200
5K-SW2(config-if)# channel-group 200

5K-SW2(config)# interface port-channel 100
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 100
5K-SW2(config-if)# vpc 100
5K-SW2(config)# interface port-channel 200
5K-SW2(config-if)# switchport mode fex-fabric
5K-SW2(config-if)# fex associate 200
5K-SW2(config-if)# vpc 200

- Create the host port-channels and mark one side for FCoE
5K-SW2(config)# interface Ethernet100/1/1
5K-SW2(config-if)# channel-group 101
5K-SW2(config)# interface port-channel 101
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10
5K-SW2(config-if)# spanning-tree port type edge trunk

5K-SW2(config)# interface Ethernet200/1/1
5K-SW2(config-if)# channel-group 201
5K-SW2(config)# interface port-channel 201
5K-SW2(config-if)# switchport mode trunk
5K-SW2(config-if)# switchport trunk native vlan 10
5K-SW2(config-if)# switchport trunk allowed vlan 10, 2000
5K-SW2(config-if)# spanning-tree port type edge trunk

- Create the virtual FC interface
5K-SW2(config)# int vfc1
5K-SW2(config-if)# bind interface Ethernet200/1/1

5K-SW2(config)# vsan database
5K-SW2(config-vsan-db)# vsan 2000 interface vfc1

Connecting a switch - Single wire

Switch to switch, allowing multiple VSAN and data traffic at the same time.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface Ethernet1/1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

Connecting a switch - Port-channel

Switch to switch with multiple wires forming a channel, allowing multiple VSAN and data traffic at the same time.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# channel-group 1 mode active
5K-SW(config)# interface port-channel 1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc1
5K-SW(config-if)# bind interface port-channel1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

There’s an interface called a vfc-port-channel, which is only available on the Nexus 7k. I haven’t been able to find a significant difference between creating a vfc-port-channel and just binding a vfc interface to a port-channel, but it’s available if you want it.

5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# channel-group 1 mode active
5K-SW(config)# interface port-channel 1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000-1001

5K-SW(config)# int vfc-port-channel 1
5K-SW(config-if)# switchport mode E
5K-SW(config-if)# switchport trunk allowed vsan 1000-1001

Connecting a switch - FCoE-NPV

7K-FCOE(config)# feature npiv
7K-FCOE(config)# interface Ethernet1/1
7K-FCOE(config-if)# switchport mode trunk
7K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000

7K-FCOE(config)# int vfc 1
7K-FCOE(config-if)# bind interface Ethernet1/1

7K-FCOE(config)# vsan database
7K-FCOE(config-vsan-db)# vsan 1000 interface vfc1

5K-SW(config)# feature fcoe-npv
5K-SW(config)# interface Ethernet1/1
5K-SW(config-if)# switchport mode trunk
5K-SW(config-if)# switchport trunk allowed vlan 10-11, 1000

5K-SW(config)# int vfc 1
5K-SW(config-if)# bind interface Ethernet1/1
5K-SW(config-if)# switchport mode NP

5K-SW(config)# vsan database
5K-SW(config-vsan-db)# vsan 1000 interface vfc1

Connecting UCS - Port-channel

fcoe-ucs-channel

5K-FCOE(config)# feature npiv
5K-FCOE(config)# feature fport-channel-trunk
5K-FCOE(config)# interface Ethernet1/1-2
5K-FCOE(config-if)# channel-group 12 mode active
5K-FCOE(config)# interface port-channel 12
5K-FCOE(config-if)# switchport mode trunk
5K-FCOE(config-if)# switchport trunk allowed vlan 10-11, 1000-1001
5K-FCOE(config-if)# spanning-tree port type edge trunk

5K-FCOE(config)# int vfc 1
5K-FCOE(config-if)# bind interface port-channel 12
5K-FCOE(config-if)# switchport trunk allowed vsan 1000-1001