This post is part of my VMware VCIX-NV Study Guide and details the possibilities to use the command line interface on all sections of NSX.

Documentation

Index

 

Command Line Interface on NSX
Traditional network infrastructure is managed via command line. Every network administrator out there is glued to the command line for configuration, troubleshooting and setup tasks. Even though this is changing by all kinds of automation tools out there that present a GUI (ACI, APIC-EM, NSX, OpenFlow GUIs, etc), most of them retain some form of command line interface (CLI). VMware NSX is no different.

NSX offers command line interfaces on the following:

  • NSX Manager
  • NSX Controller
  • NSX Edge Gateway Services appliance
  • NSX Logical Distributed Router
  • ESXi host (existing, but added functionality for NSX)

There are way too many useful commands to cover them as a whole, so I recommend fully to read the entire (yes entire!@#) NSX Command Line Interface Reference. As for me, as usual the topics are laid out from the VCIX-NV blueprint and I’ll be share the most useful commands there are, to kickstart the process.

If you’re not a network administrator, please keep in mind that the console prompt displayed like this device> is called exec mode, which you get into by default when logging into a device. The console prompt displayed like device# is enabled mode, which you can get to by using the enable command and entering your password again. The examples below are displayed in both exec and enabled mode, it’s up to you to get there.

You should also check out the brilliant post by Sébastien Braun on NSX vSphere troubleshooting, as it has a lot of useful commands.

 

Manage and report on an NSX installation status using ESXi Command Line Interface (CLI) commands &

Manage and report on an NSX Infrastructure using NSX Manager, NSX Controller, and ESXi CLI commands

We’re going to run through the NSX components top to bottom and have a look at some CLI outputs and change some stuff. While I’m going through, I’m going to assume that by know you know to login via SSH onto the NSX component that is referenced. 🙂

NSX Manager
You do not have the entire functionality of the NSX Manager at your disposal via the CLI. There does not seem to be a way to manage the integration with vCenter and SSO, among others. Lets start by looking at the basic system configuration:

You need to be in enabled mode to show or modify anything configuration related. Somewhat like a regular switch, the output shows the DNS, NTP and IP configuration of the NSX Manager. If you’d like to change something, lets say the IP address of the NSX Manager, first go into configuration mode and set a new IP address:

Not sure if this remark is needed, but if you do this, you’ll lose your connection and you’ll need to set up the SSH connection again. With any other switch-type interface, you need to save the changes to the startup configuration file to save it permanently.

Using the CLI is a good way to get detailed logging from the NSX Manager:

NSX Controller
The NSX Controllers are a Linux-based appliance that contains all information of the virtual network, it retains and controls the active state of your network. Edges are registered here, along with their interfaces and routes. When requesting information from a controller cluster, make sure you connect to the master.

Lets start by getting the controller cluster state:

Or by getting a list of all deployed NSX Edges and showing the connected interfaces from one of them:

The LR-Id is the internal ID assigned to the NSX Edge by the controllers. You’ll need that ID to get more information about a specific Edge, like getting the interface overview:

This NSX Edge has 3 interfaces, 1 traditional VLAN interface and 2 new and shiny VXLAN interfaces. Lets get some more intel on the interface with IP 2.2.2.2/24:

Did you notice that there was another internal ID used (the interface ID) to get the detailed information? The controllers are full with these internal IDs. They’re always the first column, so they’re pretty easy to spot, but you need to reference them, keep that in mind.

NSX Edge Services Gateway
Of the two types of NSX Edges, the ESG and LDR. When you’re on the CLI, you don’t see a lot of difference, apart from the ESG having a lot more commands (simply because it has more functionality). The following examples are mostly applicable to both, unless otherwise is mentioned. The CLI of both Edge types are meant for showing information and debugging traffic flow, there is no real configuration possible.

Lets get a list of interfaces to start. NSX Edges are deployed with a number of interfaces, whether or not you decide to use them. This means the output of getting the interfaces can be huge, even though you thought you just configured 2 interfaces.

Regular interfaces are called vNic_X, but you’ll notice a few other interfaces listed in the output. The VDR interface is the Virtual Distributed Router (or LDR), the br-sub interface is the Layer 2 VPN tunnel interface and the lo interface is a loopback interface which can be used in routing protocol configuration (just like regular routers).

We got the interfaces, lets see if there are any ARP entries for connected hosts:

What about the routing table which the Edge uses to make routing decisions?

One particularly handy command on the ESG is to show the firewall flows to show top network consumers:

 

Manage and report on a Logical Switch using NSX Controller and ESXi CLI commands

Using the NSX controllers and ESXi host, you’re able to retrieve a great deal of information about a logical switch. Unfortunately, there does not seem to be a way to get a list of all logical switches created, so you’re going to have to lookup the switch ID through the GUI (look for the Segment ID). Once you get the switch ID (or VNI from here and forward), you can continue to the CLI and find this logical switchs’ dirty laundry:

This summary shows the assigned controller and amount of VTEPs that are connected to this logical switch. Lets move on to finding out which ESXi hosts (VTEPs) are connected:

Now for the connected virtual machine MAC addresses on a certain ESXi host:

ESXi Host
Inside an ESXi host you’ve been able to retrieve information using the esxcli command for a while now. The network directive inside the esxcli has been expanded to contain vxlan information, which translates to useful NSX information. For instance, from an ESXi Host, you can get a list of logical switches:

From the output above, you can get the VXLAN ID (or Segment ID, or Logical Switch ID), which kind of VXLAN replication is used (above is unicast, which is why there is no multicast IP specified), which controller is assigned to the logical switch and the ports, mac and arp counts within that logical switch.

Taking one step back, we can also show the distributed vSwitch which is handling our VXLAN traffic:

You can also get the remote MAC addresses, which are pushed from the controllers to the ESXi host:

And as my last example, you can also get a list of ESXi hosts that are linked to a certain logical switch and have VXLAN tunnels set up:

 

Manage and report on a Logical Router NSX Controller, NSX Edge, and ESXi CLI commands

We’ve already covered retrieving logical router information from the NSX Controller in the first topic on this page, so we’re going to go straight to the NSX Edge itself.

There are a few commands that you might use in an operational sense. Most of them we’ve already covered in other topics, so I’m going to just list them:

Show attached hosts:

Show installed routing table:

Show attached interfaces:

Show current network flows:

ESXi Host Commands
As the distributed router is embedded in the ESXi kernel, the ESXi host needs to know about its configuration in other to handle local routing traffic. This means that the routing info that the LDR Control VM collects, is pushed down to the ESXi host, so it can make informed decisions on where to send traffic.

VMware has extended the CLI on ESXi so you can pull that information out. I’ll walk you through some of the most used ones. One thing to note is that ‘LDR’ (logical distributed router) in terminology translated to VDR (virtual distributed router) inside ESXi. Lets start with getting an overview of the logical router instances that are located on the host:

Once you get the edge identifier, you can look up the interfaces of this NSX Edge, or the logical interfaces (lif):

As mentioned, the ESXi host knows about the routes the LDR has, so it can make the informed decision on where to route the traffic. If it can do a local route, it will. To verify the network routes installed in the LDR, you can use the following command:

Now you have the most important information for the routing of the logical distributed router inside the ESXi kernel. There’s another functionality that the LDR can perform; bridging of VXLAN and VLAN networks. We talked about how it works in previous chapters, now lets see how we can get the information about the bridge settings inside the ESXi host.

To get an overview of existing bridges, you can execute the following command:

From this output you can see that there is a bridge called ‘zulu’, that it has 2 networks attached and that those networks are VXLAN 5001 and VLAN 23. For troubleshooting purposes, you can also display the learned MAC addresses on both sides:

Normally there would be a list of MAC addresses learned from the two networks, but they have all timed out in this case. 😉

 

Manage and report on a Distributed Firewall using NSX Manager and ESXi CLI commands

The Distributed Firewall is inside the ESXi kernel, so the ESXi node knows about what policies are configured on the virtual machines the ESXi node hosts. You can learn about the policies set on a VM through the commandline of ESXi.

First, we need to find the UUID of the virtual machine called App01:

Then we look for the filter name for that virtual machine UUID:

As you might notice, this App01 virtual machine has two vNICs. That is why it has two policies attached to it.

After getting the filter name, you can look up the rules for that filter:

You can also look up the address lists that these rules are using for traffic policing:

 

Manage and report on an Edge Services VPN-Plus device using NSX Edge and client OS CLI commands

Once the SSL-VPN feature on a NSX Edge has been configured, you can use the command line of the NSX Edge to report the configuration and the current usage of SSL-VPN users.

As you can see the configuration is presented in a JSON format and all settings are included. Handy for easy safe-keeping. On to some operational commands:

Checking to see whether the SSL VPN-Plus service is running.

It is possible for the service to be configured through the GUI and possibly not running due to a malfunction in the service that runs on the NSX Edge. You’ll get a “connection refused” message if or when that happens. It’ll also return a “connection refused” when the SSL VPN-Plus service has not been configured through the GUI.

If the service is configured and is accepting users, you can view what open user sessions exist:

And for the last example, possibly the most important one, we’re going to look at the current active SSL VPN tunnels on the NSX Edge:

As you can see each tunnel will be attached to an user, have uptime and bandwidth counters and contains the virtual IP address that has been assigned to the user from the IP Pool you have configured.

 

Manage and report on Load Balancers using NSX Edge CLI commands

Similar with the SSL VPN-Plus feature of the NSX ESG, the configuration for load balancing has to be done from the GUI (or API), but information can be requested from the command line. Also just like the SSL VPN-Plus feature, you can request the running configuration:

All configured settings are in the JSON output presented to you. Handy for easy safe-keeping. Now for some operational command examples. Lets start with what is probably the most important one, getting the status of the virtual servers:

The output is fairly technical, but it’s pretty readable. There is a tree for every virtual server, listing the IP address and other settings first. Then it goes into the server pool attached to the virtual server, displays the global settings of that pool and then drills into the real servers configured in that pool. All objects in the output have a status field, which shows you whether the service is online. As you can see from this example, my web1 server is working correctly and receiving incoming connections, while the web2 server is down and not receiving any connections.

You can also omit the virtual server name in the command to get an overview of all operational virtual servers.

If you’re just looking of the status of a specific server pool, you can use this command:

The output is the same as the virtual server output, minus the virtual server details.

If you’re looking for connection information, there are two commands you can turn to. One to get the active sessions located in the NSX Edge and one to get the sticky mapping table. This sticky mapping table contains the mapping between origin IP address and real server, if you have configured a sticky bit on a virtual server so that a visitor does not switch between real servers.

 

That’s it for me on the command line examples. As I mentioned before, this is surely not all and you should definitely go through the NSX Command Line Reference and go explore the CLI.

 



Share the wealth!

Leave a Reply

Your email address will not be published. Required fields are marked *