So, there is a little known feature in Cisco UCS that allows one to monitor traffic on the control plane without the hassle of actually hooking up an Ethernet analyzer. The control plane is physically the “mgmt0″ port on the Fabric Interconnect (FI) and it is used for managing the FIs themselves and for attaching to KVM sessions on the blades. Cisco UCS makes the capture process really easy and in this article I’ll show you how to do it. Remember, this process only captures control plane traffic – not data traffic (UCS has a similar function called Traffic Monitor for that). So, how is this useful? Well, there’s always the chance that something unexpected could happen in UCS Manager as a result of malformed packets entering the Fabric Interconnect’s mgmt0 port. TAC would need to see what the data looks like in order to determine the cause. But the more likely scenario here is that you are using the Cisco UCS XML API and would like to inspect the management traffic being sent to and from the Fabric Interconnect from either UCS Manager or some other external manager controlling UCS. This is an extremely useful tip to aid in the XML API learning process. If you are not familiar with the API of Cisco UCS, it’s an extremely powerful engine provided to customers, system integrators, independent software vendors (ISV). You can find additional information about the API, download it, and learn how to work with it on the following site:
http://developer.cisco.com/web/unifiedcomputing/start
To get started, open an ssh connection to the FI cluster address.
…[output omitted]…
UCS-6200-TOP-A(nxos)# ethanalyzer local interface mgmt
Capturing on eth0
wireshark-broadcom-rcpu-dissector: ethertype=0xde08, devicetype=0×0
2012-07-02 14:08:44.766481 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=68
2012-07-02 14:08:44.766742 192.168.0.111 -> 192.168.0.22 TCP 51444 > ssh [ACK] Seq=0 Ack=68 Win=16695 Len=0
2012-07-02 14:08:44.767196 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=116
2012-07-02 14:08:44.767246 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=148
2012-07-02 14:08:44.767314 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=148
2012-07-02 14:08:44.767709 192.168.0.111 -> 192.168.0.22 TCP 51444 > ssh [ACK] Seq=0 Ack=332 Win=16629 Len=0
2012-07-02 14:08:44.767734 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=444
2012-07-02 14:08:44.767879 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=148
2012-07-02 14:08:44.768019 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=148
2012-07-02 14:08:44.768207 192.168.0.111 -> 192.168.0.22 TCP 51444 > ssh [ACK] Seq=0 Ack=924 Win=16481 Len=0
10 packets captured
As you can see, the built-in protocol analyzer is called “ethanalyzer” and it’s based on Wireshark open source code. It both captures packets and displays them to the screen (by default). In the example above, it is simply capturing and displaying the last 10 packets, but it’s very flexible to capture more. Remember when I said that this is for control plane traffic only? Well that’s true, but let me add a caveat to that statement by saying that some management traffic enters the switch through the data ports (LACP, DCBX, FC, and FCoE). Ethanalyzer is able to capture these packets as well and does so using different Ethernet interfaces than mgmt0. To make this easy to understand, think of this in terms of a host that has 3 interfaces (there are physically more, but only 3 are exposed to ethanalyzer).
eth0=interface mgmt0
eth3=internal interface listening for low priority control packets on the data plane ports (IGMP, CDP, and mgmt-only TCP/UDP/IP/ARP)
eth4=internal interface listening for high priority control packets on the data plane ports (STP, LACP, DCBX, FC, and FCoE)
I mention this not because I think you will use eth3 and eth4, but if you run the command, you are going to see high and low options and I’m sure that it will make you curious. So, now to make ethanalyzer capture what you are looking for is pretty easy. Here’s an example of me capturing the first 100 frames that FI sees on mgmt0 after I execute the command:
Capturing on eth0
wireshark-broadcom-rcpu-dissector: ethertype=0xde08, devicetype=0×0
2012-07-02 14:28:40.500365 192.168.0.22 -> 192.168.0.111 SSH Encrypted response packet len=68
2012-07-02 14:28:40.500629 192.168.0.111 -> 192.168.0.22 TCP 51444 > ssh [ACK] Seq=0 Ack=68 Win=16427 Len=0
[output omitted]
Pretty cool. But this is still just spitting the output to the screen which is of limited value. It would be more useful to look at the captured packets in an actual protocol analyzer like Wireshark. With ethanalyzer, you can do just that:
Then you can copy the file to your workstation like this:
UCS-6200-TOP-A# connect local-mgmt
UCS-6200-TOP-A(local-mgmt)# copy volatile:mycap.cap ftp:
And the CLI will prompt you for all of the server info prior to the copy. If you play around with the command a bit, you’ll see there are additional options for filters, etc. My suggestion would be to capture all the packets with no filters and then use filters inside of Wireshark to inspect the data you are looking for. Be sure and take a look at the “autostop” option – it will allow you to stop capturing based on time and/or log file size. Finally, if you are looking for detailed information about the analyzer, including how to capture management traffic on data plane ports, please refer to the whitepaper “Ethanalyzer: Cisco NX-OS Software Built-In Packet Capture Utility” located here:
Thanks for stopping by!
-Jeff