So, today’s article is on VLAN separation, the problems it solves, and the problems it sometimes creates. Not all networks are cut from the same cloth. Some are simple and some are complex. Some are physical and some are virtual. Some are clean while others are quite messy. The one thing that they all have in common is that Cisco UCS works with all of them.
A Look Back
In UCS, we have a topology that we call “disjoint Layer 2” (DJL2) which simply means that a there are networks upstream from UCS that are separated from one another and cannot all be accessed by the same UCS uplink port (or port channel). For instance, you might have upstream VLANs 10, 20, and 30 on UCS uplink port 1 and VLANs 40, 50, and 60 on UCS uplink port 2. Prior to UCS 2.0, this configuration was not supported (in End Host Mode (EHM)). The main reason is that prior to 2.0, when VLANs were created, they were instantly available on ALL defined uplink ports and you could not assign certain VLANs to certain uplink ports. In addition to this, UCS uses the concept of a “designated receiver” (DR) port that is the single port (or port channel) chosen by UCSM to receive all multicast and broadcast traffic for all VLANs defined on the Fabric Interconnect (FI). To make this clear, UCS receives all multicast/broadcast traffic on this port only and drops broadcast/multicast traffic received on all other ports. Unless you have DJL2, this method works really well. If you do have DJL2, this would lead to a problem if you defined the above VLAN configuration and plugged it into pre-2.0 UCS (in EHM). In this situation, UCS would choose a designated receiver port for ALL VLANs (10-60) and assign it to one of the available uplinks. Let’s say the system chose port 1 (VLANs 10, 20, and 30) for the DR. In that situation, those networks (10, 20, 30) would work correctly, but VLANs 40, 50, and 60 (plugged into port 2) would not receive any broadcast and multicast traffic at all. The FI will learn the MAC addresses of the destinations on port 2 for 40, 50 and 60, but necessary protocols like ARP, PXE, DHCP (just to name a few) would be broken for these networks. In case you’re wondering, pin groups do not solve this problem so don’t waste your time. Instead, you need UCS 2.0+ and DJL2 which allows specific VLANs to be pinned to specific uplink ports. In addition, you now have a DR port for each defined VLAN as opposed to globally for the each FI. If you want to know more about the DR port, how it works, and how you can see which ports are the current DR on your own domain, please see the Cisco whitepaper entitled “Deploy Layer 2 Disjoint Networks Upstream in End Host Mode” located here: http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns944/white_paper_c11-692008.html
The Rules
You’ve probably figured out that if this were super easy, I wouldn’t be writing about it. Well, yes and no. It’s easy to turn on the DJL2 feature, but there are some lesser known rules around making it work. There is no “enable DJL2” button and you won’t find it by that name in UCSM. You simply enable it when you assign specific VLANs to specific uplink ports. It’s then automatically on. But many people make a mistake here. Staying with the above example, you want port 1 to carry VLANs 10-30 and port 2 to carry VLANs 40-60. When you first enter VLAN manager, you will see VLANs 10-60 defined and carried on ports 1 and 2. You might think to just take VLANs 40-60 and assign them to port 2. Well, that does remove 40-60 off of port 1, but it would also leave 10-30 on port 2 (along with 40-60). So you must isolate VLANs to their respective uplink ports. Furthermore, if you define a new VLAN, you need to go into VLAN manager and pin it to the port(s) you intend and remove it from the ports it should not be on. The main thing to remember here is that the original UCS rules on VLAN creation have not changed. That is, a created VLAN is always available on all uplink ports. That still happens even when you have DJL2 setup because UCS manager has no idea where to put that new VLAN unless you tell it – so it follows the original rule. I recommend looking at your VLAN config in NXOS (show VLAN) before you run it in production. This will verify that the changes you wanted to make are truly that changes you made in the GUI.
ENM Source Pinning Failed
So now we have DJL2 setup properly on our uplink. Let’s look at the server side as it is often an area of confusion. It’s probably also the way most of you found this blog entry because you googled for the term “ENM Source Pinning Failed”. Let me explain why. When you create vNICs on a service profile using the config we had above (10-30 on port 1 and 40-60 on port 2), you are not able to trunk/tag VLANs from BOTH port 1 and port 2 to the same vNIC. For example, you can have have a single vNIC with VLANs 10, 20, and 30 and another vNIC with VLANs 40, 50, and 60 and both vNICs can be on the same server. But you CANNOT have a single vNIC with VLANs 10 and 40. If you do, the vNIC will go into an error state and will lose link until one of the VLANs is removed. The picture below might help – keep in mind that this diagram is very simplistic and that you can also get an ENM source pin failure with just a single FI:
The above illustration shows a configuration where the user wants to have VLANs 10-50 reach a single server, but this will not work in a DJL2 configuration and will result in ENM Source Pin Failure. Instead, the illustration below would achieve the desired result of VLANs 10-50 reaching the same server, but do not violate the DJL2 rules and would work fine.
Hopefully this helped explain DJL2 a little better and maybe alleviate the ENM Source Pinning error you might be getting.
Thanks for stopping by.
-Jeff
Update: I am running UCSM version 2.2.1d in the lab at present and came across a scenario I need to share. I had vlan 34 on FI-A and vlan 35 on FI-B. I did not need failover for this so each vlan was isolated to a single FI. I set up disjoint L2 correctly and “show vlan” in NXOS mode showed me that it was indeed setup the way I intended. However, any profiles that used vlan-34 on FI-A would throw the dreaded ENM source pin failed error in the profile. I spent several hours verifying and double-checking everything, but no joy. I then ran this command:
FI-A (nxos)# show platform software enm internal info vlandb id 34
I got nothing back. nada, zilch, nothing.
Running this on FI-B, I got what I expected:
FI-B (nxos)# show platform software enm internal info vlandb id 35
vlan_id 35
————-
Designated receiver: Eth1/1
Membership:
Eth1/1
Assuming something was really confused here, I rebotoed FI-A and all was well. If you encounter this, I don’t suggest you reboot the FI (unless you’re like me and it’s in a lab), and I would call TAC instead and let them find a non-disruptive method of correcting the issue. I just want to make a note of it here in case you feel like you got it all right and it still has an issue.
2nd update:
You can run into this problem from the opposite direction if you implement the uplinks first and then create the profiles later. If that’s the case, the profiles will throw an error before being applied saying “not enough resources overall” and “Failed to find any operation uplink port that carries all vlans of the vNICs”