Fabric Extender Technology (FEX) in Nexus

Fabric Extender, the term marketed by Cisco, is basically a port extender as it is referenced in the developing IEEE 802.1Qbh (Bridge Port Extension). The 802.1Qbh standard is specific to control protocol used between the controlling bridge and the port extender, as it is referred in the draft. Supporting standards, also currently being developed like IEEE 802.1Qbg (Edge Virtual Bridging)  and IEEE 802.1Qbc

The Fabric Extender Architecture

The components involved:

Controlling Bridge (Parent Switch) to provide the control and management plane functions. This could be one or two Nexus 5000 or Nexus 7000 switches.
Port Extender which provides the physical port termination. This would be the Nexus 2000 series.
Connecting the FEX to the controlling bridge is done using SFPs over Ethernet fiber.
Encapsulation mechanism to transport frames from the FEX to the controlling bridge.
Control protocols to manage/monitor the FEXs

Cisco calls the encapsulation mechanism used on between the FEX and controlling bridge VN-Tag (previously VN-link). Controlling bridge is IEEE terminology, whereas parent switch is Cisco terminology. The IEEE 802.1Qbh working group was initiated by Cisco in a hope to standardize their VN-Tag technology. VN-Tag provides the capability to differentiate traffic between different host interfaces traversing the fabric uplinks. 

VN- Tag Header

The Fabric Extender Forwarding

A FEX or a Nexus 2000 operate as a remote linecard, but does not support local switching, all forwarding is performed on the parent switch. This is in contrast to most modular switches like the DFCs on Catalyst 6500. One of the reasons this was done was re-usability. By offloading the forwarding and intelligent decisions, the idea Cisco had in mind is that by upgrading the parent switch, the FEX being deployed in larger numbers can remain. Where the DFC on a Catalyst 6500 lives on the line card, the equivalent processing lives on the parent switch, be it the Nexus 7000/5000. Thus upgrading the parent switch upgrades that FEX capability since all it does is encapsulate traffic for identification. In large deployments where the cost of hundreds of FEXs out ways the cost of the Nexus 5000s used, this makes perfect sense. In very small deployments, this reason becomes arguable.

The Fabric Extender Management

It was briefly mentioned before that a parent switch and all its FEXs are treated as a single management device. This is accomplished by a small satellite image running on the FEX. This image is a smaller compatible version of the parent NX-OS image pushed from the parent switch. The parent switch is responsible for this and happens with no user involvement. Same applies to when the parent switch is upgraded, every attached FEX is upgraded during this time too.

The Fabric Extender Operation

Lets take a deep look at the backend operations. There are various interfaces involved:

1. HIF (Host Interface): Are the physical user/host interfaces on the FEX. These interfaces receive normal Ethernet traffic before it is encapsulated with the VN-Tag header. Each HIF interface is assigned a unique VN-Tag ID that is used with the encapsulation.

2. NIF (Network Interface): Physical uplink interfaces on the FEX. These interfaces can only connect back to the parent switch and carries only VN-Tagged traffic.

3. LIF (Logical Interface): Is the logical interface representation of the HIF and its configuration on the parent switch. Forwarding decisions are based on the LIF.

4. VIF (Virtual Interface): Is a logical interface on the FEX. The parent switch assigns/pushes the config of a LIF to the VIF of an associated FEX which is mapped to a physical HIF. This is why replacing a FEX becomes trivial in that the broken FEX is unplugged and the replacement is plugged in.