Fibre Channel over Ethernet in Data Center Design

Cisco IT Methods

Introduction
As a completely new facility, the Cisco production data center in Allen, Texas was designed to reflect current industry standards
and to use the latest Cisco technologies for cabling, servers, and networking. One technology in particular, Fibre Channel over
Ethernet (FCoE), has enabled us to reduce cabling and some of the associated equipment for wiring infrastructure, which in turn
reduces costs and simplifies data center management. These design standards and Cisco technologies will be applied in our
future data center builds and retrofits.

Solution
Traditional data center designs require separate cabling systems for an Ethernet LAN and a Fibre Channel SAN. In our Allen data
center, we use Cisco FCoE technology to support both Fibre Channel and Ethernet with a single cabling system. An industrystandard, FCoE carries Fibre Channel frames over Ethernet links.
FCoE gives us ongoing savings in infrastructure and cost because we need to deploy less cabling and fewer I/O adapters and
switches in the data center. Because 40 Gbps Ethernet and 100 Gbps Ethernet support FCoE, we will be able to adopt these
faster networking technologies without rewiring. Additionally, FCoE means that we are able to wire once and support any storage
protocol, which also simplifies our data center management and reduces total cost of ownership (TCO).

Deployment
As shown in Figure 1, we apply the TIA 942 standard when designing the cabling hierarchy for our data centers. This hierarchy
includes the following elements:

Entrance Facility: Typically includes cable rooms, cross-connects, and related infrastructure for connectivity to Access
Providers (AP)

Main Distribution Area (MDA): Provides the backbone for aggregation of all network rooms in a standalone data center
location. The MDA includes cross-connect areas and equipment rows.

Horizontal Distribution Area (HDA): Provides the data center aggregation point for all equipment distribution areas (EDAs);
includes cross-connect areas and equipment rows.

Equipment Distribution Area (EDA): Includes the location for server and storage equipment in the data center halls.

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 1 of 6

Figure 1.

Cabling Hierarchy in Cisco Data Centers

Figure 2 shows how these cabling elements are implemented in the Allen data center.
Figure 2.

Allen Data Center Floor Plan

So what do you

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 2 of 6

The cabling infrastructure of the Allen data center also encompasses the following design principles:

Any-to-any cabling design for network flexibility.

Infrastructure is distributed from overhead, which is one factor in eliminating the need for a raised floor and substations for cable
management.

Limited copper cabling is installed. Multifiber push-on (MPO) and multifiber termination push-on (MTP) optical multimode 3
(OM3) fiber is installed to support Ethernet speeds up to 40 Gbps and 100 Gbps.

Cisco UCS 6200 Series Fabric Interconnects are used as FCoE switches, eliminating the need for dedicated Fibre Channel
switches.

The MPO/MTP parallel optics play a particularly important role in the cabling design.

Parallel Optics
The Allen data center was designed for 10 Gbps optics, leveraging the MTP/MPO fiber assembly, which consists of multiple
strands of fiber in one cable sheath and a MTP/MPO connector at each end. We have standardized on 12strand MTP cables.
Using traditional transmit (Tx) and receive (Rx) duplex fiber optics, we now have six 10 Gbps ports (i.e., two strands of fiber equal
one port; 12 strands divided by 2 = 6) available in each MTP/MPO cable.
For the Fibre Channel LC connectors, the MTP/MPO connector can be broken out to six LC ports using a hydra patch cable (also
called a breakout cable), where one end has an MTP connector and the other end breaks out to six 10 Gbps LC duplex ports.
Although it is possible to map six different devices onto the same physical cable, we dont do this given the complexity involved in
documenting the mapping. Instead, we choose to map one MPO to one LC connection on the breakout cable.
This approach requires careful cable management to organize the large number of cables for each switch as well as the remaining
five unused LC connections. However, we believe this approach will allow simple migration when storage devices support FCoE
as a native protocol in the future. At that point, we will be able to adopt 40 Gbps and 100 Gbps speeds simply by replacing the
Fibre Channel LC connector breakout cable with a MPO-to-MPO cable.
Figure 3 shows our current connectivity design for 10 Gbps networking.
Figure 3.

Connectivity Design for 10 Gbps Networking

We are also using 40 Gbps network connections in some instances today; one MTP equals one 40 Gbps port. In this case, the
parallel optics technology uses four 10 Gbps channels to reach 40 Gbps transmissions, with eight strands of fiber used in the 12strand MTP cable. (Figure 4)

So what do you

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 3 of 6

Figure 4.

Connectivity Design for 40 Gbps Networking

In the future we can migrate to 100 Gbps simply by adding MTP/MPO cabling. When migrating to 100 Gbps via MTP/MPO with
parallel optics, we now need two 12strand MTP/MPO connectors/cables for one 100 Gbps port. As shown in Figure 5, it will take
ten 10 Gbps channels to reach 100 Gbps transmissions, with 20 fibers being used out of the 24 fibers in the cable.
Figure 5.

Connectivity Design for 100 Gbps Networking

So what do you

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 4 of 6

Management
We use the Cisco UCS Manager for managing the FCoE cabling that links the servers to the Cisco UCS 6200 Series Fabric
Interconnects. For managing cabling in the SAN, we use Cisco Prime Data Center Network Manager and the Smart Call Home
feature on our Cisco MDS fabric switches. Both of these capabilities send alerts for Layer 1 and Layer 2 errors, including
excessive bit error rates and cyclic redundancy check (CRC) errors, as well as hardware issues such as card failures.

Service and Support

Cabling support is provided by the Cisco IT data center operations teams as part of their routine activity for data center
management.

Lessons Learned
The MTP/MPO infrastructure cabling generally costs more as an initial investment because MTP/MPO materials cost more to
manufacture. However, because all fiber connectors are preterminated and installed at the factory, we expect to save on labor
costs for installation. In comparison, terminating fiber in the field is costly because it is so labor-intensive.
We also expect to save costs over time because we wont need to remove and upgrade our infrastructure cabling every time we
deploy new network technologies. In addition, preterminated fiber helps us ensure the integrity of the fiber as network
performance increases to 40 Gbps and 100 Gbps, when the loss budget of the fiber and connector becomes a critical factor in
planning the cable layout.

For More Information

For more information on the Cisco products and technologies discussed in this document, visit:

Cisco FCoE technology: www.cisco.com/go/fcoe

Cisco UCS 6200 Series Fabric Interconnects: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6200series-fabric-interconnects/index.html

Cisco UCS Manager: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

Cisco Prime Data Center Network Manager: http://www.cisco.com/c/en/us/products/cloud-systems-management/prime-datacenter-network-manager/index.html

To read additional Cisco IT case studies on a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT
www.cisco.com/go/ciscoit

Note
This publication describes how Cisco has benefited from the deployment of its own products. Many factors may have contributed
to the results and benefits described; Cisco does not guarantee comparable results elsewhere.
CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some
jurisdictions do not allow disclaimer of express or implied warranties, therefore this disclaimer may not apply to you.

So what do you

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 5 of 6

So what do you

2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.

Page 6 of 6