FNA Installation Guide 8.03

Facebook Network Appliance Deployment | 1
© 2015-2016 Facebook. All rights reserved.

Cisco®, Cisco Nexus®, and the Cisco Logo are registered trademarks of Cisco Systems, Inc.
FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation.
IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other
countries.
Juniper Networks® and Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries.
Linux® is a registered trademark of Linus Torvalds in the United States.
Mac® and Mac OS® are registered trademarks of Apple Inc.
Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company.
Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Thank you for choosing to install the Facebook Network Appliance (FNA)!
Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Service Providers
(ISPs) with an efficient means of delivering static Facebook content from within their existing network. Upon
deployment, an ISP will offload a significant amount of Facebook content from its backbone network and vastly
improve the Facebook user experience.
The FNA kit consists of a Cisco Nexus Top-of-Rack (ToR) switch, four Hewlett Packard ProLiant DL380
servers, and their required interconnects. The hardware is suitable for deployment in data centers, colocation
facilities, and outside plant environments (the industry standard 19-inch form factor allows integration into most
network environments).
This document describes the procedures necessary to install, configure, and operate the FNA.
This section lists equipment required for FNA installation. Ensure that you have received all of the equipment
required for either your SKU1 or SKU2 order (Tables 1 and 2, respectively).
The following table lists equipment that is included with a SKU1 order. If equipment is missing, refer to the
Support section for contact information and seek assistance.
Table 1: FNA SKU1 Provided Equipment
(1) Cisco Nexus 3132Q-X switch2 1RU

(4) HP ProLiant DL380 Generation 9 (Gen9) server3 2RU
(4) 4-Post rail kit N/A
(8) C13-C14 AC Power Cord 2 Meters
(4) Network Cable 3 Meters
HP ProLiant DL380 Generation 9 (Gen9) server user

(1) N/A
manual
(1) Cisco Nexus 3132Q-X switch user manual N/A
(2) SFP+ optical transceiver, LR or SR, (2) per SKU4 N/A
1See Appendix I: Hardware Dimensions and Weight for more dimension details.
2Hereafter referred to as Nexus switch.
3Hereafter referred to as ProLiant DL380 server.
4Caution: SFP+ optical transceivers are sensitive to static electricity.

The following table shows equipment that is included a SKU2 order. If equipment is missing, refer to the
Support section for contact information and seek assistance.
Table 2: FNA SKU2 Provided Equipment
(4) HP ProLiant DL380 Generation 8 (Gen9) server2 2RU1

(4) 4-Post rail kit N/A
(8) C13-C14 power cable 2 Meters
(4) Network cable 3 Meters
HP ProLiant DL380p Generation 8 (Gen8) server

(4) N/A
user manual
(1) Text configuration file N/A
(2) SFP+ optical transceiver, LR or SR, (2) per SKU3 N/A
1See Appendix I: Hardware Dimensions and Weight for more dimension details.
3Hereafter referred to as ProLiant DL380 server.
3Caution: SFP+ optical transceivers are sensitive to static electricity.
NOTE: SKU2 does not include a Cisco Nexus switch. However, the order includes SFP+ optical transceivers for
additional uplink capacity. Use the existing switch at your FNA deployment.
NOTE: The quantity of servers you receive depends on your current and forecasted Facebook traffic volume.
Servers are shipped in groups of four. The equipment will arrive at your facility in one shipment. IBM (FNA’s
logistics partner) will contact you with shipping information.
This section shows additional equipment required for the FNA installation that is not included with a SKU order.
Table 3: FNA SKU1 and SKU2 Equipment Required but Not Included
(2) Fiber Patch Cables from Cisco Nexus 3132Q-X switch to router1,2
1The FNA accepts both SR and LR optical transceivers. Facebook provides the most optimal
optical transceiver type corresponding to the type you currently use to connect your routers.
2
Fiber connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the
router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.
This section describes how to install equipment to build an FNA cluster. After installing the equipment, you will
need to configure the equipment to work in your facility. This information is provided in section 4 Network
Configuration.
Use the pre-installation checklist to confirm that your environment has the resources required to install the FNA
into your existing network:
Table 4: FNA Pre-Install Checklist
n x 10 Gigabit (10Gb) available ports on ISP switch or router1

n available SFP+ (or compatible) optics of same type - Long Range (LR)
or Short Range (SR)1
n available fiber patch cables (multi-mode or single-mode)1

n available input power connections2
Power connections are divided evenly between two different power
sources
Sufficient power (8kW) for each Stock Keeping Unit (SKU)
Sufficient rack space is available3
1
Where n = 2 for every FNA SKU deployed.
2
Where n = 2 times the number of devices (e.g., SKU1 is (5) devices, so (10) input power
connections are required).
3
For initial deployment (SKU1), 9RU (nine rack unit) is required. For additional capacity
(SKU2), 8RU is required.
Install the ProLiant DL380 servers into a compatible server rack.

Utilize the manufacturer instructions for the model included in your order:
DL380P Gen8: Server User Guide (Edition: 5)
DL380 Gen9: Server User Guide (Edition: 3)
Connect C13-C14 AC power cables from each server to your power distribution units (PDUs).
NOTE: If you install the servers in a 2-post rack, you must provide the proper 4-post to 2-post conversion
hardware.
Install the Cisco Nexus switch in the rack.

Utilize the instructions in the Cisco Nexus 3000 Series Hardware Installation Guide (April
22,2015).
Connect power cables from the switch to your PDUs.
Using the supplied network cables, connect each ProLiant DL380 server from network interface
controller (NIC) port 2 to their respective QSFP+ ports on the Cisco Nexus switch (see and 3).
Begin with the server at the highest location in the rack (labeled fna001) and work your way to
the bottom-most server. Connect fna001 to port 32 on the switch, fna002 to port 31, and so on.
See Figure 30 in Appendix F: Additional Installation Images for a reference on server
connectivity order.
NOTE: Each ProLiant DL380 server and QSFP+ connector have a round green sticker to show the correct
orientation. Ensure both green dots on each QSFP+ connector and server chassis are facing upward.
NOTE: The Cisco Nexus switch QSFP+ ports 9 ~ 12 are not used and should not be utilized for any FNA or
non-FNA equipment. Each Cisco Nexus switch supports up to twenty servers.
Connect the Cisco Nexus switch to your router.

Use SFP+ ports 1 and 2 for the first (4) servers (see Figure 4). These ports are pre-configured
for the Link Aggregation Control Protocol (LACP). See Appendix C: Link
Aggregation/Link Aggregation Control Protocol (LACP) Configuration for more
information.
NOTE: The Cisco Nexus’ (4) SFP+ ports and QSFP+ port 1 cannot be utilized at the same time. These ports are
internally multiplexed. See Figure 30 for more information
Power on all FNA hardware.

Press the power buttons on the ProLiant DL380 servers and power on any interfacing systems
(i.e., PDUs, existing router, etc.).
NOTE: The Cisco Nexus switch does not have power buttons. They receive power when connected to a PDU with
adequate power.
Check and record the LED states for each PSU, NIC, and Solid State Drive (SSD) for each
ProLiant DL380 server in the cluster. For LED definitions, refer to the LED Definitions section in
Appendix B: Troubleshooting.
Check the light levels (for deployments utilizing a Cisco Nexus switch, refer to Measuring the Optical
Level through IOS, Document ID: 16150). Light levels need to be between -2dB and -7dB for proper
operation. Light levels outside of this range may be indicative of a connectivity issue. If you find the
levels out of range, see Appendix B: Troubleshooting.
This completes the FNA hardware installation procedure. Equipment installation should look similar
to Figure 7.
NOTE: If you are adding additional equipment to your initial FNA installation, see section 6 System Upgrades.
This section describes the procedure to configure the FNA to peer with Facebook’s network.
Connect a VGA monitor and USB keyboard to one of the FNA servers. Upon server boot-up, you will
see the FNA Main Screen in Figure 8:
NOTE: The servers should be preconfigured with IP addresses. If this is not the case, you will need to create and use
a custom USB boot disk. See Section 9.1 Creating a Using a USB Boot Disk.
Configure the point-to-point connection. This connection will be configured on the link-aggregation
control interface (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol
(LACP) Configuration for additional configuration details.
Untag the Virtual LAN (VLAN) link-aggregation interface.
Set a static route to the allocated subnets via the link-aggregation interface.
NOTE: Facebook requires that the FNA have internet connectivity before BGP peering can be established. The
BGP peering session does not need to be routing before the FNA has internet connectivity.
Configure BGP.
Use the IPv4 and IPv6 addresses originally supplied in the FNA order form.
Enable Graceful Restart (GR). See Appendix D: BGP Configuration Details for configuration
information.
Enable External BGP (EBGP) Multi-hop.
The Autonomous System Number (ASN) that you will be peering with is 63293.
Configure your router’s BGP settings. See Appendix D: BGP Configuration Details for
configuration information.
NOTE: Once the router’s BGP settings have been properly configured, a latency based routing test will be initiated to
optimize routing paths. It is required that all FNA servable netblocks be announced for the purpose of this test. This
process takes about twenty-four hours to complete. BGP establishment is not required during this time. See
section 4.4 Latency Based Routing for more information.
NOTE: The BGP peering session is intended to be a router-to-server configuration and not a switch-to-switch
configuration.
NOTE: Multi-hop sessions are supported.
Facebook will begin initializing the FNA system for operation.

The FNA cluster will begin to download application data. The time to finish the download
depends on your connection speed.
Within about two business days, BGP peering will be established:
The connection will be initiated by one of the FNA servers.
The connection will be coming from the 16th IP address in each allocated subnet
(e.g. 192.168.1.16, fe80::10).
FNA will not advertise any routes/prefixes.
NOTE: The remote software delivery session will begin only after all of the FNA hardware has been powered on, the
point-to-point connection has been established, and routing to the internet is enabled.
This completes the network configuration of the FNA.

Contact IBMFBSupport@us.ibm.com to inform them that you have completed the installation.
FNA applies inbound filtering to reject the following address spaces on BGP peering sessions to ensure
successful BGP establishment:
 Prefixes smaller than /24 (255 IPs)
 Default route
 Bogon prefixes
 RFC 1918
 Facebook’s IP prefixes
The FNA utilizes a latency based routing test to optimize traffic routing. For the purpose of this test, it is critical
that all netblocks that can be served be announced to the FNA. Note that prefixes announced to the FNA
only implies those prefixes are eligible to serve Facebook traffic.
The latency based routing test evaluates the round-trip latency of your network. This is accomplished by
sending a small sample of static content from the netblock population you announced. Initially, only a small
sample set is sent to the FNA cluster. The sample volume will ramp up during this test. This makes it possible
for Facebook to observe and construct the most optimal routing paths for your FNA deployment.
The accuracy of the latency based routing service is dependent on the percentage of FNA servable netblocks
announced. Providing a large percentage of your FNA servable netblock population allows the service to
construct more optimal (low-latency) routing paths. A small percentage of netblocks announced (i.e., a test
netblock) will not allow the test to function as intended and traffic may be routed in a seemingly random course.
It is critical that every FNA servable netblock is provided to Facebook to optimize routing paths for
your FNA deployment.
Please complete this checklist after you have installed the FNA to verify proper installation.
Table 5: FNA Installation Completion Checklist
All ProLiant DL380 servers in the cluster are powered on.

Each power supply of each ProLiant DL380 server indicates an ON,
Steady Green state.
The Life (L) and Status (S) LEDs for each SSD in each ProLiant DL380
server indicate an On, Steady Green state.
All accompanying LEDs for the connected fiber uplink SFP+ ports are
green.
All network cables are seated properly to each ProLiant DL380 server.
The port 2 (P2) Link LED (LNK) on each NIC in the cluster indicates an
On, Steady Green state.
Light level readings for all uplinks are between -2dB and -7dB
This section describes how to add additional capacity your original FNA deployment.
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause
service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and
Monitoring for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the additional ProLiant DL380 servers, connect the peripheral cables (power and network), and
power on the servers. Utilize section 3 Hardware Installation for reference.
Connect (2) additional 10G fiber uplink cables from port 3 and port 4 on the Cisco Nexus switch to the
same router (or switch) where the initial two 10G fiber uplinks are connected (see Figure 9).
Update the Link Aggregation (LAG) configuration on the router (or switch) to accommodate the
additional interfaces.
NOTE: All cluster augments are implemented with an increment of four ProLiant DL380 servers.
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause
service interruption. If this is the case, it is recommended to drain network traffic first. See the Maintenance section for
instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the ProLiant DL380 servers, connect the peripheral cables (power and network), and power on
the additional servers. Utilize the Hardware Installation section for reference.
Connect the provided QSFP+ to 4x SFP+ breakout cable assembly to port 2 on the Cisco Nexus
switch. See Figure 10 for breakout cable interface location.1
If you are upgrading from a (16) server to a (20) server cluster, connect the 2nd provided
QSFP to 4x SFP+ breakout cable to port 3 on the Cisco Nexus switch.
Connect (2) additional 10G fiber uplink cables for each group of (4) servers from the Cisco Nexus
switch to the same router (or switch) where the initial 2x 10G fiber uplinks are connected.
Update the LAG configuration on the router (or switch) to accommodate the additional interfaces.
1
NOTE: The provided QSFP to 4x SFP+ breakout cable assembly only supports LR single-mode connections.
NOTE: Refer to Figure 32: System Architecture Diagram for a diagram of a (20) server FNA cluster.
This section provides a reference for best practices regarding augmentations. This growth path optimizes
for deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).
NOTE: When building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new
rack and maintain equal size clusters (as depicted in growth path 3 and 4).
When performing either singular or recurring maintenance jobs, email fna-ops@fb.com to report the planned
maintenance.
Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This
procedure describes how to properly drain traffic from the cluster:
IMPORTANT: This procedure requires withdrawing prefixes. It is important that the BGP peering session is
maintained while withdrawing prefixes.
Withdraw prefixes.
While maintaining the BGP peering session, begin withdrawing BGP prefixes.
Traffic will begin to drain.
Traffic will fully drain in less than one hour.
You should shut down the FNA only if it is absolutely required and traffic has been properly drained from the
system (see 7.2 Draining Network Traffic). This should be a rare occurrence. To shut down the FNA, follow
the procedure outlined below or contact fna-ops@fb.com for assistance:
Connect a monitor and keyboard to the server you want to power off.
Use the arrow keys navigate to the Shut Down text.
Press the [Enter] key. The FNA will power off.
Facebook regularly monitors internet reachability from various simulated endpoints. When reachability issues
are detected on these Virtual IPs (VIPs), Facebook may drain traffic from an FNA cluster. This measure
ensures that Facebook maintains a high quality of service for the Facebook platform.
The Terms of Service agreements that you signed when you installed the FNA in your facility included specific
language pertaining to the disposal of storage media. That language is replicated here for your convenience.
Your ISP has agreed to:
 Only deal with and process Edge user data in compliance with, and subject to, the instructions received
from Edge and will not use or process the Edge User Data for any other purpose whatsoever.
 Take custody of and secure, and support the destruction of, all server storage media (example: hard
disk, solid state disks, flash memory cards, etc.) that are uninstalled from the servers, and provide
secure handling, storage, packing, and shipping of the storage media per IBM provided instructions and
at IBM's expense to an IBM designated asset recovery center for destruction.
FNA equipment comes preconfigured with the base software installation and IP configuration that was
specified in the FNA kit order. In some instances, the IP configuration may need to be changed (such as
configuration changes made between processing an order and receiving the kit). If this is the case, you will
need to reconfigure the equipment by creating and using a USB Boot Disk.
To create a USB boot disk with to change the system configuration, follow these steps:
Log into the FNA portal.
Click the Caching tab and Manage FNA.
Locate the name of the FNA kit that needs to be reconfigured under the Installed Kits pane. Verify that
the FNA IP Prefix and every P2P address are correct.
If any information is incorrect, stop and inform the FNA team of the changes that need to be
made (see section 11 Support). Only continue to the next step after the information has
been corrected.
Locate the USB Boot Disk item under the Installed Kits pane. Click the Disk Image and Kit Config
links to begin downloading (see Figure 13).
Connect a blank USB flash drive into a computer running either Windows, MacOS, or Linux.
Instructions for creating a USB boot disk for each of these operating systems is provided in this section.
Create the USB boot disk. Utilize the instructions for the appropriate operating system:
WARNING: This drive will be completely overwritten.
Instructions for Windows Operating Systems
Download and install Win32 Disk Imager and 7-Zip.

Use 7-Zip to extract the disk image (see Figure 14).
Use Win32 Disk Imager to write the Disk Image to the appropriate USB drive.
Instructions for Mac OS
Run diskutil list to find the mount point of the USB drive.
Diskutil list
/dev/disk0 (internal, physical):
/dev/disk1 (internal, virtual):
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *15.5 GB disk2
1: DOS_FAT_32 UNTITLED 15.5 GB disk2s1
Unlocked Encrypted
Confirm you have selected the correct mounting point by running diskutil info disk# and
verifying the drive name, size, and protocol.
Run diskutil unmountDisk <disk#> to unmount the USB drive.
Diskutil unmountDisk /dev/disk2

Unmount of all volumes on disk2 was successful
Run gunzip to unzip the disk image.
WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before
continuing.
gunzip –d ~/Downloads/caelus.img.gz
Run dd to write the disk image to the USB drive.
Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync

67+1 records in
67+1 records out
268435456 bytes transferred in 183.506575 secs (1462811 bytes/sec)
Instructions for Linux Operating Systems
Run mount or blkid to find the mount point of the USB drive.
Run umount to unmount the USB drive.
Run gunzip to unzip the disk image:
gunzip –d ~/Downloads/caelus.img.gz
Run dd to write the disk image to the USB drive.
WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before
continuing.
Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync

67+1 records in
67+1 records out
268435456 bytes transferred in 183.506575 secs (1462811 bytes/sec)
Configure the USB boot disk by dragging the FNA kit config JSON file
(<Kit Name>_config.json) to the CONFIG partition on the USB drive.
Connect the USB drive to the USB port on the server labeled ‘fna001’ (see Figure 17).
NOTE: The USB boot disk will configure the entire FNA system. You do not need to connect the USB drive into the
remaining servers.
Reboot ‘fna001’ while the USB drive is connected.

The server will automatically boot from the USB drive. If it does not automatically boot from the
USB drive, access the BIOS settings and set the USB drive as the first boot option.
The remainder of the configuration process is automated. You do not need to interact with the FNA.
You can connect a VGA monitor into the VGA port of ‘fna001’ to watch the configuration
process.
After a few minutes, the host will boot from the USB boot disk and you will see the
FreeBSD boot screen (Figure 18):
The USB boot disk will attempt to access the internet and download software
necessary for the FNA to operate.
NOTE: This process will take a significant amount of time.
If any errors appear on the monitor, take a picture or note the errors seen. Contact the
FNA team (see the Support section) and submit these errors.
The network partner portal is the central hub for requesting

additional FNA kits, reviewing deployment information and metrics,
and troubleshooting issues you may encounter with your
deployment.
URL: https://partners.facebook.com/network/
Logging into the NPP is as easy as logging into Facebook. If you

have an existing Facebook account associated with your work
email address, the portal will use that account to log in. If you have a Facebook account associated with your
personal email address, you must associate your work email address with your Facebook account.
If you don't have a Facebook account, you need to create one. Go to http://www.facebook.com and follow the
instructions for creating an account. To associate your work email address with your Facebook account,
perform the following steps:
Click the Menu icon ( ) located at the top right of a Facebook page. Click Settings.
Click Email.
Click Add another email or mobile number.
Enter your work email address in the New Email field. Click Add.
Re-enter your Facebook password when prompted.
A confirmation code will be sent to verify the email address. Click Confirm and enter the code. Click
OK.
Click Save Changes.

The portal allows you to authorize others as administrators and set their permissions.
To add a new administrator, perform the following steps:
Click the Settings icon ( ) located at the top right of the portal page.
Enter the name and email of the user you wish to make an administrator.
Click Add.
An invite will be sent to the user's email address. Their permission status will remain "pending" until
they accept the invite.
NOTE: Admin access grants access and control to all settings and data within the portal.
NOTE: You can also set the user's permissions to "Read-Only Access". This gives the user complete access to all
portal data, but does not allow access to any settings or controls. To set a user's permission to read-only access,
follow the steps in the Modifying Permissions section and select Read-Only Access.
To modify an existing user's permissions, perform the following steps:
Click the Settings icon ( ).
Click the Edit link on the user's card.
Click the chevron icon ( ) below the user's email address. Select either Admin Access or Read-Only
Access.
Click Done.
To remove a user from the portal, perform the following steps:
Locate the user you would like to remove.
Click the Remove link on the user's card.
Enter your Facebook password if prompted.

FNA deployments need to connect to your network. To enable this connection, you need to provide your ISP's
Autonomous System Number (ASN). Facebook requires this data to estimate the number of servers your FNA
deployment will require. Facebook also uses this information during the provisioning process to properly route
traffic to the FNA.
To add your ASN, follow these steps:
Click Autonomous Systems located in the left menu.
Enter your ASN and AS Name.
Enter your ISP's ASN and AS Name.
Click Add.
To remove an ASN, follow these steps:
Click Autonomous Systems located in the left menu.
Click the Remove link next to the ASN entry you wish to remove.
NOTE: You can enter multiple ASNs. All ASNs should be kept up-to-date.
NOTE: If the ASN entry has a yellow clock icon ( ), it is currently being reviewed by Facebook.
Requesting an FNA kit is a multi-step process that takes several weeks to complete. Facebook evaluates
submissions for qualified ISPs that can utilize an FNA. If your application is approved to receive an FNA,
Facebook will send an FNA kit to the designated ISP address.
To request an FNA kit, follow these steps:
Click the Support link at the top of the page. The Support page will appear.
Click the Request a New Kit hyperlink.
Enter the deployment country where the FNA will be installed. Click Next.
Select the Type of Kit.
Review the information and click the Begin Order Form button. An online order form will appear in a
new window or tab. Use table 6 as a guideline for order form field entry:
Provide contact information for the This can be the engineering contact.
Shipping primary contact that will receive
Contact
FNA shipments.
Provide contact information for the This can be the engineering contact.
primary contact that will install or
Installation
lead installation of the FNA
Contact
equipment.
Provide contact information for the

Engineering primary engineering contact that
Contact can answer technical questions
about the network.
Shipping Provide the location where FNA
Address equipment will be sent.
Site Installation Provide the location where FNA
Address equipment will be installed.
Special Provide any notable unique shipping This is an optional field.

Shipping instructions.
Instructions
Rack Name Enter a name for the FNA rack. This is an optional field.
 Provide power source

requirements for the FNA
deployment (AC or DC).
Facility /  Provide the type of optical
Deployment transceiver your deployment
Technical would require – either long
Information range (LR) or short range
(SR)
 Provide the power outlet
type your deployment would
require.
 Provide the primary peer

Networking and ASN.
IP Information  Provide any additional ASNs
or communities to be served.
Provide IPv4 allocation for caches,  Gateway and Cache Peer-
gateway address and network, to-Peer (P2P) IP addresses
cache address and network, and must be within the same
IPv4 Settings Border Gateway Protocol (BGP) subnet.
peer address.
 BGP address / network
should not overlap with IP
allocation for caches.
Provide IPv6 allocation for caches,  Gateway and Cache Peer-
gateway address and network, to-Peer (P2P) IP addresses
cache address and network, and must be within the same
IPv6 Settings BGP peer address. subnet.
 BGP address / network
should not overlap with IP
allocation for caches.
Fill out and submit any required order forms located in the Required Forms section near the top of the
order form. Forms that require submission are denoted with the "X" icon ( ). Forms that are currently
pending review are denoted with the "exclamation" icon ( ).
Click Submit if you have completed the order form or Save Draft if you plan to complete the order form
at a later time.
If Facebook selects you to participate in the FNA program, International Business Machines (IBM) will contact
the shipping contact you indicated with further details and will make arrangements to deliver FNA equipment.
IBM is Facebook's logistics partner for FNA equipment.
Once you have installed your FNA equipment and it is brought online, access the Manage FNA to:
 Review Installed Kit Information
 View Traffic Information
 Review Contact, Exchange, and Facility Information
 View Peering Sessions
The installed kit information page allows you to view the number of servers per cluster, IPv4 and IPv6 network
addresses, advertised prefixes, and disk image. To view this page, follow these steps:
Click the Caching tab located at the top of the portal.
Click Manage FNA.
This page displays FNA information sorted by cluster name. You can also request a new kit by clicking
Request a New Kit in the top-right corner.
FNA traffic metrics are located in the Insights page. This page allow you to observe cache-fill and retransmit
information sorted by FNA cluster name. To view traffic information, follow these steps:
Click the Caching tab located at the top of the portal.
Click Insights.
The FNA Traffic information will appear. Each plot represents either "To Subscribers" or cache fill for
each FNA cluster installed.
To view traffic retransmission information, click the button at the top left of the graph. Click FNA
Retransmits. This page will show traffic retransmission as a percentage of the total FNA traffic.
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
Click the Peering tab.
Click Peering Summary (this should load by default).
This page allows you to review your contacts, exchanges, and facilities information.
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
Click the Peering tab.
Click Peering Sessions.
This page allows you to view peering session details, including ASN, location, peer type, link speed,
router address, peer port, and status.
If you have issues receiving equipment or installation and configuration issues, use Table 7 to contact the most
appropriate support team for assistance.
Table 7: Support and Contact Information
Email:  Network Support is

fna-ops@fb.com Required
Facebook
Network  BGP Peering Issues
Appliance  Assistance with
Draining Network
Traffic
Email:  Hardware Support is  Notification of

IBMFBSupport@us.ibm.com Required Hardware
 Hardware is missing Installation
from order. Provide the Completion
IBM following:
o Missing Item(s)
o ISP Name
o ISP Address
o Equipment
Received Date
o Your Contact
Information
NOTE: In some cases, it may be necessary to return the equipment. It is recommended to keep the original
packaging material for two months.
What type of data is being stored on FNA?
An FNA cluster will cache the following types of data:
 Static Facebook content

 Static Instagram content
FNA does not cache WhatsApp or Oculus content.
How much traffic can the FNA cluster serve?
The amount of traffic that the FNA cluster may serve varies based on:
 The size of the cluster and the number of available machines

 The amount of prefixes being advertised / number of users
 The RTT latency to advertised prefixes
 Content type variance: profile pictures, large images, and video
Each server in the cluster is capable of delivering ~4Gbps of traffic. However, depending on the deployment
region, service type (e.g., mobile vs. fixed line), and variance in content type (e.g., images and video), the FNA
cluster may deliver less overall traffic (~3Gbps is expected).
Generally, a high variance in content type will yield lower traffic throughput and low content type variance will
yield higher traffic throughput.
The installation procedure indicates installing the supplied switch directly to our existing router. Our
facility aggregates to our existing switches before connecting to a router. How does this modify the
installation process?
FNA provides aggregate connections from the cache servers directly to the router. However, if you connect the
supplied FNA switch to an intermediary aggregation switch, this configuration is fine and the logical topology
will remain the same.
Is FNA Carrier-Grade NAT (CGNAT) compatible?
Yes. FNA is compatible with CGNAT for the RFC 1918 and RFC 6598 address spaces.
I’ve advertised a certain prefix but it doesn’t seem to be served by the FNA. Why is this happening?
Advertising a prefix marks means it as eligible to be served from the FNA, but does not guarantee it will be
served by FNA. There are a few reasons why the FNA may not be serving a prefix which has been advertised
to it:
 If it has been less than 24 hours since the FNA has been online or since you have started advertising
the prefix (see Latency Based Routing). Facebook may not have collected enough latency samples in
order to decide whether the FNA cluster will serve the prefix.
 If Facebook is observing less latency to the prefix from a nearby Point of Presence (PoP), then the PoP
will be preferred for serving the prefix.
 If the system resources (e.g. CPU utilization) have been exhausted.
Which routes should I advertise towards Facebook and how does failover occur between FNA’s?
In most cases, the best option is to announce all customer prefixes. In cases where there may be congestion
from region to region, the best practice is to announce local routes only.
Why should I advertise all of my prefixes?
Facebook’s content routing platform measures Round Trip Time (RTT) latency between each Point of
Presence (PoP) and FNA deployment to all prefixes being advertised by a peer ISP. Based on this
measurement and a number of other factors (such as available system resources and link capacity), a client is
routed to the most optimal location for making content requests.
This section provides troubleshooting resources for events that may be encountered during installation or
operation.
Each ProLiant DL380 server and Nexus switch has LEDs that can be used to diagnose the FNA for issues.
These LEDs also useful for verifying whether the FNA was installed correctly and for receiving quick technical
support.
The server is powered off or the power

Off ____________________
supply has failed.
On, Steady Green ____________________ The power supply is on.
Activity
On, Blinking Green ____________________ There is data activity on the card.
(A)
The drive has sufficient life for
On, Steady Green ____________________
programming and erasing the memory.
The drive has approximately 10% or less
On, Steady Yellow ____________________ life remaining for programming and
Life (L)
erasing the memory.
The drive no longer has program or
On, Steady Red ____________________ erase cycles. Data can be read, but
writing data is at risk.
On, Steady Green ____________________ The drive is operating normally.
A network administrator is toggling the
On, Blinking Green ____________________
LED to locate the server.
Drive warning. One of the following has
Status
occurred:
(S)
On, Steady Yellow ____________________  The maximum temperature threshold
has been reached.
 Another issue has occurred.
On, Blinking Red ____________________ A firmware fault code has been issued.
Fatal drive warning. One of the following

has occurred:
 Drive failure.
 Drive has exceeded its maximum
Status temperature threshold.
(S) On, Steady Red ____________________  The RAID volume has failed.
 The RAID volume is not configured.
 The drive’s internal backup power
supply has failed.
 Another issue has occurred.
Activity
On, Blinking Green ____________________ There is network activity.
(ACT)
Link A connection exists between the server
On, Steady Green ____________________
(LNK) and the network.
Power
Supply On, Steady Green ____________________ The power supply is on.
Fan On, Steady Green ____________________ The fan is powered on.
On, Steady Green ____________________ The system is operating normally.
System The system has posted an environmental

On, Steady Orange ____________________
Status issue code.
The system has posted an environmental

On, Steady Red ____________________
issue code.
In cases where light levels are outside the normal range (between -2dB and -7dB), utilize Table 9 to resolve
the issue.
Table 9: Signal Level Troubleshooting
The signal is too strong. This Fix with your normal operating procedures
Above -2 could be caused by many for strong signals.
reasons.
None (Normal Operating Range) None (Normal Operating Range)

-2 to -7
 A degraded fiber optic cable  Disconnect, clean, and reconnect the

is installed fiber optic cable(s) and/or SFP+ optical
-8 to -20  A degraded SFP+ optical transceiver(s).
transceiver is installed
If this does not resolve the issue, either
 A fiber optic cable has a dirty replace TX/RX hardware or solve with your
connection normal fiber replacement procedures.
 A fiber optic cable is bent  Inspect the fiber optic cable(s) along its
 Network connectivity has not length. Gently roll the fiber if you
been established observe bends or kinks.
-21 to -40
 All cables are connected to  Check that the port connection is
their proper ports, but the enabled
port(s) is disabled  Ensure the cables are connected to the
 All ports are enabled, but a proper port interface. See Figure 10 for
cable is connected to the port interfaces.
incorrect port
The following are examples of LACP configurations for the most common Cisco IOS and Juniper Junos
routers. Apply the lines of code (below) that are appropriate to your specific router model.
The following configuration settings should be applied to a Juniper Junos router for FNA LACP compatibility:
ae1 {
aggregated-ether-options {
lacp {
active;
periodic fast;
}
}
unit 0 {
description FACEBOOK-FNA;
family inet {
address <address/mask>;
}
family inet6 {
address <address/mask>;
}
}
}
xe-0/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
xe-1/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
To confirm whether the LACP settings were properly applied, run the following commands:
show lacp interfaces xe-4/3/3
show interfaces ae3 | grep Physical
If the following output appears, the LACP settings were properly configured for the Juniper Junos router:
show lacp interfaces xe-4/3/3
Aggregated interface: ae4

LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-4/3/3 Actor No No Yes Yes Yes Yes Fast Active
xe-4/3/3 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-4/3/3 Current Fast periodic Collecting distributing
show interfaces ae4 | grep Physical
Physical interface: ae4, Enabled, Physical link is Up
The following configuration settings should be applied to a Cisco IOS-XR router for FNA LACP compatibility:
!
interface Bundle-Ether1
description FACEBOOK-FNA
ipv4 address <address/mask>
ipv6 address <address/mask>
!
interface TenGigE0/0/0/0
bundle id 1 mode active
lacp period short
!
interface TenGigE0/1/0/0
bundle id 1 mode active
lacp period short
!
To confirm whether the Cisco IOS-XR LACP was properly configured, run the following command:
show port-channel sum interface port-channel 911
If the following output appears, the LACP settings were properly configured on the Cisco IOS-XR router:
show port-channel sum interface port-channel 911

Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
911 Po911(RU) Eth LACP Eth18/9(P) Eth18/10(P) Eth18/11(P)
Eth18/12(P)
The following are examples of BGP peering code for the most common Cisco OOS and Juniper Junos routers.
These examples allow you to configure BGP peering sessions to send Facebook traffic to the FNA cluster.
Contact Facebook (fna-ops@fb.com) if you encounter problems when setting up BGP peering.
NOTE: After the FNA cluster has network access, it will begin downloading application data that enables the FNA
to operate. All application data must be installed before the BPG session is established. The application
data may take some time to download.
The following router BGP configuration should be used for FNA deployments that utilize a Juniper Junos
router:
protocols {
bgp {
group FACEBOOK-FNA-IPv4 {
neighbor <FNA IPv4 address> {
multihop;
family inet {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
group FACEBOOK-FNA-IPv6 {
neighbor <FNA IPv6 address> {
multihop;
family inet6 {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
}
}
policy-options {
policy-statement FACEBOOK-FNA-EXPORT {
term export {
from {
prefix-list FACEBOOK-FNA;
}
then accept;
}
term reject {
then reject;
}
To enable BGP Graceful Restart on a Juniper Junos router, perform the following procedure:
Utilize Juniper’s Configuring Graceful Restart Options for BGP reference document.
Navigate to the appropriate hierarchy level and change the disable statement to enable.
Evaluate whether GR was successfully negotiated by running the following command:
show bgp neighbor <IP Address> | grep Options
Locate the applicable Junos device and evaluate whether GracefulResart is within the Options
output:
show bgp neighbor <IP Address> | grep Options
Options: <Preference AdvertiseInactive GracefulRestart LogUpDown AddressFamily

PrefixLimit Refresh>
Options: <MtuDiscovery>
If the Options output string contains GracefulRestart, GR was successfully negotiated.

REMINDER: GR is disabled on Junos devices by default. Enabling GR is required for the FNA deployment.
NOTE: Enabling Graceful Restart on the Junos device does not imply that it is in use. Both sides of the session must
support and negotiate Graceful Restart.
The following router BGP configuration should be used for FNA deployments that utilize a Cisco IOX-XR
routers:
!
router bgp x
!
address-family ipv4 unicast
!
!
neighbor <FNA IPv4 address>
remote-as 63293
route-policy FACEBOOK-FNA-EXPORT out
!
neighbor <FNA IPv6 address>
remote-as 63293
route-policy FACEBOOK-FNA-EXPORT out
!
To enable BGP Graceful Restart on a Cisco NX-OS router, perform the following procedure:
Utilize Cisco’s Configuring a Graceful Restart procedure.
show ip bgp neighbors <IP Address> | grep "Graceful Restart"
If the output string ‘Graceful Restart capability: advertised received’ appears, GR was
successfully negotiated.
The following diagram demonstrates the route of a Facebook request to your ISP both before and after the
FNA deployment:
REMINDER: QSFP+ port 1 is internally multiplexed with the (4) SFP+ uplink ports. The (4) SFP+ ports and the QSFP+
breakout port cannot be utilized at the same time.
The following diagram shows the topology of a (20) server FNA installation:
NOTE: This diagram depicts an FNA cluster with the maximum number of supported servers per switch.
However, when building an FNA cluster beyond eight servers, it is best practice to install the additional servers in
a new rack and maintain equal size clusters (as demonstrated in Section 6.2 Growth Path).
Table 10: FNA System Benchmarks
Expected Bandwidth Offload 65%

Maximum Sustained Throughput 4Gbps per cluster
Table 11: AC Power and Temperature Requirements
Input Current (by PSU) 8.9A @100V

4.3A @ 200V
Steady State Power 750W @ 100 V to 120V
750W @ 200V to 240V
Frequency 50-60Hz
Dual Path Power Required
Dual PDUs Required
Operating Temperature Range 10°C to 35°C
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optics @ 100% load; (2) PSUs
Maximum Power 340W
Input Voltage by PSU 100V to 240V
Frequency 50-60Hz
Efficiency 89% to 91% @ 220V
Dual PDUs Required

Typical Heat Dissipation 717 BTU/hr (with Twinax at 100% load)
819 BTU/hr (with SR4 optics at 100% load)
Maximum Heat Dissipation 1160 BTU/hr
Cordset Rating 10A @ 250V
SKU1 ~ 800W (per SKU)

Table 12: DC Power and Temperature Requirements
Input Voltage Range (V DC) 36-72V DC

Nominal Input Voltage (V DC) 36/48/72V DC
Maximum Rated Output 750W @ 36V DC / 48V DC / 72V DC
Wattage Rating (W)=
23A @ 36V DC
Nominal Input Current (A) 17A @ 48V DC
11A @ 72V DC
Maximum Rated Input Wattage 815W @ 36V DC
Rating (Watts) 805W @ 48V DC
795W @ 72V DC
Maximum British Thermal Unit 2780 BTU-Hr @ 36 V DC
Rating (BTU-Hr) 2740 BTU-Hr @ 48 V DC
2720 BTU-Hr @ 72 V DC
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optical transceiver @ 100% load;
(2) PSUs
Maximum Power 340W
Input Voltage by PSU –40V DC to –72V DC
Maximum Current 33A
Efficiency 85% to 88%
Power Supply Efficiency 89% to 91% @ 220V
Dual PDUs Required

Typical Heat Dissipation 717 BTU/hr (with Twinax @ 100% load)
819 BTU/hr (with SR4 optical transceiver @ 100%
load)
Maximum Heat Dissipation 1160 BTU/hr
Table 13: Hardware Dimensions and Weight
Dimensions 8.74 x 44.55 x 69.85 cm (3.44 x 17.54 x 27.50 in)

(Height x Width x Depth)
Weight (Approximate)

Weight (Approximate)

Weight 9.3 kg (21.5 lb)

Weight (Approximate) 9.3 kg (21.5 lb)
Length 2 Meters
Plug SS10A
Connector HS10S
American Wire Gauge (AWG) 30

Length 3 Meters
Dimensions 8.5 x 13.4 x 56.5 mm

Weight 75 grams (2.64 ounce)

FNA Installation Guide 8.03

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FNA Installation Guide 8.03

Uploaded by

Copyright:

Available Formats

Facebook Network Appliance Deployment | 1

© 2015-2016 Facebook. All rights reserved.

Linux® is a registered trademark of Linus Torvalds in the United States.

Mac® and Mac OS® are registered trademarks of Apple Inc.

Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company.

Table 1: FNA SKU1 Provided Equipment

(1) Cisco Nexus 3132Q-X switch2 1RU

(8) C13-C14 AC Power Cord 2 Meters

(4) Network Cable 3 Meters

HP ProLiant DL380 Generation 9 (Gen9) server user

2Hereafter referred to as Nexus switch.

3Hereafter referred to as ProLiant DL380 server.

4Caution: SFP+ optical transceivers are sensitive to static electricity.

Table 2: FNA SKU2 Provided Equipment

(4) HP ProLiant DL380 Generation 8 (Gen9) server2 2RU1

(8) C13-C14 power cable 2 Meters

(4) Network cable 3 Meters

HP ProLiant DL380p Generation 8 (Gen8) server

3Hereafter referred to as ProLiant DL380 server.

3Caution: SFP+ optical transceivers are sensitive to static electricity.

Table 4: FNA Pre-Install Checklist

n x 10 Gigabit (10Gb) available ports on ISP switch or router1

n available fiber patch cables (multi-mode or single-mode)1

Install the ProLiant DL380 servers into a compatible server rack.

Install the Cisco Nexus switch in the rack.

Connect the Cisco Nexus switch to your router.

Power on all FNA hardware.

NOTE: Multi-hop sessions are supported.

Facebook will begin initializing the FNA system for operation.

This completes the network configuration of the FNA.

 Prefixes smaller than /24 (255 IPs)

Table 5: FNA Installation Completion Checklist

All ProLiant DL380 servers in the cluster are powered on.

Your ISP has agreed to:

Log into the FNA portal.

Click the Caching tab and Manage FNA.

Instructions for Windows Operating Systems

Download and install Win32 Disk Imager and 7-Zip.

Instructions for Mac OS

Diskutil unmountDisk /dev/disk2

Run gunzip to unzip the disk image.

Run dd to write the disk image to the USB drive.

Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync

Instructions for Linux Operating Systems

Run dd to write the disk image to the USB drive.

Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync

Reboot ‘fna001’ while the USB drive is connected.

NOTE: This process will take a significant amount of time.

The network partner portal is the central hub for requesting

Logging into the NPP is as easy as logging into Facebook. If you

Click Add another email or mobile number.

Re-enter your Facebook password when prompted.

Click Save Changes.

To add a new administrator, perform the following steps:

To modify an existing user's permissions, perform the following steps:

Click the Settings icon ( ).

Click the Edit link on the user's card.

To remove a user from the portal, perform the following steps:

Locate the user you would like to remove.

Click the Remove link on the user's card.

Enter your Facebook password if prompted.

To add your ASN, follow these steps:

Click the Settings icon ( ).

Click Autonomous Systems located in the left menu.