BGP: The Protocol That Holds the Internet Together (And Sometimes Breaks It)
On October 4, 2021, Facebook went down. Not just the website — the entire global infrastructure. Facebook, Instagram, WhatsApp, and Oculus. Billions of people couldn't reach any Facebook service for about six hours. It was one of the most significant internet outages in history.
The cause wasn't a hacker. It wasn't a server failure or a power outage. It was a misconfiguration in BGP — the Border Gateway Protocol. A routine maintenance command accidentally removed Facebook's BGP announcements, making the global internet forget how to reach Facebook's servers. The servers were running. The data was intact. But with no BGP routes, the internet had no path to get there.
BGP is so fundamental that when it breaks, nothing else matters.
What Is BGP?
BGP is the routing protocol of the internet. It's the protocol that allows different networks — your ISP's network, Google's network, Facebook's network, university networks, enterprise networks — to share routing information with each other and work out how to send traffic between them.
BGP stands for Border Gateway Protocol (currently version 4, defined in RFC 4271). It's what makes the internet a network of networks rather than a collection of isolated islands.
To understand BGP, you first need to understand the concept of an Autonomous System (AS).
Autonomous Systems: The Building Blocks
The internet is not one network. It's a collection of thousands of independently operated networks. Your ISP is a network. Google is a network. AWS is a network. University networks are networks. Cable companies, mobile carriers, CDN providers — each is its own independently operated network.
These independently operated networks are called Autonomous Systems (AS). Each AS is assigned a unique ASN (Autonomous System Number) by a regional internet registry. ASNs are 16-bit (1 to 65535) or 32-bit numbers, though they're often written in the format `AS15169` (Google) or `AS16509` (Amazon).
Within an AS, the network operator uses whatever routing protocol they want — OSPF, IS-IS, EIGRP — to route traffic internally. This is called Interior Gateway Protocol (IGP) routing. What happens inside an AS is the AS's own business.
Between Autonomous Systems, BGP is the standardized protocol for exchanging routing information. This is called Exterior Gateway Protocol (EGP) routing.
How BGP Works
BGP is a path vector protocol. When a router announces that it can reach a particular network, it includes the full list of AS numbers the packet would have to traverse to get there — the AS path. This path information serves two purposes:
1. Loop prevention: If a router sees its own ASN in the path of an announcement, it knows accepting that route would create a loop, and rejects it.
2. Policy enforcement: Network operators can make routing decisions based on paths — preferring routes through certain ASes, avoiding others, preferring shorter paths.
Here's a simplified example:
As announcements propagate, every AS learns paths to every IP prefix in the internet — and typically receives multiple different paths (through different sets of ASes). BGP selects the "best" path using a complex set of criteria.
iBGP vs. eBGP
There are two flavors of BGP:
eBGP (External BGP): BGP sessions between different Autonomous Systems. This is the "real" internet routing BGP — the sessions between your ISP and Google, between ISPs, between carriers. eBGP sessions run between border routers at the edge of each AS.
iBGP (Internal BGP): BGP sessions within a single Autonomous System. Large networks use iBGP to distribute the BGP routing information they learned from external peers to all internal routers. iBGP has a rule: BGP routes learned from an iBGP peer are not re-advertised to other iBGP peers (to prevent loops). This means every iBGP router must have a session with every other iBGP router — a full mesh. In large networks, this is managed with route reflectors — designated routers that re-advertise routes to other iBGP peers, reducing the full mesh requirement.
BGP Route Selection
When a BGP router has multiple paths to the same destination (same IP prefix), it must choose one "best" path to actually use. BGP evaluates candidate paths against a list of criteria in order. The first criterion that differentiates paths wins:
1. Highest local preference: A locally configured preference value. Used to prefer certain exit points from your AS. Higher wins.
2. Shortest AS path: Fewer ASes in the path. Shorter paths are preferred.
3. Lowest origin type: IGP < EGP < Incomplete. (Usually IGP.)
4. Lowest MED (Multi-Exit Discriminator): A metric that can be set by a neighboring AS to influence which of its entry points you use. Lower MED is preferred.
5. Prefer eBGP over iBGP: Routes learned from external peers preferred over internal ones.
6. Lowest IGP metric to the BGP next-hop: Prefer the path whose next-hop is closest internally.
7. Oldest route: In a tiebreaker, prefer the route that's been in the table longest (stability).
8. Lowest Router ID: Final tiebreaker — the router with the lowest BGP Router ID wins.
These rules encode complex routing policy in a systematic way. Network engineers use local preference (rule 1) heavily to express "we prefer to send traffic out through ISP A rather than ISP B."
BGP Peering: How Networks Connect
BGP sessions are established between specific pairs of routers that the network operators have agreed to connect. This agreement is called peering.
Networks peer in two primary ways:
Transit: One network pays another for internet access. Your ISP provides you "transit" — you pay them, and they accept your traffic and route it anywhere on the internet. Your traffic travels through their BGP network and exits through their upstream providers. Transit is a business relationship where money flows from the smaller network to the larger.
Peering: Two networks exchange traffic with each other for free (or at low cost) because it's mutually beneficial. Your ISP and Google may peer directly — Google wants to reach your ISP's customers cheaply, and your ISP wants to deliver Google traffic cheaply. Neither pays the other; they just exchange traffic directly.
Direct peering often happens at Internet Exchange Points (IXPs) — physical facilities where many networks bring their routers and peer with each other over a shared switching fabric. The Amsterdam Internet Exchange (AMS-IX) and the DE-CIX Frankfurt are among the world's largest, carrying hundreds of terabits of traffic per second. Rather than paying transit for each network-to-network connection, networks at an IXP can peer with dozens or hundreds of others over a single physical connection.
BGP Security: A Known Weakness
BGP was designed in a more trusting era of the internet. It has a fundamental security weakness: there's no automatic verification that the network announcing a route actually has the right to announce it. Any BGP router can announce any IP prefix.
BGP Hijacking: An attacker or misconfigured router announces prefixes they don't own. Other networks see this announcement and potentially route traffic toward the attacker instead of the legitimate destination. This has happened many times — sometimes accidentally, sometimes maliciously.
In 2010, a Chinese ISP (China Telecom) accidentally (or not — it's disputed) announced around 50,000 BGP prefixes belonging to other networks. For about 18 minutes, significant amounts of US military and government traffic was routed through Chinese networks. In 2008, Pakistan Telecom intentionally (trying to block YouTube internally) accidentally hijacked YouTube's BGP prefix globally, making YouTube inaccessible worldwide for about two hours.
The proposed solution is RPKI (Resource Public Key Infrastructure) — a cryptographic system where each block of IP addresses is signed by a certificate that verifies who has the right to originate routes for it. Routers can then validate incoming BGP announcements against RPKI and reject invalid ones.
RPKI deployment has been growing but is still incomplete. As of 2024, significant portions of global route origins are covered by RPKI records, but not all networks enforce RPKI validation.
Route Leaks: A Related Problem
A BGP route leak is when a network re-advertises routes it shouldn't. A common scenario: a company has both a transit ISP and a peer. The company receives routes from both. Normally, you don't re-advertise your transit provider's routes to your peer (that would make you an unauthorized transit provider). If software misconfigures this and accidentally re-advertises those routes, traffic can be redirected unexpectedly.
This is exactly what happened in 2015 when a small ISP in Oklahoma accidentally re-advertised hundreds of thousands of routes learned from their transit providers back to one of the largest internet exchanges in the US, briefly disrupting routing for major networks.
The Facebook Outage, Explained
Back to October 2021. Facebook's internal infrastructure is an Autonomous System. Facebook's BGP routers announce their IP prefixes to the internet via their transit providers — so the world knows where to send traffic for Facebook's services.
A maintenance operation that day involved updating Facebook's border router configurations. The tool that made the change had a bug: it sent commands that, while individually valid, collectively removed all of Facebook's BGP route announcements. Facebook's routers stopped telling the world, "I can reach `157.240.x.x` (Facebook's IP range)."
Within minutes, those routes disappeared from global BGP tables. Every BGP router on the internet that had learned "to reach Facebook, go this way" forgot that information. DNS also stopped working because Facebook's DNS servers (which answer queries about `facebook.com`) are at those same IP addresses — and you couldn't reach the DNS servers without the BGP routes.
The compounding tragedy: Facebook uses its own internal tools for remote access to its infrastructure. Those tools used the internal DNS, which relied on the BGP routes that were now gone. Facebook engineers couldn't remotely access the routers to fix the problem because they couldn't reach the routers. It took hours of people physically traveling to data centers with access cards and console cables to manually fix the configuration.
The full restoration took about six hours. It was a stark reminder that BGP, and the humans who configure it, hold the internet together moment to moment.
Why We Should All Care About BGP
You will probably never configure BGP. For most people in technology, understanding BGP means understanding that it exists, roughly how it works, and why it fails in spectacular ways.
But the broader lesson is about the fragility of the internet's routing infrastructure. BGP is a 35-year-old protocol running on thousands of independently operated networks, held together by configuration conventions, business agreements, and — frequently — trust. Major outages happen multiple times per year due to BGP incidents.
The work of improving BGP security (RPKI, route leak prevention, better operational practices) is slow, collaborative, and doesn't make headlines when it's working. But it's essential infrastructure maintenance for the global communication system that most of the world now depends on.
The next time Facebook or another major service disappears for a few hours, there's a reasonable chance a BGP misconfiguration is somewhere in the chain of causes. And you'll now know what that means.