Explained: What led to Facebook, Instagram, WhatsApp's global outage

Facebook blamed the "faulty configuration change" for nearly six hours of global outage adding that its internal tools and systems were also impacted complicating attempts to diagnose and resolve the problem.

"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt," Facebook noted.

Expanding on Facebook's statement on the root cause behind the outage, tech experts explain a Domain Name System (DNS) mishap.

“Their [Facebook and its affiliates] DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had "pulled the cables" from their data centers all at once and disconnected them from the Internet,” two senior Engineers at Cloudflare, a web infrastructure and Web security company explained in a blog post.

According to the duo, Facebook stopped announcing routes to its DNS server through Border Gateway Protocol (BGP), a mechanism to exchange routing information between autonomous systems (AS) on the Internet.

Simply put, the internet that is connected through several networks, exchanges route information constantly to deliver every network packet to their final destinations. These networks are bound together by BGP that allows Facebook and other websites to advertise their presence to other networks that form the Internet, which went offline causing the outage.

When anyone tries to reach Facebook in the browser through its URL https://facebook.com, the DNS resolver that translates this domain into the IP address checks the cache and uses it. In case, there is no such cache available, it answers by the domain nameservers, typically hosted by the entity that owns it. If such nameservers are also unavailable or fail to respond, the browser displays an error to the user.

“At 1658 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes. That meant that, at least, Facebook’s DNS servers were unavailable. Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com,” the blog reiterated along with a screenshot of the failed Facebook DNS servers.

Meanwhile, other Facebook IP addresses remained routed but weren’t particularly useful since without DNS Facebook and related services were effectively unavailable, it added.

Essentially, Facebook and its affiliates disconnected themselves from the Internet.

Coincidently, the outage came a day after “Facebook’s whistleblower” appeared in a major interview on ‘60 Minutes’ alleging that the tech giant knew its products were fueling hate and harming children's mental health.

Impact

As the most popular social media networks went down around the world, more people resorted to other messenger applications like Signal that saw a sudden surge in downloads.

People look for alternatives and want to know more or discuss what’s going on. When Facebook became unreachable, we started seeing increased DNS queries to Twitter, Signal and other messaging and social media platforms, the blog noted.

“Signups are way up on Signal (welcome everyone!) We also know what it’s like to work through an outage, and wish the best for the engineers working on bringing back service on other platforms #Mondays,” Signal confirmed in a Twitter post.

The outage was also a reminder about the scale at which Facebook, Whatsapp, and Instagram connect the people around the world and its impact.