Update: Facebook, Instagram, WhatsApp, Messenger and Oculus VR are back online as of 6:10 p.m. ET.
The dust has settled after Facebook's six-plus-hours-long outage yesterday, October 4. As the social media giant's blue website went down, so did Instagram, WhatsApp, Messenger and Oculus VR.
The company blames what it calls a "faulty configuration change." Essentially, the routers that chauffer people's web requests went offline. This had a rippling effect that brought down all facets of the entire company. A source told NBC News that employees couldn't even get into conference rooms as scanning ID cards all go through the same system.
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," said Santosh Janardhan, VP of infrastructure at Facebook via a blog post. "This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt."
Facebook's extended hiatus not only prevented aunts and uncles from sharing baby photos, but, according to a report by the New York Times, businesses in Turkey couldn't sell items nor could non-profits in Columbia use WhatsApp to communicate with victims of gender-based violence. It goes to show how one company has become so significantly ingrained in every part of human life. That's why there have been calls from antitrust watchers to break up the company.
Not only that, a Facebook whistleblower, Frances Haugen, is set to testify in front of Congress today about the way the company has hid its own internal data about its harms. Not only that, but she claims that Facebook consistently chose profit, even if it meant pushing more caustic content.
Below we've left an accounting of our updates from October 4 intact for future reference.
Facebook, Instagram and WhatsApp outage: latest updates
- At exactly 11.39 a.m. ET, Facebook, Instagram, WhatsApp, Messenger and Oculus VR went down. The outage is now closing in on the seven-hour mark.
- Facebook has not given a reason for the outage, but has tweeted that its working on getting the issue resolved.
- According to investigative journalist Brian Krebs, when speaking to Doug Madory of Kentik, a network observability company, Facebook's DNS records were withdrawn from global routing tables.
- According to Philip Crowther of the Associated Press, it's "mayhem" over at Facebook. Internal systems aren't working. Employees are communicating via text.
- According to the New York Times, Facebook employees' badges aren't even working, meaning they can't enter the building.
- As of 6:10 p.m. ET, Facebook, Instagram and WhatsApp are back online. We're still waiting on an official statement from Facebook regarding the outage.
While the Twitter accounts for Facebook, Instagram and WhatsApp were slow to update users, all three services have since put out short statements. Each is a variation of acknowledging the issue and telling users that work is being done to address the issue. Still, there's official release on when this outage will end or why the outage has occurred.
Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them. Bear with us, we’re on it! #instagramdownOctober 4, 2021
We’re aware that some people are experiencing issues with WhatsApp at the moment. We’re working to get things back to normal and will send an update here as soon as possible. Thanks for your patience!October 4, 2021
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.October 4, 2021
Of course, a flurry of silly memes poking fun at Facebook and its suite of services have begun cropping up, with many lauding Twitter for being active throughout all of this. Twitter itself is now in on the joke.
When whatsapp , instagram and Facebook , all three are down at the same time Le twitter :#Whatsapp pic.twitter.com/AY0i0nlS8jOctober 4, 2021
hello literally everyoneOctober 4, 2021
having a good time being onlineOctober 4, 2021
Possible outage cause
As to why the outage has occurred, that has yet to be determined. But according to investigative journalist Brian Krebs, when speaking to Doug Madory of Kentik, a network observability company, Facebook's DNS records were withdrawn from global routing tables.
DNS stands for Domain Name System. It's how the name of a website, such as tomsguide.com, is translated into a raw IP address. When the DNS isn't working, web browsers can't find the website being called.
It also seems that the BGP, or Border Gateway Protocol, routes have been pulled from the internet. Cloudflare describes BGP as the postal service for the internet. When someone wants to access data over the internet, it's BGP that tries to find the fastest route possible. Without BGP routes, there's no way for data to access Facebook, Instagram, WhatsApp, etc. Not only that, Facebook itself can't communicate within Facebook.
According to ArsTechnica, a user on Reddit who claims to be a Facebook employee, posits that network engineers may have been pushing a configuration change that accidentally locked them out. In this case, local data center technicians with local physical access to the routers are the only ones able to fix this. Per this Reddit user, the outage is not because of a malicious attack.
"The ongoing outage of WhatsApp, Instagram and Facebook (including Facebook Messenger and Facebook Workplace) highlights that global outages are one of the major downsides of a centralized system," said Matthew Hodgson, CEO of Element, technical co-founder of the Matrix open standard, in a statement to Tom's Guide. Element is a decentralized collaboration and messaging platform and Matrix is an open source API for decentralized communication with end-to-end encryption.
"Centralized apps mean that all the eggs are in one basket. When that basket breaks, all the eggs get smashed. We saw the same last week when Slack went down."
Hodgson argues that a decentralized system is ultimately more reliable as there's no single point of failure.
Today's outage is the second longest in Facebook history. The longest still goes to the one that occurred on March 13, 2019, which lasted nearly 12 hours.
Other trouble at Facebook
While there's no official statement as to why the outage has occurred, but for Facebook, this news comes at the heels of multiple damning reports. It's likely why #DeleteFacebook is also trending on Twitter.
Not only has the supposed DNS delisting thrown Facebook off the internet, it seems that internal systems across the company are down. According to Philip Crowther of the Associated Press, in speaking to a source at Facebook, "it's mayhem over here, all internal systems are down too." Employees are being forced to communicate via text message and Outlook email systems. Not only that, employee badges aren't even working, meaning many can't even enter the building.
60 Minutes recently published a report in which a former Facebook employee, Frances Haugen, blew the whistle on internal practices at the company which encouraged the spread of anger-inducing content as a ways to keep up engagement. This, of course, has led to much criticism being lobbed at the company as putting profits over civic health. Facebook, in internal memos, has denied its platform as being used as a tool which led to the Capitol insurrection on January 6 of this year.
The latest news on Facebook comes after the Wall Street Journal reported last month about the company squashing its own research about Instagram and the affect it has on younger users. It's mixed up in now cancelled plans for Facebook to launch Instagram for kids, which caught the alarm of multiple officials and led to lawsuits by attorneys general from 44 states.
- More: These are the best WhatsApp alternatives
If I were the CIO at Facebook I would have minimum 3 redundant servers farms at each site and then at least 6 worldwide IT sites with the ability to host all the traffic at any one site perhaps with performance hits but definitely not an outage. Facebook has more than enough money to build massive redundancy and extreme availability with failover almost instantaneous to another IT site. In today's world highly available and massively parallel application servers are known technologies as are the data replication and the failover technologies.
They have just not decided that they want to pay for true high availability and redundancy and all of the testing required to continuously validate the architecture and resources. Their objectives are very straight forward maximum Availability and Security. For that, you hire the best and you pay for the required infrastructure.