Those of you reading this online here starting on October 7th obviously remember the Facebook and Instagram outage on Monday, October 4th. For those of you reading this afterwards, I would imagine the 5-hour dark period where we couldn’t post or read Facebook (hopefully) should be a long-forgotten blip in an otherwise routine workday.




Wha Happened?

The long and short of it is that none of Facebook’s and Instagram’s many servers could be reached. The error would look different in a desktop browser (a more typical that-page-couldn’t-be-loaded error) than a phone (where the app wanted you to log in and claimed that your internet connection was the likely problem). Either way, the internet basically couldn’t find Facebook.

IMHO, if not posting on Facebook or Instagram was not a big deal to you then congratulations, you are living a healthy life. Although if you were running an ad campaign on Facebook or Instagram, then that’s another issue with Zuckerberg and Co. where prorated refund policy is unknown as of the time of this writing. (Also, if you use Facebook and Instagram just to run ads and don’t share a lot of personal info then we offer a second round of congratulations ... as they say in "The Social Dilemma" documentary, you are then the customer and not the product.)


Facebookschadenfreude

The more obvious online phenomenon I saw during the outage was the constant stream of Twitter jokes at Facebook’s expense. If you found one that was actually funny, let us know.

You might be discerning in the first four paragraphs here that I’m not a huge fan of Facebook. I do use it for work-related posts, and we also help some clients running Facebook ad campaigns. By the way, I and several others in the marketing/consulting business have found it perplexing that Facebook’s Ad Manager is one of the most non-intuitive and frustrating apps/sites/tools that we see anywhere; and yet it continues that way even though that’s where people are paying Facebook. It’s a head-scratcher.

So, the October 4th outage was a good old-fashioned day of schadenfreude for Facebook and Instagram haters out there. But there are a few other lessons or takeaways for businesses and websites to be had out there.


FYI, What Actually Happened At Facebook To Cause It

We urge you to read this terrific WIRED article on what likely caused the outage. It’s well-written, is careful and smart in its speculation on the Why’s and educates you about the process.

Basically, it seems like DNS records went down, expired, disappeared, etc. For most personal or smaller businesses websites running on a garden-variety server or host, the DNS records are what you’re paying for when it comes to domain registration. Typically, you might register a domain name with somebody like GoDaddy or Network Solutions, whereas the hosting is a separate serving. The hosting is literally keeping your files on a server while the domain registration is creating DNS records telling the Internet where your files are.

Larger cloud infrastructure systems such as Amazon Web Services or Microsoft Azure have some extra layers of distributing your data to multiple servers. And next-level self-contained behemoths like Facebook have even more complex versions of this but basically it seems like Facebook may have screwed up the accessibility of their DNS records. According to that WIRED article cited above, Facebook may have even had to have a couple of employees drive out to a server farm in a remote location to do some sort of manual reset. We’ll likely never really know. Nor does it matter that much.


Facebook User Authentication Service - Uh-Oh

Now here’s a more problematic part of the outages. A lot of sites use Facebook or Google third-party services for authentication, meaning that you don’t have to create a separate password for the site, you just use a Facebook login as long as you don’t have trust issues with Facebook. Easy for users since it saves the hassle of a new password. Easy for sites instead of building or buying another solution for user login/password or authentication code. And really a great thing for Facebook and their plan of world domination.

Some sites use Facebook authentication as the only way of being able to log into their site (instead of just one FB option among some other more traditional ones). So that means when Facebook was down, so were thousands of other sites. Even for these clients using these Facebook authentication services that had good and talented IT departments, there was nothing for them to do but wait for Facebook to come online, while they lost business or angered clients.

There is a great, scathing article in The Register (which features a great breakdown of how DNS records went down because of an embarrassing shortcut at Facebook, give it a read). In that article they quote Mark Zuckerberg’s mild-mea-culpa post of "Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today - I know how much you rely on our services to stay connected with the people you care about."

Then they follow it up by pointing out that Zuckerberg’s post was immediately preceded in its timeline by a video of Zuckerberg on a yacht. (Sorry, that’s just good schadenfreude.)


Relying On Outside Services

The lesson here is nothing so catty as "Don’t trust Facebook." To us, the takeaway is more just to be aware of the pros and cons and relying on outside services.

This just as easily could have been Google. PayPal money processing services and APIs are fantastic for small businesses, but they’ve had problems that have crippled businesses. GitHub had a major outage in 2020 that we remember bringing down lots of giant services like Amazon, Dropbox and some retail outlets.

In a way, we’re amazed that these things don’t happen more often. It’s the kind of thing that major players like Facebook, Apple, Amazon, Google do a great job and have lots of redundancies to prevent. But from time to time, they do happen on this kind of a scale where it’s a newsworthy topic.


Build vs. Buy ... vs. Renting

But when it comes to third-part apps or APIs that you or your relatively smaller company might be using, when those services go down it might not make the news or trend on Twitter, but you’re gonna catch hell from your customers.

There’s no way I am saying to not use third-party solutions. We live in a complex society. I don’t recommend growing all over your food, making your own guitar, or even changing your own oil - even though I can change my oil doesn’t mean I should because there’s a business two blocks away from me that will do it for $29 and get rid of the oil; your time is worth something.

The classic dilemma when it comes to pretty much all software is "Build vs. Buy". If you have the luxury of "building" then it’s the best option. Yes, if you have the time and means to grow your own tomatoes, do it - I do and they are fantastic. And the computer code that we at Plannedscape write directly for clients is the best in that it does exactly what they need. Our clients can also call us when they have a problem. Whereas if they have a WordPress site and a paid theme but want some changes that can’t be done, then they’re out of options. But we cost more than WordPress so there are obvious pros and cons there.

The "Buy" part of "Build vs. Buy" is to by services and code solutions. But a true "buy" in that scenario is that the customer has bought the tech outright and is then installed on their servers or configured within their enterprise system.

Solutions like using Facebook for user authentication, or any subscription service or paid API, is really "renting" instead of the "buy" in "Build vs. Buy". Yes, like a house renter when something goes wrong the landlord will fix it. That fix will be on the landlord’s timeline and within his or her budget of what they want to spend.


What In Your System Can Change Out From Under You

If any of this scared you and you want to be proactive, perhaps it’s a good time to audit your technologies and see what can change without your control.

On your website, any plug-ins and APIs could theoretically change. A good way for your current web designer to get a snapshot of that would be to look at some of your HTML and see any external URLs for script sources. Or even simpler, looks in HTML for any ".com" references on the page to see what services are coming from another place.

Or for any of your internet code for in-house apps can be searched for external references from web APIs (look for .com or web references) or check project API references like what .DLLs are wrapped into the project.

We don’t want this to be scary. Your programmer of choice can probably do any of this for you. Or if you really want our help, feel free to reach out - we’d be happy to audit this for you.


Meanwhile, Enjoy The Day Off Facebook

Apart from the lessons for your technology or software, we hope you enjoyed the day off from gossip, other people’s over-sharing and worrying about Likes. If you did, then maybe the next step is to plan a few digital vacations here and there.


The Feel-Good Takeaway

Here’s our real takeaway for you. Facebook has over 60,000 employees and they screwed the pooch on this one, so to speak. The next time you guys have a little hiccup or small mistake on your site or in your business, keep that in mind or find a way to gently remind this to your temporarily angry customer.

As we always say, customer loyalty doesn’t come from you never making a mistake. It comes from how well you handle your mistakes.