Facebook went down!! Let's use that to talk about BGP, DNS, and Anycast
ฝัง
- เผยแพร่เมื่อ 31 พ.ค. 2024
- On Monday, Facebook had one of their biggest outages in recent history. The world was unable to access Facebook.com, or other Facebook properties (WhatsApp, Instagram, etc...). The outage created a wonderful opportunity to talk through BGP, DNS, and Anycast, which were all involved in the root cause.
Disclaimer: Let's be clear.... I do not work for Facebook. I don't actually know what happened. =)
I simply pieced together the little I saw from my corner of the Internet, in conjunction with a few other reports to outline my best interpretation of what happened. The goal of this video is not to claim any objective truth about the exact sequence of events. My goal is simply to use what happened to teach BGP, DNS, and Anycast from an unique perspective.
Want to learn Networking?
• Networking Fundamentals
00:00 - Start / Intro
02:39 - Tech Fail #1 - Teaching on Mute - SKIP THIS CHAPTER
06:24 - Me realizing my Audio was muted (watch for a laugh)
07:40 - Plan for the Stream, take 2
09:05 - Tech Fail #2 - Teaching without Screen Share - SKIP THIS CHAPTER
10:22 - Me realizing my Screen Share was off (watch for a laugh)
10:55 - DNS - A Records & NS Records
13:09 - DNS - Types of Name Servers: Caching, Root, TLD, Authoritative
16:22 - DNS - Process for Resolving a Record
25:50 - BGP
32:09 - Anycast
40:05 - Tying it all together
44:30 - Q&A
48:16 - Outro - วิทยาศาสตร์และเทคโนโลยี
![[TH] 2024 PMSL SEA GF D1 | Summer | แข่งที่ไทย ต้องใส่ให้สุด](http://i.ytimg.com/vi/Jkuf7nCS7jI/mqdefault.jpg)
The way you break down networking is crazy… Thank you for all your time for teaching us!
You're welcome, John. Thank you for looking past all the tech glitches in this video, heh.
Well, I had a bunch of technical issues with this live stream. There is irony of having tech issues while explaining the tech issues Facebook dealt with on Monday. ;) Either way, I've added chapters to the video. The real content is at these timestamps:
00:00 - Intro [until 02:39]
10:55 - DNS - A Records & NS Records
13:09 - DNS - Types of Name Servers: Caching, Root, TLD, Authoritative
16:22 - DNS - Process for Resolving a Record
25:50 - BGP
32:09 - Anycast
40:05 - Tying it all together
44:30 - Q&A
48:16 - Outro
lolz it was fun to watch you get all serious and explain stuff. but again ty for this video. keep it up.
But I enjoyed it.. 😀😀 Learning should be fun right 😀.. Thanks for the video
@@mariamfirdous8456 Heh, it was a good laugh after the fact (and during). A little embarrassing though. ;)
@@thivakaran6849 This one certainly brought about a few laughs ;). Glad you enjoyed it!
If I can smash the like button more than once, I would do it a thousand times. This channel has so much knowledge and I can't stress how valuable it is. Thank you for all the effort you've put into it all, Ed!!
Also, gotta love how even the most technical people out there still mess up the simplest things.. Humans be humans :)
Thanks for the kind words, Fares =).
And yea, I had so many glitches that day, it was not my best tech day. My perfectionism has desperately wanted me to remake the video, but I'm forcing myself to simply accept it for what it is.
Cheers Fares!
😂😂me too!! Ed is like the Instructor of the millennium!😊
@Practical Networking no worries. It was so helpful. I thought TikTok and Facebook had a fight and TikTok won!!😂😂
We lost 4 minutes talk ! Thank you for taking time and explaining based on your deep understanding of networking perspective. You are awesome , Ed. This is super important for everyone!
I watched all of Networking Fundamentals, NAT vs PAT and subnetting concepts in preparation of my AWS Advanced Networking Specialty exam. I am also learning IBGP, EBGP , Direct Connect. Your Networking & security concepts cleared my most of questions.
Much appreciated, Surender. Thank you for all the sharing you've done with my content!
Glad the videos helped w/ the AWS certification! When is your exam scheduled?
Ed, You are definitely one of the best networking gurus on the planet. I have learned so much about networking from your videos as VMware Consulting Architect. Please continue to share your vast knowledge on networking to YT. I has been extremely valuable to me in terms of piecing together concepts that have been a black hole for some time. Again, thank you and best of luck in your Practical Networking venture on YT. All the Best, William "Bill" J. Torelli, Jr. MS | VCP | ITIL, Transition Systems LLC
Hi Bill, thanks for the kind words! Glad my content has helped you professionally =).
If you're willing... Could you do me a favor? Do you mind sharing some of my videos (doesn't have to be this one) on Linked In, Reddit, Twitter, Facebook, or any other social media you use? As an independent creator, that would be an _enormous_ help, and I would appreciate it _greatly_ .
Ed, I will absolutely recommend your content. Your content, by far, is the best I've seen based on knowledge, presentation and delivery in comparison to the other creators in your space. Best of luck in building your channel, Bill
@@BillTorelli Thank you Bill, I appreciate your support =)
You are the Goat of teaching
Man you're simply awesome . Most people will fail in interview just because of they are not good at Fundamentals of how networking is working. your each and every videos are saving plenty of time reading the books or watching other network instructor videos. And also it will be really helpful for interview preparation as well.
Thank you for the kind words =). Glad you've enjoyed this content.
Thank you much more than I am able to express. I have learnt so much through your channel. You have tremendous capacity to make difficult concepts simple and clear.
Thank you for the kind words, Adeloye. I'm happy you found this channel beneficial. =)
This is a wonderful use case. Thank you for this lecture. Looked like you were under the weather, but you delivered regardless. I appreciate you.
Let's blame that for the tech difficulties =) Glad you enjoyed the content!
I just love you, man. Great work, and thanks for all you do. Your breakdowns are as good or better than I got at 3 quarters at Edmonds College. Great stuff, and thank you.
Glad to help, Scott. Glad you're enjoying this content =)
don't know why am laughing at your mishaps Lol. But this is quite insightful, just discovered the channel, i like it. Subscribed!
Don't worry, I laughed at them too =)
Thanks for your support!
Awesome, I will love the way you break down the topic
The topics , handsdown thank you sir
Great video , Thank you... at that time i heard this was caused due to some flowspec configuration issues in the BGP caused the BGP to withdrawn the routes.
Hello Ed, Thank you for this great course. Please do more deep dive videos like this.
Glad you enjoyed it, Dharmil! Noted about wanting more of these =)
Awesome, you've explained so well the notions. Thanks & big up ❤
Your way of making complicated subjcts so easy to understand is on another level 👌. Would like a video of BGP in more detail and the DNSec would be interesting too . Thanks 😊.
Thank you, Don =). All three are on my list to cover at some point. =)
@@PracticalNetworking great. Will look out for the notification 👌 😎
I can now solve subnetting issues with ease, thank you. Your videos are intuitive and your teaching style is priceless. Please I want you to do series on BGP, LINUX, DNS and Wireshark. I need your videos before I take my certification exam in Linux; I want a one-off approach. Thanks in advance.
That's great to hear, Silas. Glad you found the Subnetting series helpful. Thank you for the kind words =)
DNS / BGP are on my list. Once I can get more traction marketing my TLS course, I'll then focus efforts on the next course.
You are amazing! I learned a lot from your videos . Thank you for this insightful information.
You're welcome, Edward. Happy to help!
This is a great video, you obviously did a magnificent job clarifying FB outage... don't worry about minor audio issue, after all you did repeat everything from the very start...
Thank you for the kind words =). Glad it was useful despite the tech issues.
Ed, you are such a gem man. Thank you so much for doing what you do. You are such an incredible teacher, and I just really, really appreciate all the work you've put into your videos and and teaching us. Thank you from the bottom of my heart man, thank you! You're the best!
(And p.s., I know I'm 2 years late to this video, but hey, better late than never right! 😂)
😅😂 Ed, you were having a time w/ this one, huh?😊 😂😂 NO worries. 👍🏽This video is excellent as usual. 🎉THANK YOU.
Amazing video!!! Learnt so much!
A fun way to revise DNS, BGP and Anycast!..thank you
Agreed! =)
Hey , awesome content. I've recently started working with AWS and my networking knowledge gaps are too much to really build the right mental model. I'll check your other videos out.
Please do a course on Networking Virtualization on Cloud. SO MUCH TO LEARN!
>SO MUCH TO LEARN
Yes, 100% =). And yes, hope you enjoy the other Networking Videos:
th-cam.com/play/PLIFyRwBY_4bRLmKfP1KnZA6rZbRHtxmXi.html
Amazing work, truly appreciate your time spent on this topic...Can you please also dive as deep into the DNS security in one of ur videos ? Thank you so much for the technical explanations!
It would be fun to unpack DNS in more detail. Glad you enjoyed this video. Thanks for the recommendation =)
@@PracticalNetworking thanks for taking in consideration & for the reply!
Thanks a lot for this Amazing informative session Ed!!!...One question I have is , What's the lesson learnt - esp. one of the reason for cascading effect is using their domain name for AAA and badge access. What could be better design solutions?
That's a good question. First suggestion that jumps to mind is a separate domain / DNS host for "services FB staff uses" vs "external FB customers use". But you'd also need to host the separate domain on different infrastructure than the "customer facing" DNS infrastructure.
your standby expression after you miss to turn-on camera was hilarious...
;)
Normally routers are configured to authenticate via AAA server as the first priority and if the AAA server is inaccessible due to some reason a locally created disaster username/password can be used. The service provider has this arrangement in place and frequently ( once every quarter normally) changes the password for this local user name. The question is, was there any such thing in place for FB and if yes, why did they need physical access to the router, they would have been able to readvertise the anycast IP remotely via local username. I hope you will reply. BTW good explaination. keep up the good work.
Yes, the option of configuring a fall back user with full privileges always exists. But I've come across some enterprise networks where this was intentionally not the case. The idea for networks that big (and hopefully that redundant) is if access to a Router/Switch is lost, the Router/Switch is simply decommissioned and reset.
Consider, those organizations aren't concerned with 1 or 2 routers, but with 10000 and 20000 routers.
Glad you enjoyed the explanation =).
stuck with ya, no worries about the volume the content was very educational
Ha, yea so many tech fails in this video. I should re-do the content in dedicated videos on DNS & Anycast.
You really made understanding of networking concepts easy, fun and interesting, superb work !! I am not sure if its too much of ask - but can you come up with something on DNS area. I know this concept not directly linked to networking itself, but its used quite often. Things like domain, subdomain, hostname, namespace, DNS records makes it confusing. Its also ok if you can provide some good reference pointer, thanks again !!
A deeper dive into DNS is on my list, as it were. Right now I'm busy trying to market my SSL course: classes.pracnet.net/courses/practical-tls
Actually you are explaining complex thing simply and we are getting confident after hearing each session on a topic...
Also please mention sir what you studied
H
Glad you're enjoying my content, Anmar. Cheers!
I studied through CCNA/CCNP(x2) then was a CCSI for a while. Everything else was learned on the job.
Thank you for your work! Very informative and clear as always.
You're welcome, AlittA. Cheers@
thank you Sir, you are one of the best Teacher i have ever seen, awesome explanation Point to Point.
Glad you enjoyed it, Rafay!
Good stuff.👍
I had to see the video again. Still insightful. Thanks.
You make networking easy and interesting. Thankyou! sir
Thank you for the kind words =)
Hello Ed, recursive lookup is performed from nameserver , why they don't do iterative lookup are there any benefits of using Recursive or is it just a choice
Recursive is asking the name server to get the answer for you (this is what your Client does when speaking to a DNS server).
Iterative is asking the name server to give me the best possible answer it knows, or the nameserver of who should be asked next. This is typically what your DNS server itself is doing.
You can make your Client perform an iterative search using the "dig" tool and the "+trace" argument.
Thank you for this informative video. I really enjoyed it.
You're welcome, Venkat =)
Very good and easy explanation 🐧
Great explanation, thanks for the video. Qustion: If those AAA servers weren't accesible via DNS, couldn't they use IP adresses directly to connect? I bet they knew them.
another question - could you access the Facebook main page with its IP address? It's almost the same question, but I think, that it would not be possible in both our cases. The main page, AuthDNS and AAA servers are probably within the same ip block, but without the BGP, you can't access it...
@@jenycek2222 ah, that's right. I focused only on dns and forget about bgp
@@michawojcik1519 but I am not sure, I'm not an expert...
@Michał Wójcik & @Jan Novák -- great thoughts / questions!
Yes, if FB did withdraw more IP space than just their DNS servers from BGP, so even if us (random internet users) knew the IP we needed to reach, we likely would still not be able to access any FB IP space.
Internally, the FB network still existed, so internal devices *likely* had access to FB IP space. The issue is telling those devices to use a hard coded IP instead of a DNS lookup involved a configuration change -- and the configuration change involved authentication against "aaa.facebook.com". So it was a bit of a catch 22.
And again, all this is speculation =). It's unlikely Facebook will tell us what really happened.
Thank you for wonderful explanation. I have difficulty understanding anycast though. In some of the other internet resource, it was told that servers are distributed all over the world with same ip address and these are advertized via BGP. So, shortest route can be taken. I am just wondering, how is it possible to have same ip address for multiple machines on the internet? Can you please clarify. Thanks in advance.
Glad you enjoyed it!
Duplicate IPs are a problem on the same network. Duplicate IPs on either side of the Internet are not a problem... it's just about which instance of an IP address are you actually communicating with.
Routing protocols will manage all that. They'll try to route you to the "closest" instance of an IP address.
Can you please let me know what kind of pad you're using for drawing things? I'd like to buy something like that to take notes in my PDF documents. Thanks.
Just a touch screen laptop and a pen. Getting a full sketching tablet is on my list and would make things easier, I think.
@@PracticalNetworking Thanks for the response! :)
Great work i appreciate your time and efforts for helping us with your videos 🙂✅🖥️
Glad you enjoyed it. Cheers, Amarildinio
Excellent explanation!!
Thank you, Mohammad!
No Audio after start of the session. We love your efforts but please check once
Yup, I had all sorts of tech fails in this video. I've added chapters so you can jump to the parts w/o issues. See the pinned comment =)
The explanation is superb. I have seen lots of explanation on this and other issues by networking professionals on TH-cam or in real life , none can match your teaching skill. I don't really care about the minor technical issue you experienced in presenting this video. In my opinion, you are better than the two giants, David Bombal and Kevin wallce in teaching.
Heh. Thank's for sitting through the tech difficulties and getting to the 'meat' of the issue. Glad you enjoyed it =)
the best teacher
Thank you, Vikas.
Awesome video , however I have one doubt if on router aaa access is not working so mostly every organization has the local user itself on every router if any case aaa access is not working then they can use the local access .
Glad you enjoyed it!
At large enough organizations, where there are 100s, or 1000s, of other devices supporting the redundancy, there are sometimes no local users. The idea is if one device is behaving poorly, it can be taken offline and reset while the other devices handle the load. No single device is so important that a local full rights admin user is worth the risk.
In Facebook's case, it wasn't expected that *all devices* would loose access to the AAA servers.
Either way... you'd still run into the physical access problem. =)
@@PracticalNetworking thank you for clearing my doubt .thanks a lot
WoW explanation
=)
but when only the Auth DNS things were down due to BGP, then you could connect to Facebook straight with the IP address or were the IP addresses of actual servers down too due to BGP? It would make sense hence they may be in the same ip block / network.
I responded to you in the other thread with Michal W. But yes, it appears more than just the auth DNS servers were withdrawn from BGP by Facebook. So even if we knew the IP we needed to reach, we wouldn't have been able to access them.
Thank you!!!
You're welcome!
does auth ns has all the domain and ip? if yes why client doesnt direct request to auth dns instead requesting to caching/root dns first
Follow up question.... How would the Client know the Auth DNS server for _all_ domains? There has to be a way to query from zero previous knowledge, hence starting at the Root servers.
Thanks so much
You're welcome!
Was it possible to access facebook during that day by just typing the ip address out in a browser
Good question. But alas, no... since the IP addresses were withdrawn from BGP, even if you had the right IP, packets wouldn't get routed to the actual FB datacenters.
Lo vere mas tarde, gracias por compartirlo.
De nada =)
Could we have accessed the Facebook servers via the relevant IPs if we would know which IP to access, whilst the DNS was down?
The BGP withdrawal included more IPs than just the FB DNS servers. So any Internet user (even if they knew the IP they wanted to reach) would still be unable to access FB services.
Kudos
Cheers!
thank you sir really helpfull
You're welcome!
There seems to be some audio issues intermittently..
Yup, I had all sorts of tech fails in this video. I've added chapters so you can jump to the parts w/o issues. See the pinned comment =)
Why ips werent cached on the client side？
Even if It was the routers didn't know how to get there because Facebook's BGP was not advertising their "location"
10:48 start of slide notes
So basically the audio issue you had was similar in a way to the BGP issue at Facebook since no one arround the world could hear you. LOL
Ha, yea pretty much !
Laughing 10:26 anyway thanks Ed 😊
What a coincidence while providing a Facebook DNS solution you faced your own DNS glitches...🤣🤣
I know, the irony is thick ;)
It was a big fall)
Funny
Yea, I had a laugh =)
Hey sound gone
It comes back later in the video. See the timestamps for links.