Feedback & Followups
- Another example of 2FA-bypass attacks in use in the wild: How Coinbase Phishers Steal One-Time Passwords – Krebs on Security — krebsonsecurity.com/…
- 🇪🇺 Pegasus Project: European Parliament awards journalism prize to investigation of use of spyware — www.brusselstimes.com/…
- Apple have released an updated and more detailed paper arguing against mandated side-loading on iOS: Building a Trusted Ecosystem for Millions of Apps: A threat analysis of sideloading —www.apple.com/…
- A Good summary: Apple: App sideloading on iPhones would ‘cripple’ security protections — www.tomsguide.com/…
- Observations from Bart:
- Apple really stress the illustrative value of Android here. They make a point of separating out the impact of laxer app store rules and side-loading on Android, with numbers showing that 93% of Android malware infections come via side-loading
- Apple open up about the abuses they see in the limited and very tightly controlled side-loading system they have — enterprise apps. They describe actual malware attacks that have been perpetrated using the mechanism and throw Facebook under the bus for good measure with their Onavo spyware VPN. The point being that just having the most tightly controlled side-loading possible has already caused problems, things would obviously get a lot worse if the flood-gates were opened.
- Apple argue if there were side-loading even users who don’t want to sideload would be pressured and/or tricked into it by schools, employers, developers, and cybercriminals
- There’s an entire page of the report dedicated to quotes from European and American government agencies issuing advice on blocking side-loading for security reasons.
- Continuing Social Media Tweaks:
Deep Dive — Facebook’s Very Bad Day
Unless you’ve been living under a rock, you know Facebook went down for 6 hours recently. While the outage was going on there were all kinds of speculation about what might be happening, not helped by the fact that the outage roughly coincided with a whistleblower giving evidence to the US congress.
My gut feeling was either a rogue employee making a point and/or a dramatic exit, or, a sysadmin having a really bad day. Turns out it was a sysadmin making a small mistake that lead to a cascade of failures that was extremely difficult to recover from.
At the root of the outage is some automation around one of the back-bone technologies underpinning the core of the internet — BGP, the border gateway protocol. This is the so-called routing algorithm that allows the routers that actually power the internet to build up a map of how the actual cables that carry internet traffic are connected to each other, and which IP addresses are where.
BGP is the absolutely work-horse of the internet, but it flies under the radar of most regular folk most of the time because there is no equivalent of it within our home networks. Routing within a typical home network is trivial, even if you set up three routers in a Y-shaped configuration to segregate off your IoT devices. What makes it trivial is that there is exactly one path between any two devices on the network, and between the internet and any device on the network. There are no choices to make, and there is no possibility of a loop.
The core of the internet is much more complicated, it’s made up of a massively interconnected grid of routers. Each router connects to many other routers, and there are many possible paths between any two routers, and bad routing decisions could easily set up loops trapping traffic. What’s worse is that routers come and go constantly as cables are added, removed, taken offline for maintenance, or break, cut by machinery, eaten by rodents (that happens a lot!), snapped by underwater landslides, or cut through by errant ship anchors.
No human could manage the chaos, so the routers have to figure it all out for themselves. This is the problem BGP solves, and a big part of the solution is that all routers are effectively gossips, telling all their neighbours everything they know. This means that information ripples through the internet as salacious news does through a village!
The source of all this gossip is announcements from routers with responsibility for specific blocks of IP addresses advertising (telling everyone that’s listening) that they’ve just come online and are ready to accept packets for their IP ranges, or, that they’ve changed their minds, and no longer want packets for those IPs (retractions).
Finally, we think of IPs as belonging to single devices, but out of the internet, that’s not true. Large CDNs use BGP to offer multiple possible end-points for a given IP address. This is how content delivery networks (CDNs) allow for fast downloads — the DNS for the servers map the name of the content-hosting server to a given IP, and BGP then offers lots of possible paths to that IP, each leading to a different server in a different part of the world that has a copy of the content. Each router uses the shortest path it knows about, so Irish customers end up at a server in one of the data centres ringing Dublin, and someone in Australia ends up talking to a server in Sidney or Melbourne etc.
When you have multiple servers powering a single IP, you need to update your advertised routes as servers are added to the pool, or removed from it.
Facebook decided to automate this process through some automation running on their DNS infrastructure, and through a whoopsie, accidentally caused their DNS servers to send out BGP advertisements retracting all routes to the IP addresses of their DNS servers. This means their DNS servers took themselves off the internet, and all Facebook domains became impossible to translate from human-friendly name to IP address, including the internal DNS records powering the infrastructure employees needed to securely connect from home. In effect, Facebook knocked themselves off the internet in such a way that the only solution was to physically get into the data centres, connect directly to the routers, and send out updated BGP advertisements. Because so many people are working from home, and because Facebooks data centres need superb physical security, it took hours to figure out what happened, get physically to the data centres, get into the right rooms, and get the routes published.
Basically, it was a cascading failure. It reminded me of the worst day of my professional career when a swan had an even worse day and shorted some high voltage cables shorting out electricity in most of our county. That cascaded with a battery failure in a UPS, that took down our entire infrastructure for the first time in years, and when we went to power up our private cloud we discovered its startup procedures depended on our DNS VMs which were hosted on our private cloud. One circular dependency, one very bad day! (The fix was a few hard-coded /etc/hosts files based on a document someone found on their computer that references the IP addresses belonging to the critical DNS names.)
I felt really sorry for the Facebook sysadmins — they now have one heck of a war story to regale fellow nerds with in the pub at tech conferences 🙂
Links
- Understanding How Facebook Disappeared from the Internet — blog.cloudflare.com/…
- What Happened to Facebook, Instagram, & WhatsApp? – Krebs on Security — krebsonsecurity.com/…
- Facebook says single error caused massive outage — www.imore.com/…
- Telegram bagged 70 million new users when WhatsApp was down for a few hours — www.imore.com/…
- About the Facebook Outage — Know a Little More — Overcast — overcast.fm/…
- Facebook Outage Increased Developer Throughput by 32% — www.usehaystack.io/…
❗ Action Alerts
- iOS 15.0.2, iPadOS 15.0.2, and watchOS 8.0.1 Fix Bugs, Major Security Flaw — tidbits.com/… & iOS 15.0.2 fixes a zero-day vulnerability but Apple hasn’t given credit to the researcher who found it — www.imore.com/…
- Patch Tuesday, October 2021 Edition — krebsonsecurity.com/…
- If you run your own Apache-based web server, be sure it’s fully patched. A very easy to exploit but was found in the web server software that was quickly, but poorly, patched, and a few days later another patch was released — nakedsecurity.sophos.com/…
Worthy Warnings
- A massive infrastructure provider for phone carriers around the world (about 235), including big names like AT&T, T-Mobile, Verizon, Vodafone, and China Mobile has revealed that hackers were active in their systems from May 2016 until May this year. Attackers could see call metadata like who called who for how long, and, the contents of SMS messages. Yet another reason to avoid SMS for 2FA when you have other options — www.vice.com/…
- Huge Twitch leak exposes source code, passwords – what you need to do — www.imore.com/…
- 🇺🇸 A breach at Verizon carrier Visible has resulted in fraudulent orders of iPhones being charged to people’s connected payment methods — www.imore.com/…
Notable News
- Apple’s support pages were briefly updated to list Safari Bookmarks as end-to-end encrypted, but the page updated again a few days later to say the bookmarks are encrypted in transit and while stored. That’s good, but not as good as E2EE — www.imore.com/…
- Apple have warned developers that if their app allows users to create an account, it must also allow users to delete their accounts. The deadline for developers to comply is the end of January 2022 — www.imore.com/…
- 🇬🇧 UK judge rules Ring video doorbell breach of neighbor’s privacy — www.imore.com/… (This seems to be more than just a simple case of a Ring camera being used in the normal way, but the fact that the Judge found that video of the neighbour belonged to the neighbour from a data protection POV seems significant, as did the focus on the Ring’s audio recording capabilities)
- Related: 🎧 The first episode of the excellent Nice Try! podcast dedicates an entire section of the show to the legal questions around Ring in the US, especially its audio features: Nice Try! – The Doorbell — overcast.fm/…
- 1Password announces easy item sharing with people who aren’t using it — www.imore.com/…
Top Tips
- Add Two-Factor Codes to Password Entries in iOS 15, iPadOS 15, and Safari 15 — tidbits.com/…
- How to use iCloud Private Relay on iPhone and iPad — www.imore.com/…
Interesting Insights
- Revealed: Facebook’s Secret Blacklist of “Dangerous Individuals and Organizations” — theintercept.com/…
- 🎧 The Ezra Klein Show: A Crypto Optimist and a Crypto Skeptic Walk Into a Podcast Studio — overcast.fm/… (A fascinating discussion of the big-picture changes the blockchain could bring to society if you assume the tech becomes easy to use and ubiquitous)
Palate Cleansers
- The NASA Astronaut Shane Kimbrough has been tweeting up a storm from the ISS, including some lovely photos of the earth at night. Two personal highlights:
- 🇮🇪 Dublin — twitter.com/…
- 🇧🇪 Brussels — twitter.com/…
- A great explanation of what the change of rain percentage on your weather apps actually means, and why it can say 100%, you can stay totally dry, and the app can still be completely correct — www.macobserver.com/…
Legend
When the textual description of a link is part of the link it is the title of the page being linked to, when the text describing a link is not part of the link it is a description written by Bart.
| Emoji | Meaning |
|---|---|
| 🎧 | A link to audio content, probably a podcast. |
| ❗ | A call to action. |
| flag | The story is particularly relevant to people living in a specific country, or, the organisation the story is about is affiliated with the government of a specific country. |
| 📊 | A link to graphical content, probably a chart, graph, or diagram. |
| 🧯 | A story that has been over-hyped in the media, or, “no need to light your hair on fire” 🙂 |
| 💵 | A link to an article behind a paywall. |
| 📌 | A pinned story, i.e. one to keep an eye on that’s likely to develop into something significant in the future. |
| 🎩 | A tip of the hat to thank a member of the community for bringing the story to our attention. |