September 27, 2024 • 43 mins
Just over a month after the CrowdStrike debacle, Ed Zitron is joined by journalist and author Chris Stokel-Walker to "stokel-walk" through the brittle patchwork of open source, non-profit and for-profit entities that hold up the internet - and how calamitous it would be if any of them buckled.
Follow Chris: https://x.com/stokel
---
LINKS: https://www.tinyurl.com/betterofflinelinks
Newsletter: https://www.wheresyoured.at/
Reddit: https://www.reddit.com/r/BetterOffline/
Discord: chat.wheresyoured.at
Ed's Socials:
https://www.instagram.com/edzitron
https://bsky.app/profile/zitron.bsky.social
https://www.threads.net/@edzitron
See omnystudio.com/listener for privacy information.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:02):
All Zone Media.
Speaker 2 (00:05):
Hello and welcome to Better Offline. I'm your host ed Zichron.
Now you may have forgotten, but about a month or
two ago we had a huge, complete meltdown of the
(00:28):
computer systems of the world when CrowdStrike failed. I did
an episode on it, and well we've all just kind
of forgotten about it. And today I'm joined by Christokal Walker,
who's an author, journalist, lecturer and starting new column at
The Guardian as well.
Speaker 1 (00:43):
Aren't you yeah for a whole weeknad, I'm a whole month.
I'm taking over tech space, which will be very exciting
for the month of September.
Speaker 2 (00:52):
But the reason I brought you on is you wrote
a great article for The Independent back at the end
of July about how CrowdStrike isn't the only cyber company
that could trigger a glow, will melt down the second
they fail. And this is a subject that fascinates me
because I love disaster movies and also this article was terrifying,
So why don't you walk me through it?
Speaker 1 (01:11):
Yeah, So, basically, back in July we had that odd
outage that people might remember. Basically people woke up in
Asia Australia and then eventually the UK and encountered blue
screens of death on Microsoft Windows, which is something amazing
that you know. I'm thirty five now and I remember
(01:32):
that happening when I was a kid, and then it's
never happened since, like for all that, you know, people
make fun of Microsoft. Actually their PCs are decent, but
then suddenly everything went to hell in the handbasket. So
it turns out that, you know, CrowdStrike, which is one
of the big service providers for kind of antivirus tools
and software, had mist configured basically the thing that protects
(01:56):
us so that it actually harmed us, which is just
the ultimate in fantastic arriagories.
Speaker 2 (02:01):
And it turned out that it was actually within CrowdStrike,
the thing that was bugged failed because the bug checking
thing had a bug in it, which is so good.
It's so good that we have everything built on.
Speaker 1 (02:15):
Yeah, and this is this is this is the thing,
right it is? And I guess this is This is
something that a lot of your listeners and others will
chime with because you share a similar sensibility to me,
which is that we have built a huge thing which
is kind of like on a house of cards that
is actually hiding the fact that humans are involved in this,
(02:36):
and humans screw up frequently. Yet we think that actually,
because this is wiz bang tech, you don't have to
worry about it. It will worry of automation, Yeah, precisely,
And yet it's not as some guy who's overworked, overtired
and can't type properly and has fat fingers like you mean.
Speaker 2 (02:53):
Yeah, And a combination of private software companies like CrowdStrike,
and then as you'll get into open source solutions that
are a lot of people doing them for the love
of the game, which is wonderful. But at the same time,
we've got this patchwork system that holds up the Internet
and a lot of the tech we rely on and
we don't really know, and the fact that so much
(03:14):
of it is automated is also terrifying.
Speaker 1 (03:17):
Yeah, And this is the thing that that update was
sent out to millions of PCs over the course of
an evening while people were sleeping, and people didn't realize
that it was a massive issue with it until they
started to wake up, and at that point it is
essentially too late. And the best part of this story,
I think, is that to fix the issue, and we
(03:38):
have kind of fixed the issue a couple months on.
For a lot of them, you had to actually get
individual people to go out to either a computer or
a terminal somewhere and put in the actual updates to
unscrew up the problem that had existed initially, but you
highlighted there to kind of the issue of how much
of this is seen as kind of public utilities nowaday
(04:00):
is like the plumbing of our kind of entire world,
and yet it is kind of really rickety and hell
together with scotch tape and a little bit of chewing gum.
I did think a story a decade ago about how
we had a similar issue with a thing called heartleaed,
which was another co So basically it was another similar
thing where there was an update to a thing called
(04:21):
open SSL, which is the software tool that encrypts all
the data that is sent through payment systems and passwords
and so on and so forth. So whenever you see
that kind of yellow lock on your web browser, that
is usually running open SSL, and there was an issue
with it, which essentially meant that all of the information
(04:42):
that people thought was being shared in encrypted form was
actually being shown in plain texts. So someone could in
theory come along and snoop on everything that you're putting
in there, from bank account details to passwords and so
on and so forth. The reason why that happened is essentially,
the Internet at the time and to a certain extent
now or less so, was being run by two guys
(05:03):
named Steve. The whole thing was developed by a guy
called Steve Marquess and his friend who was also called Steve.
They were kind of this weird transatlantic romance where they
kept going open SSL. It was this initial volunteer run
project that kind of became a key part of the
Internet that frankly, these multi billion dollar companies used day in,
(05:24):
day out, but they decided that they didn't really want
to pay a penny for the upkeep and.
Speaker 2 (05:29):
Open SSL is one of the named kind of things
that could break the entire Internet. From your article as.
Speaker 1 (05:36):
Well, Yeah, this is kind of the key part of
the Internet's plumbing. And there are loads of these, right Like,
this is the thing that we don't realize until things
go wrong, and generally in it, people want to have
ninety nine point nine nine nine percent up time until
you have that kind of miniature final element where actually
(05:56):
something does screw up and actually you start to recognize
that this thing is held together pretty precariously. We don't
realize it, but there is a sort of cabal of
half a dozen or so companies whose job is essentially
to keep this stuff running, and sometimes they do screw.
Speaker 2 (06:14):
Up and open SSL. How was that actually funded? Is
it donations or Yeah?
Speaker 1 (06:20):
So Initially at the time, back in twenty fourteen, when
heartle was Hartleed was kind of an issue. They were
essentially relying on to nations. This was open source software
and that it's kind of the basic principle of the web.
We often forget about this. Actually it is hobbyists that
set this thing up alongside a huge military industrial.
Speaker 2 (06:38):
Company XKCD comic, which is everything's held up by a
guy called Runk.
Speaker 1 (06:42):
Yeah. Yeah, it turns out it was actually Steve but
basically the same sort of thing, and that is the
problem is they weren't fully funded. I did a follow
up story back in twenty fourteen because Hertleed kind of
drew the attention to this. And it comes back to
that idea of how when you started our podcast here
where you said this thing happened, this huge chaotic thing,
(07:05):
and you've probably forgotten about it because it's been two
months and we moved on same thing with heartleads. This happened,
there was this huge outcry. They got a bit of
funding around about a million dollars that was meant to
kind of make them back on an even keel and
so that they could be sustainable. Reality is kind of
didn't work.
Speaker 2 (07:23):
What do you mean it didn't work.
Speaker 1 (07:24):
Well in the sense of they still had this issue
happen again and again. So what was meant to be
kind of a fronting up. I suppose of big tech
companies saying, actually, you know what, we recognized that this
hobbyist service is a vital part of our internet's running.
We will fund it so it is sustainable. Didn't necessarily happen,
(07:47):
not to the extent that we've had another heartlead from
open SSL. But they do struggle still to kind of
keep things going. And that is I think the big problem,
which is the news agenda moves on. People forget very quickly,
and because there is then not a problem for a
(08:07):
little while longer, we kind of lurch from one catastrophic
near miss to another.
Speaker 2 (08:14):
So open SSL what does it actually do? Though? I
know you it's the little padlock on browsers. But what
is its foundational point?
Speaker 1 (08:25):
Yeah, so it basically it shepherds across data from from
a user to kind of a service provider. So if
you think about it as kind of you input text
on your laptop, your phone, wherever you are. It will
then encrypt it. It will transfer it over to a
(08:48):
payment provider, to your bank, to frankly also pretty much
anywhere that you put a password in and it will
ensure that that is encrypted all away. But there was
an issue with the coding of it, which meant that actually,
again comes back to fat fingers, some elements of what
(09:08):
was being transferred went into kind of excess memory, which
basically meant that bits of it were encrypted, but then
large parts of it weren't. So if you were unlucky,
then the bits that weren't encrypted could be your credit
card details, your sort code, and your account number and
a CBC number. And that's why there was kind of
this big red flashing light back in a decade or
(09:30):
so ago where people thought, you know what, actually, this exploit,
if left unchecked, could become a massive issue and a
real boon for cyber criminals. I think actually had it
happened in twenty twenty four, we would have seen much
more of a sort of significant issue in terms of
second third or more ramifications because cybercriminals would have been
(09:53):
all over that stuff.
Speaker 2 (09:54):
And if open SSL breaks again, and that just means
that the Internet it is not really encrypted, but every
place that uses it is kind of at risk every
transaction on every place.
Speaker 1 (10:07):
Yeah, and that is that is the issue, that is
the high wire act that is the Internet. And we've
seen outages like this come and go time and time again.
You know, we've had the crowd strike instant where you
can actually get onto your desktop or your laptops. We've
had outages on major payment platforms. We've had banks going offline,
(10:32):
we've had social networks kind of disappearing for hours at
a time. And invariably, this is just a very simple
error that snowballs repeatedly and we're kind of doomed to
repeat it. And I guess the challenge is, like, how
do we put the web on a firmer footing that
prevents this from happening again and again.
Speaker 2 (10:52):
It feels like funding the open SSL movement of foundation
would probably be a good start, But let's I imagine
and that's not happening. We need to build the computer
that makes pictures of Garfield with a gun exactly. This
is the thing.
Speaker 1 (11:06):
Commercial interests always come into this, and the reality is,
as you and I both know, and as many of
our listeners will know, companies tech companies in particular, will
take action when they realize that the spotlight is on
them and that there is this kind of intrinsic demand
for them to do that as soon as it's off,
they're actually doing the same old thing. They are happy
(11:26):
with the status quo as it is.
Speaker 2 (11:28):
Which is crazy as well, because what worries me about
open SSL is that nobody will really be to blame
and thus nobody will really feel responsible. They might kick
in some money here and there. Google especially very reliant
on them, but I just don't see them doing it.
Speaker 1 (11:46):
Yeah, and this is the thing that the only people
that I really noticed when I was reporting out that
story a decade ago who felt any kind of like
guilt or personal alarm kind of just like even responsibility
for it were actually those two Steves, like they were
they will gut it. The story I did, Yeah, the
(12:11):
story I did for bus you back then was it
was very difficult to report out because they had been
taken out of not to overly stereotype, but they were
like super tech, nuity people like they they were very
happy being in the background tinkering with this thing. They
realized that it was important and they took their jobs
(12:33):
very seriously, but they had never been put in a spotlight,
and they were initially very wary of speaking to me
because suddenly this thing happened. And it's over the course
of like hours that they got kind of thrust into
the limelight. They had the daily mail knocking at their door,
which was one of the reasons why they were the
super wary of talking to me. So it took actually
(12:53):
a few days of winning them over and saying, you
know what, like this isn't going to be a hit job,
Like I'm not looking to kind of hold you up
and say this is the person responsible for this happening.
It's more I wanted to tell the story of why
this has happened, why it's an issue, and why we
shouldn't have the ability for kind of slight errors in
(13:17):
upkey do cause catastrophic effects.
Speaker 2 (13:31):
It almost feels like people getting mad at the homeless
to some extent where it's like, oh, this person is
on the street and there are problems that are happening
around them, and they are and you blame the person
who is the victim here. You blame the fact that
when you look at the Internet right now and it's instability,
you're like, oh, well, these open source people who are
(13:53):
doing it for free, it's therefore because they should have
fucking they should have been better at doing this thing
for free. That holds up the entire versus the fact
that the problem is that the entire Internet relies on
this underfunded group of people, and it really is. I
just want to be clear for listeners and Chris you
of course know this. When I say this holds up
the entire Internet, I do actually mean that. It's very
(14:16):
easy to fall vout to hyperbole sometimes, but this is
genuinely that level. When Heartbleod happened, it sounds like it
could have been truly catastrophic.
Speaker 1 (14:26):
Good. It really good. And this is the thing that
we overlook is either there are kind of not amateurs,
because these people are super professional in terms of what
they do and they take their jobs around seriously, but
they are either not paid or they're paid a pittance,
particularly in comparison to the total compensation packages that you
(14:47):
see washing around Silicon Valley. And yet there is this
kind of super extractive approach from big tech companies of
we will kind of roll in whatever it is that
you provided us soften open source. And this is the
kind of big secret right of A large part of
big techs success is they rely on these open source
(15:08):
developments that have kind of underpinned key parts of their tech,
and if things go wrong, they can always shift the
blame onto those open source things and say, well, this
isn't actually our fault, this is the fault of our supplier,
the thing that came second or third order down the line.
Speaker 2 (15:27):
And you've got this big movement in cloud as well
towards like composable architecture, which involves a lot of slotting
in open source solutions as well. It's just it almost
feels like we need a big tech mutual aid thing
for open source. I wish that. I don't think there's
any way we could get a government to do this,
but I think they should force big tech to put
(15:48):
like a percentage of revenues, not profits, into open source
and have very defined lairs for them, because otherwise you
get situations where I don't know, the entire Internet is
underpinned by two steeves.
Speaker 1 (16:01):
Yeah, exactly, And I think this is this is the
unfortunate thing is that should be the lesson that we learned.
It should have been a lesson that we learned from
heart bleed. It should have been the lesson that we
learned from xe utils, which was another issue that we
encountered relatively recently, where there was it turned out, we
believe a bad actor kind of deliberately inserting milicious code
(16:25):
into another thing that kind of underpinds large parts of
our digital lives. Turns out that the volunteers that were
running that couldn't keep track of it. One of them
literally talked about their burnout and how they've kind of
taken a step away from the project. And yeah, we
always see these things kind of passing by in the
rearview window. We say, oh, you know what, isn't that
(16:45):
such a shame? We ought to do something about that,
And then we move on to the next thing, and
we don't.
Speaker 2 (16:50):
Pay attention taking a step back. What was exe utils
for the American listeners, ex for British and Canadian what
happened there?
Speaker 1 (16:59):
Yeah, this is kind of again another bit of free software.
This was back in sort of spring of twenty twenty four.
A kind of malicious hacker had, we think, basically socially
engineered their way into the upkeep of this bit of
(17:22):
open source utility, which is essentially designed to kind of
compress data. So the idea was that it would kind
of take a big file, chunk it up, make it smaller,
get rid of the bits that you don't need. And
it was kind of in the same way as you
talked about cloud architecture slotting in lots of really useful
open source tools, this is a similar thing where you
(17:44):
could slot in exit utails x utails into whatever you're
building and it would be fantastic. This guy had kind
of offered to volunteer at a time when the original developers,
the custodians of this tool were feeling very burned out,
(18:05):
said I will help. The original person then took their
eyes off the ball. This malicious actor started putting in
back doors intentionally the ways of accessing kind of the
data within. And it was only spotted basically by a
Microsoft developer who happened to come across.
Speaker 2 (18:23):
This, and I said, and just to be clear, though
exit utils looks like it's a big part of Linux,
which is a bit which people who use the smosftware
may not realize is basically underpinning most server architects. Like
a ton of server architecture, a ton of web architecture.
Speaker 1 (18:40):
Right, yeah, so servers, web servers, cloud hosting tools, lovely webcams,
basically anything that connected. Probably your fridge, if you have
an Internet of Things fridge, if you are that frivolous,
then that will be connected in some way to Linux.
I hope not. I hope that your diet coke is
not being kept cold by an IoT fridge.
Speaker 2 (19:01):
And there's someone who knows me so. But so this
Microsoft developer found it. And so it turns out that
just the corruption of open source happen like it can
happen as well with these open source projects, particularly I
imagine when they're underfunded and the people get burned out.
Speaker 1 (19:16):
Yeah, and this is this is the thing is again
it's another example of a kind of hobby project that
turned into something bigger. Nobody who has the money either
realized or decided that it was important enough to fund
directly the people involved who are often you know, again,
(19:37):
I don't want to stereotype, and I don't want to
kind of make them seem too much like a victim here,
But these folks are often super humble, super helpful, just
trying to keep their heads above water, essentially because they've
made something that has proved very, very useful, and they
don't want to trouble people by shouting for help. In
(20:00):
In this case, same thing happened. Single person in charge
of this tool didn't want to shout too loudly about
the problems that it would cause him in his life.
He kind of took a step away, decided to get
help from someone. Turned out they were bad, and nobody
decided to shout about it. So again this one was
This one was more deliberate rather than the fucker.
Speaker 2 (20:20):
I would also say they are victims. These people are
doing some of the most important work in the world
while san Dhar Pashai gets two hundred million dollars a year.
That I see these people as heroes and victims at
the same time.
Speaker 1 (20:36):
Yeah, and I agree, I think. I think what I
mean by that is they would not want to be
seen as either the hero or the victim in the piece, right.
They don't. They don't they have I think to be
involved in open source software more generally, you have to
have a very kind of uh it's almost quaint, right,
(21:01):
You have to be very, very community minded, very kind
of I am doing this for the good of everybody,
but I also don't want the praise for it, and
so either kind of being presented as like this hero
defending us against all of the bad stuff, or being
presented as the victim who is you need pity. I
think that's the thing they don't want. They don't want right,
(21:24):
but they just want money.
Speaker 2 (21:25):
And I think we get back to the systemic problem then,
because I don't see them as like any kind of
pathetic thing or indeed, I mean that I think that's
something heroic about what they're doing. But I think what
they're doing is cool. I think what the problem is
the systemic lack of support for them. We blame these things,
we don't but people may blame these projects for breaking
(21:46):
Oh it didn't work as well as it should, But
it turns out that it's just we we put all
of this pressure and these requirements on these people and
on these projects and then don't give them the sport
at all. So naturally, I'm going to say to my listeners,
please go and fund your open source movements. Brought on
Molly with Wikipedia. Molly Molly White of course about Wikipedia
(22:07):
very early on in the show. Fund these projects because
they deserve it. But the funny thing is is that
some of the sometimes I've seen very stupid idiots say
things like, well, if they were fun, if they were
private and corporate entities, they'd be fined and nothing would
go wrong. Except we look at CrowdStrike and it's the
complete opposite.
Speaker 1 (22:26):
Yeah, And the worst thing is with CrowdStrike is it's
they thought they knew better, right, like this is this
is the key thing. They thought that they were doing
everything perfectly. They kind of crowed about how good their
tools were, how well they could protect people, and then
they didn't. And this is kind of the endemic thing
(22:49):
is you you can't introduce profit to the equation because
if you do your kind of you're looking to cut corners.
Now we still don't really fully you know why this happened,
and crowdstrikeer is still taking a hit to its business.
(23:09):
It still has the threat of legal action from those
customers who were affected, and they were an awful lot.
You know, airlines were knocked off for basically an entire
week in the United States, you couldn't get anywhere on
some airlines because the systems were just so completely broken.
But the idea that you can just kind of throw
(23:30):
money at it through a big tech lens doesn't really
work because you need that idea, that kind of ethos
of I'm doing this not for profit, not for myself
and not for the company that I work for, but
I'm doing it for kind of the greater good. And
I think the problem if you brought this into a
(23:51):
Google or even a crowd strike or whatever, is that
you end up looking at the bottom line and realizing, actually,
I need to acquire customers. I need to keep them.
That's going to be my focus, not just making good
stuff and making it work.
Speaker 2 (24:05):
So onto profit seeking entities. One of the others you
mentioned is Fastly, so fastly. Let's why do you walk
me through Fastly? Because I know there are other companies
in this realm too.
Speaker 1 (24:17):
Yeah, so fastly is it's kind of what you would
call an edge cloud provider. So that is basically an
attempt to try and bring the internet speeds up a
little bit, make them a bit quicker. So the idea
of bringing files out are commonly used, or websites that
they're commonly used closer to where the users want to
(24:40):
request them. The thing that people often overlook is that,
you know, the web is essentially still a data transmission system,
and so you have to If I was to pull
up a YouTube video from my home in the UK,
it would be very silly for me to put that
request through YouTube servers in United States because I would
(25:02):
have to send the request to the United States, the
request would have to be fulfilled, YouTube would have to
go looking for the video, It would then have to
send the video back to me, and then it would
have to be played. Now we're really talking about kind
of a fraction of a second there, but it can
be done quicker by serving it closer to me physically.
Speaker 2 (25:18):
A content delivery network like vastly exactly.
Speaker 1 (25:21):
So that is what Vastly does. The problem is that
it went wrong around about three years ago, again like
crowd strike, a misconfigured file got pushed out of the
company systems. Because fastly is used by Amazon, by read,
by Twitch, by the UK government, by PayPal, all of
(25:43):
those platforms were affected, which is kind of a big issue.
This guy you know who runs Fastly, hugely wealthy man
and you know, has done an awful lot of good
makes the Internet faster. But the problem is a game.
It is a private company. It is a single point
(26:05):
of failure for many many platforms and many websites that
we use day in, day out, and so if something
goes wrong, it goes really really wrong.
Speaker 2 (26:14):
And what's weird about that is you'd think that Amazon,
for example, would have their own CDN.
Speaker 1 (26:22):
And they do have their own CDNs in some ways,
but they still the part of the thing is these
companies are so sprawling and these services that they provide
are so huge that they tend to try and bucket
them in different ways. And so while you know, the
fast the element went down, they still had other bits,
(26:43):
but it was kind of very much concentrate on Fastly.
So that's why we had those outages there.
Speaker 2 (26:49):
And there are other companies like this, like Achimaize the
other one, where if they buckle or fall, just chunks
of the Internet fall offline.
Speaker 1 (26:58):
Yeah, and again it goes back to around about maybe
got the late nineties early two thousands. We took a
series of decisions that essentially decided we are going to
take this thing that was previously like a kind of
hobbyist's home developed by frankly amateurs but actually kind of worked.
(27:24):
And we're going to turn this into like a massive
profit making machine, and we're going to privatize large parts
of it, and we're going to simultaneously have you know,
big business and also kind of you know, public goods
and services being transacted on it. And we've kind of
existed in that awkward space forever, and you've done episodes
(27:47):
in the past about loads of parts of social media
and the fact that there is this kind of challenge
of this is as Eezylon Musk's favorite raisor de facto
public square, but it is based on essentially private land.
And as soon as you kind of take what was
initially kind of like an educational base communications network and
(28:09):
you turn it into something that is for profit, you
really complicate things in a way that means you have
single points of failure and a lot of banks on
those things working, and when they don't, it causes big.
Speaker 2 (28:25):
Issues and it's it. It is a bit worrying, and
I try not to do too much fud on this show,
(28:46):
but this is the stuff that actually keeps me up
at night. This is the thing, especially as we have
the increasing electricity use of AI as, especially as we
have any basic strain on these companies that hold up
the Internet. The other thing I think about is what
if there are problems with I mean, we've seen this
tons of times with Amazon Web Services, with Microsoft as
your Google Cloud and so on and so forth. They
(29:08):
feel like also a huge point of failure.
Speaker 1 (29:12):
Yeah, and you see kind of rumblings of this right,
Like down detector is constantly pinging with things. Down Detector
is kind of the website that everybody goes to whenever
something either isn't working or isn't responding, to see whether
or not other people are noticing these sorts of issues.
It's strange, right because it's like we have and it
(29:35):
happens every month or two. We have kind of like
pretty significant tremors that put cracks in our walls, and
we kind of go, oh, you know what, let's just
plaster them up and I'll be okay, well okay, and
we kind of overlook it, and it's it's I suppose
the question is to what extent are those tremors kind
(29:55):
of like the pre warnings of like a massive rupture,
A huge kind of that is going to affect things
or are we able to just kind of keep it
ticking over and we have occasional outages and we fix
it and that's okay. Yeah.
Speaker 2 (30:09):
I was speaking with Burial in the other day. He's
an anti monopoly expert and he kind of made this point. Though.
We also have absolutely no public kind of measurement of
success or efficacy or indeed safety with any of these
cloud providers. We have it for power plants, we have
it for sewerage, we have it for water. We don't
(30:30):
treat despite those utilities, but really cloud services are utilities too,
and we just don't. We have no idea, We don't know,
and we have no quality standards. So who even knows
as they push these massive data centers whether they stay up.
And it terrifies me, it really does.
Speaker 1 (30:52):
Yeah. I think what's interesting is any data that we
do have is also provided by them, and it's kind
of it's bundled into marketing materials. Right They say we
have like six digit up time, which is that kind
of six nines after ninety nine points to highlight how
how well they maintain their services and how likely it
(31:13):
is that you will never encounter an outage. But the
reality is even that kind of point, not not one percent,
over a long course of time, can be quite a
significant outage. And if it's the thing, you know, if
it's if it's an outage that happens that is, you know,
keeping a hospital online or keeping your banking system online
(31:34):
at a time when everybody needs it. Even the smallest
outage on these kind of too big to fail services
can be huge, and we don't we don't realize them because,
as you say, there is no centralized record of this
is when we've had outages. This is when we've had issues.
They just come along every couple of months. They kind
of grab the attention. In the case of crowd strike,
(31:56):
it grabbed the attention because it was quite so massive
and quite so visual and visceral. But then we move
on and we forget about it, and actually we're only
ever reminded the next time, and by that point we're
so far beyond it that we forget actually how significant
it was. I mean, people couldn't go to work on
that Friday because they couldn't use their computers.
Speaker 2 (32:18):
Yeah, it's just really terrifying. This is the actual crisis,
and I feel as if it's almost it feels like
screaming into the void at times. One of the reasons
I wanted to do this episode was because of this,
because I don't think most people realize how brittle everything is.
You've got, oh the way that most transactions are in
crypto on the Internet, that's by two steves and everything
(32:40):
holding everything up is like a patchwork of a few
companies that are pretty much do not have They don't
get held accountable until something breaks. It's it's very bad,
but let's get it. Let's make it worse. So the
last two you brought up in your article, I can
and vera sign when you talk about why, they're also
(33:00):
very worrying.
Speaker 1 (33:02):
Yeah, so I currently is it's how do I describe this? Basically,
I can't at its heart run to what are called DNS,
the domain name system, which is kind of the address book.
So you type in a URL to your web browser
(33:22):
that is not machine readable, so it gets converted into
an IP address, which is a bunch of digits essentially,
and that gets rooted through what is called the DNS,
which is essentially a massive address book, and it's run
many of them, not all of them, three of the
kind of dozen or so that exists are run by
(33:42):
i can, which is a sort of nonprofit that is
one of the kind of earliest major organizations and involved
in kind of the early web, and also VeriSign, which
is kind of a private company. So if these things
(34:03):
go offline, then like everything breaks, because if the DNS,
if the kind of the address system of the Internet
of the Web is corrupted in some way. I don't
know about you, but I don't remember the IP address
of like the BBC News website.
Speaker 2 (34:21):
I remember my own phone number and nobody else's. That's
that should tell you everything. And just to be clear,
every website you visit, without exception, is actually just the
an IP address which has then gone through DNS. That's
that's good.
Speaker 1 (34:36):
Yeah. And so unless you are I don't know, some
sort of amazing memory, powerful individual who can remember every
single IP address.
Speaker 2 (34:48):
Who's also who also knows them because we don't get
exposed to them by the nature of the DNS system. DNS.
Speaker 1 (34:55):
Yeah, you just you just type it and it works,
and that is you know, it's one of those things.
It's we've we've traded off convenience for actually understanding how
this technology works, which is great because it works, but
if it doesn't work, then we're in real trouble. And
I think that is if you think about kind of
the economic impacts of crowd striking, the outages because they
(35:15):
couldn't people couldn't get onto their devices. Think about what
happens if people can get onto their devices, but they
don't know how to access their bank or they don't
know how to access the websites that they need for
day to day working. That is the really interesting thing.
And you know, I CAN is nonprofit. It has around
about four hundred staff, so like it is well staffed.
(35:38):
This isn't too Steves and a dog, but it is.
I suppose four hundred seems fewer people than you need
for something as important as this, right when you consider
the huge numbers that are employed by big tech companies,
you would think that I CAN would have.
Speaker 2 (35:56):
More than Well, the un has thirty six thousand people
working for it, and this is probably the size that like,
this is a probably a little bit more important than
the u N if you really think about it.
Speaker 1 (36:06):
Yeah, the website would not work without these things, and
so that is the.
Speaker 2 (36:10):
Email would email break as well?
Speaker 1 (36:13):
That's a good question. I think it would be.
Speaker 2 (36:16):
If you were accessing through through the webit world.
Speaker 1 (36:19):
Yeah, and also presumably I would I don't fully know
the answer, but I would presume that actually, yeah, because
you're putting in a a kind of domain name something
at something dot com or dot co, dot UK or
dot net or whatever, that it would still be routed
through the same systems.
Speaker 2 (36:36):
So yeah, And a quick Google says that that's the
case too. This is how I learned things. And also
another website I wonldn't be able to access the DNS
was down. That's that's so good.
Speaker 1 (36:46):
Yeah, exactly, this is the thing you would sometimes It's
happened when I used to work prior to journals, I
used to work in an office and sometimes the like
the actual router would fail and you would just kind
of be stuck there twiddling your thumbs and things, Well,
what can I do? Like imagine that but everybody in
the world, all at once unable to do the most
(37:09):
basic stuff. And think about how reliant we are on
all of the the internet connected services and tools that
we use, and then think about what would be the
impact if all of those stopped suddenly and we didn't
know what to do afterwards.
Speaker 2 (37:24):
And it says here in your article. There's thirteen of
the largest DNS servers a run by i CAN. So
three of the three of the thirteen largest run by
i CAN. So if you took if someone took out
I CAN, it would still function, but I imagine there'd be
a massive outage just kind of connecting the bits.
Speaker 1 (37:42):
Yeah, so they all have different route service, which is
kind of like the they have kind of the original
phone books as it were. You can get copies of
copies of copies of copies, which are increasingly less reliable.
It kind of generally seems to work geographically, so it
would probably affect parts of of the world rather than
the entire world, depending on which way you were served
(38:05):
through in terms of which quote unquote phone book you got,
which root server. But it's it's kind of a huge issue,
and the problem is we don't fully understand and wouldn't
fully understand until it happened, what the impact could be
because we know, okay, if it affected those three servers,
(38:27):
those three root servers, fine, but is there something on
the other root servers, or the websites or the back
ends of the organizations that operate the other root servers
that relies on those root servers to get access to
Like it's kind of could the domino effect start to
(38:48):
play out here where actually one pretty significant error anyway
could actually spread further and further and further.
Speaker 2 (38:56):
It's yeah, it's almost as if everyone It's would be
like if they one forgot how to speak. Yeah, you
could perhaps write letters, but speaking was off the table.
It's terrifying. And again, three of them are held up
by nonprofits. It's which is good, but all of them
should be. It's so strange. As countries we can all
(39:18):
get together to go to war or help support a
war perhaps, but we genocide, I guess in that case.
But we are in this situation where it's fucking we
can't put the money together to support the literal way
that people communicate online.
Speaker 1 (39:36):
Which is because we get through Yeah.
Speaker 2 (39:40):
It must works right now.
Speaker 1 (39:42):
Yeah, it works right now, and when it goes wrong,
it hasn't gone completely wrong one hundred percent all over
the world, and so we kind of go, well, that's
a whoopsie, Okay, we can deal with it and move on.
Hopefully it won't happen again. Fingers crossed. Let's hope that's
all okay, and that the way that it's kind of working.
That's the status quote a minute.
Speaker 2 (40:04):
I guess there's nothing. It's one of these I like
to end episodes by being like, what can regular people do?
It doesn't feel like we can.
Speaker 1 (40:10):
And you have read some of my journalism, you know
that I'm one of the most pessimistic people. I do
a radio slot here in the UK where I introduce
tech stories to people who don't necessarily know lots about tech,
and every single week I get harangued by the hosts
because I always end with depressing notes, and unfortunately that
is the case here.
Speaker 2 (40:29):
I think, Yeah, I think that something I like to
come back to though, is knowledge is power. I think
that I wonder if there is This is one of
the dumber things I've thought up, but I wonder if
there is actually a way of most people downloading the
phone book of DNA with DNS phone book just distributed.
Speaker 1 (40:46):
It could be crypto far off. I can be on
a blockchain.
Speaker 2 (40:50):
Bit of crypto. And it is funny as well, because
you see all of this AI bullshit and you got
the crypto bullshit and they're like, yeah, this is the future.
This is so cooled important. Objectively, DNA is cool, like
this stuff is that it's actually insane. The Internet works
at all.
Speaker 1 (41:04):
Yeah, Like I wrote a book called The History of
the Internet in bite sized Trucks, And as I said,
I'm thirty five. I kind of I joined the Web
when I was about ten or eleven, and yeah.
Speaker 2 (41:19):
I'm thirty eight, by the way, so we're right there.
Speaker 1 (41:21):
They go kind of got interested in it and found
it incredible, but forgot that, like I lost that wonder
because we don't see how it works anymore. You don't
see the crankshafts, you don't see the gears working in
the way that you used to. Kids nowadays don't know
how to store files on a computer because they just
(41:42):
have cloud storage. It's just always accessible easily there. They
don't have to structure a file system or something like that.
And so we take it for granted that these things
work and we just assume that like it's all okay.
But actually, yeah, knowledge is power. And knowing that there
is a person behind this, Knowing that there is a
system behind this and kind of getting a sense a
(42:03):
little bit of how it works means that you understand
more perhaps when things go wrong, and importantly, you can
kind of advocate maybe for how to make sure that
it doesn't go wrong again in the future.
Speaker 2 (42:17):
Chris, thank you so much for joining me. Where can
people find you?
Speaker 1 (42:20):
They can unfortunately find me on X that stokel that
is my am. I'm going down with that ship long
li SD Okay.
Speaker 2 (42:30):
Yeah, all right, thank you so much. Chris. You've been
listening to Better Offline. You know where to find me.
There's the same thing that comes on after it that
you'll complain because they haven't changed it in a wow,
thank you for listening everyone, and then it's gonna say
thank you for listening again. Thank you for listening to
(42:51):
Better Offline.
Speaker 3 (42:52):
The editor and composer of the Better Offline theme song
is Metasowski. You can check out more of his music
and audio projects at Metasowski dot com, M A T
T O S O W s ki dot com. You
can email me at easy at Better Offline dot com,
or visit Better Offline dot com to find more podcast
links and of course my newsletter. I also really recommend
(43:14):
you go to chat dot where's youreaed dot at to
visit the discord, and go to our slash.
Speaker 2 (43:18):
Better Offline to check out our reddit. Thank you so
much for.
Speaker 3 (43:21):
Listening Better Offline is a production of cool Zone Media.
For more from cool Zone Media, visit our website cool
zonemedia dot com, or check us
Speaker 1 (43:30):
Out on the iHeartRadio app, Apple Podcasts, or wherever you
get your podcasts.
All Zone Media.
Speaker 2 (00:05):
Hello and welcome to Better Offline. I'm your host ed Zichron.
Now you may have forgotten, but about a month or
two ago we had a huge, complete meltdown of the
(00:28):
computer systems of the world when CrowdStrike failed. I did
an episode on it, and well we've all just kind
of forgotten about it. And today I'm joined by Christokal Walker,
who's an author, journalist, lecturer and starting new column at
The Guardian as well.
Speaker 1 (00:43):
Aren't you yeah for a whole weeknad, I'm a whole month.
I'm taking over tech space, which will be very exciting
for the month of September.
Speaker 2 (00:52):
But the reason I brought you on is you wrote
a great article for The Independent back at the end
of July about how CrowdStrike isn't the only cyber company
that could trigger a glow, will melt down the second
they fail. And this is a subject that fascinates me
because I love disaster movies and also this article was terrifying,
So why don't you walk me through it?
Speaker 1 (01:11):
Yeah, So, basically, back in July we had that odd
outage that people might remember. Basically people woke up in
Asia Australia and then eventually the UK and encountered blue
screens of death on Microsoft Windows, which is something amazing
that you know. I'm thirty five now and I remember
(01:32):
that happening when I was a kid, and then it's
never happened since, like for all that, you know, people
make fun of Microsoft. Actually their PCs are decent, but
then suddenly everything went to hell in the handbasket. So
it turns out that, you know, CrowdStrike, which is one
of the big service providers for kind of antivirus tools
and software, had mist configured basically the thing that protects
(01:56):
us so that it actually harmed us, which is just
the ultimate in fantastic arriagories.
Speaker 2 (02:01):
And it turned out that it was actually within CrowdStrike,
the thing that was bugged failed because the bug checking
thing had a bug in it, which is so good.
It's so good that we have everything built on.
Speaker 1 (02:15):
Yeah, and this is this is this is the thing,
right it is? And I guess this is This is
something that a lot of your listeners and others will
chime with because you share a similar sensibility to me,
which is that we have built a huge thing which
is kind of like on a house of cards that
is actually hiding the fact that humans are involved in this,
(02:36):
and humans screw up frequently. Yet we think that actually,
because this is wiz bang tech, you don't have to
worry about it. It will worry of automation, Yeah, precisely,
And yet it's not as some guy who's overworked, overtired
and can't type properly and has fat fingers like you mean.
Speaker 2 (02:53):
Yeah, And a combination of private software companies like CrowdStrike,
and then as you'll get into open source solutions that
are a lot of people doing them for the love
of the game, which is wonderful. But at the same time,
we've got this patchwork system that holds up the Internet
and a lot of the tech we rely on and
we don't really know, and the fact that so much
(03:14):
of it is automated is also terrifying.
Speaker 1 (03:17):
Yeah, And this is the thing that that update was
sent out to millions of PCs over the course of
an evening while people were sleeping, and people didn't realize
that it was a massive issue with it until they
started to wake up, and at that point it is
essentially too late. And the best part of this story,
I think, is that to fix the issue, and we
(03:38):
have kind of fixed the issue a couple months on.
For a lot of them, you had to actually get
individual people to go out to either a computer or
a terminal somewhere and put in the actual updates to
unscrew up the problem that had existed initially, but you
highlighted there to kind of the issue of how much
of this is seen as kind of public utilities nowaday
(04:00):
is like the plumbing of our kind of entire world,
and yet it is kind of really rickety and hell
together with scotch tape and a little bit of chewing gum.
I did think a story a decade ago about how
we had a similar issue with a thing called heartleaed,
which was another co So basically it was another similar
thing where there was an update to a thing called
(04:21):
open SSL, which is the software tool that encrypts all
the data that is sent through payment systems and passwords
and so on and so forth. So whenever you see
that kind of yellow lock on your web browser, that
is usually running open SSL, and there was an issue
with it, which essentially meant that all of the information
(04:42):
that people thought was being shared in encrypted form was
actually being shown in plain texts. So someone could in
theory come along and snoop on everything that you're putting
in there, from bank account details to passwords and so
on and so forth. The reason why that happened is essentially,
the Internet at the time and to a certain extent
now or less so, was being run by two guys
(05:03):
named Steve. The whole thing was developed by a guy
called Steve Marquess and his friend who was also called Steve.
They were kind of this weird transatlantic romance where they
kept going open SSL. It was this initial volunteer run
project that kind of became a key part of the
Internet that frankly, these multi billion dollar companies used day in,
(05:24):
day out, but they decided that they didn't really want
to pay a penny for the upkeep and.
Speaker 2 (05:29):
Open SSL is one of the named kind of things
that could break the entire Internet. From your article as.
Speaker 1 (05:36):
Well, Yeah, this is kind of the key part of
the Internet's plumbing. And there are loads of these, right Like,
this is the thing that we don't realize until things
go wrong, and generally in it, people want to have
ninety nine point nine nine nine percent up time until
you have that kind of miniature final element where actually
(05:56):
something does screw up and actually you start to recognize
that this thing is held together pretty precariously. We don't
realize it, but there is a sort of cabal of
half a dozen or so companies whose job is essentially
to keep this stuff running, and sometimes they do screw.
Speaker 2 (06:14):
Up and open SSL. How was that actually funded? Is
it donations or Yeah?
Speaker 1 (06:20):
So Initially at the time, back in twenty fourteen, when
heartle was Hartleed was kind of an issue. They were
essentially relying on to nations. This was open source software
and that it's kind of the basic principle of the web.
We often forget about this. Actually it is hobbyists that
set this thing up alongside a huge military industrial.
Speaker 2 (06:38):
Company XKCD comic, which is everything's held up by a
guy called Runk.
Speaker 1 (06:42):
Yeah. Yeah, it turns out it was actually Steve but
basically the same sort of thing, and that is the
problem is they weren't fully funded. I did a follow
up story back in twenty fourteen because Hertleed kind of
drew the attention to this. And it comes back to
that idea of how when you started our podcast here
where you said this thing happened, this huge chaotic thing,
(07:05):
and you've probably forgotten about it because it's been two
months and we moved on same thing with heartleads. This happened,
there was this huge outcry. They got a bit of
funding around about a million dollars that was meant to
kind of make them back on an even keel and
so that they could be sustainable. Reality is kind of
didn't work.
Speaker 2 (07:23):
What do you mean it didn't work.
Speaker 1 (07:24):
Well in the sense of they still had this issue
happen again and again. So what was meant to be
kind of a fronting up. I suppose of big tech
companies saying, actually, you know what, we recognized that this
hobbyist service is a vital part of our internet's running.
We will fund it so it is sustainable. Didn't necessarily happen,
(07:47):
not to the extent that we've had another heartlead from
open SSL. But they do struggle still to kind of
keep things going. And that is I think the big problem,
which is the news agenda moves on. People forget very quickly,
and because there is then not a problem for a
(08:07):
little while longer, we kind of lurch from one catastrophic
near miss to another.
Speaker 2 (08:14):
So open SSL what does it actually do? Though? I
know you it's the little padlock on browsers. But what
is its foundational point?
Speaker 1 (08:25):
Yeah, so it basically it shepherds across data from from
a user to kind of a service provider. So if
you think about it as kind of you input text
on your laptop, your phone, wherever you are. It will
then encrypt it. It will transfer it over to a
(08:48):
payment provider, to your bank, to frankly also pretty much
anywhere that you put a password in and it will
ensure that that is encrypted all away. But there was
an issue with the coding of it, which meant that actually,
again comes back to fat fingers, some elements of what
(09:08):
was being transferred went into kind of excess memory, which
basically meant that bits of it were encrypted, but then
large parts of it weren't. So if you were unlucky,
then the bits that weren't encrypted could be your credit
card details, your sort code, and your account number and
a CBC number. And that's why there was kind of
this big red flashing light back in a decade or
(09:30):
so ago where people thought, you know what, actually, this exploit,
if left unchecked, could become a massive issue and a
real boon for cyber criminals. I think actually had it
happened in twenty twenty four, we would have seen much
more of a sort of significant issue in terms of
second third or more ramifications because cybercriminals would have been
(09:53):
all over that stuff.
Speaker 2 (09:54):
And if open SSL breaks again, and that just means
that the Internet it is not really encrypted, but every
place that uses it is kind of at risk every
transaction on every place.
Speaker 1 (10:07):
Yeah, and that is that is the issue, that is
the high wire act that is the Internet. And we've
seen outages like this come and go time and time again.
You know, we've had the crowd strike instant where you
can actually get onto your desktop or your laptops. We've
had outages on major payment platforms. We've had banks going offline,
(10:32):
we've had social networks kind of disappearing for hours at
a time. And invariably, this is just a very simple
error that snowballs repeatedly and we're kind of doomed to
repeat it. And I guess the challenge is, like, how
do we put the web on a firmer footing that
prevents this from happening again and again.
Speaker 2 (10:52):
It feels like funding the open SSL movement of foundation
would probably be a good start, But let's I imagine
and that's not happening. We need to build the computer
that makes pictures of Garfield with a gun exactly. This
is the thing.
Speaker 1 (11:06):
Commercial interests always come into this, and the reality is,
as you and I both know, and as many of
our listeners will know, companies tech companies in particular, will
take action when they realize that the spotlight is on
them and that there is this kind of intrinsic demand
for them to do that as soon as it's off,
they're actually doing the same old thing. They are happy
(11:26):
with the status quo as it is.
Speaker 2 (11:28):
Which is crazy as well, because what worries me about
open SSL is that nobody will really be to blame
and thus nobody will really feel responsible. They might kick
in some money here and there. Google especially very reliant
on them, but I just don't see them doing it.
Speaker 1 (11:46):
Yeah, and this is the thing that the only people
that I really noticed when I was reporting out that
story a decade ago who felt any kind of like
guilt or personal alarm kind of just like even responsibility
for it were actually those two Steves, like they were
they will gut it. The story I did, Yeah, the
(12:11):
story I did for bus you back then was it
was very difficult to report out because they had been
taken out of not to overly stereotype, but they were
like super tech, nuity people like they they were very
happy being in the background tinkering with this thing. They
realized that it was important and they took their jobs
(12:33):
very seriously, but they had never been put in a spotlight,
and they were initially very wary of speaking to me
because suddenly this thing happened. And it's over the course
of like hours that they got kind of thrust into
the limelight. They had the daily mail knocking at their door,
which was one of the reasons why they were the
super wary of talking to me. So it took actually
(12:53):
a few days of winning them over and saying, you
know what, like this isn't going to be a hit job,
Like I'm not looking to kind of hold you up
and say this is the person responsible for this happening.
It's more I wanted to tell the story of why
this has happened, why it's an issue, and why we
shouldn't have the ability for kind of slight errors in
(13:17):
upkey do cause catastrophic effects.
Speaker 2 (13:31):
It almost feels like people getting mad at the homeless
to some extent where it's like, oh, this person is
on the street and there are problems that are happening
around them, and they are and you blame the person
who is the victim here. You blame the fact that
when you look at the Internet right now and it's instability,
you're like, oh, well, these open source people who are
(13:53):
doing it for free, it's therefore because they should have
fucking they should have been better at doing this thing
for free. That holds up the entire versus the fact
that the problem is that the entire Internet relies on
this underfunded group of people, and it really is. I
just want to be clear for listeners and Chris you
of course know this. When I say this holds up
the entire Internet, I do actually mean that. It's very
(14:16):
easy to fall vout to hyperbole sometimes, but this is
genuinely that level. When Heartbleod happened, it sounds like it
could have been truly catastrophic.
Speaker 1 (14:26):
Good. It really good. And this is the thing that
we overlook is either there are kind of not amateurs,
because these people are super professional in terms of what
they do and they take their jobs around seriously, but
they are either not paid or they're paid a pittance,
particularly in comparison to the total compensation packages that you
(14:47):
see washing around Silicon Valley. And yet there is this
kind of super extractive approach from big tech companies of
we will kind of roll in whatever it is that
you provided us soften open source. And this is the
kind of big secret right of A large part of
big techs success is they rely on these open source
(15:08):
developments that have kind of underpinned key parts of their tech,
and if things go wrong, they can always shift the
blame onto those open source things and say, well, this
isn't actually our fault, this is the fault of our supplier,
the thing that came second or third order down the line.
Speaker 2 (15:27):
And you've got this big movement in cloud as well
towards like composable architecture, which involves a lot of slotting
in open source solutions as well. It's just it almost
feels like we need a big tech mutual aid thing
for open source. I wish that. I don't think there's
any way we could get a government to do this,
but I think they should force big tech to put
(15:48):
like a percentage of revenues, not profits, into open source
and have very defined lairs for them, because otherwise you
get situations where I don't know, the entire Internet is
underpinned by two steeves.
Speaker 1 (16:01):
Yeah, exactly, And I think this is this is the
unfortunate thing is that should be the lesson that we learned.
It should have been a lesson that we learned from
heart bleed. It should have been the lesson that we
learned from xe utils, which was another issue that we
encountered relatively recently, where there was it turned out, we
believe a bad actor kind of deliberately inserting milicious code
(16:25):
into another thing that kind of underpinds large parts of
our digital lives. Turns out that the volunteers that were
running that couldn't keep track of it. One of them
literally talked about their burnout and how they've kind of
taken a step away from the project. And yeah, we
always see these things kind of passing by in the
rearview window. We say, oh, you know what, isn't that
(16:45):
such a shame? We ought to do something about that,
And then we move on to the next thing, and
we don't.
Speaker 2 (16:50):
Pay attention taking a step back. What was exe utils
for the American listeners, ex for British and Canadian what
happened there?
Speaker 1 (16:59):
Yeah, this is kind of again another bit of free software.
This was back in sort of spring of twenty twenty four.
A kind of malicious hacker had, we think, basically socially
engineered their way into the upkeep of this bit of
(17:22):
open source utility, which is essentially designed to kind of
compress data. So the idea was that it would kind
of take a big file, chunk it up, make it smaller,
get rid of the bits that you don't need. And
it was kind of in the same way as you
talked about cloud architecture slotting in lots of really useful
open source tools, this is a similar thing where you
(17:44):
could slot in exit utails x utails into whatever you're
building and it would be fantastic. This guy had kind
of offered to volunteer at a time when the original developers,
the custodians of this tool were feeling very burned out,
(18:05):
said I will help. The original person then took their
eyes off the ball. This malicious actor started putting in
back doors intentionally the ways of accessing kind of the
data within. And it was only spotted basically by a
Microsoft developer who happened to come across.
Speaker 2 (18:23):
This, and I said, and just to be clear, though
exit utils looks like it's a big part of Linux,
which is a bit which people who use the smosftware
may not realize is basically underpinning most server architects. Like
a ton of server architecture, a ton of web architecture.
Speaker 1 (18:40):
Right, yeah, so servers, web servers, cloud hosting tools, lovely webcams,
basically anything that connected. Probably your fridge, if you have
an Internet of Things fridge, if you are that frivolous,
then that will be connected in some way to Linux.
I hope not. I hope that your diet coke is
not being kept cold by an IoT fridge.
Speaker 2 (19:01):
And there's someone who knows me so. But so this
Microsoft developer found it. And so it turns out that
just the corruption of open source happen like it can
happen as well with these open source projects, particularly I
imagine when they're underfunded and the people get burned out.
Speaker 1 (19:16):
Yeah, and this is this is the thing is again
it's another example of a kind of hobby project that
turned into something bigger. Nobody who has the money either
realized or decided that it was important enough to fund
directly the people involved who are often you know, again,
(19:37):
I don't want to stereotype, and I don't want to
kind of make them seem too much like a victim here,
But these folks are often super humble, super helpful, just
trying to keep their heads above water, essentially because they've
made something that has proved very, very useful, and they
don't want to trouble people by shouting for help. In
(20:00):
In this case, same thing happened. Single person in charge
of this tool didn't want to shout too loudly about
the problems that it would cause him in his life.
He kind of took a step away, decided to get
help from someone. Turned out they were bad, and nobody
decided to shout about it. So again this one was
This one was more deliberate rather than the fucker.
Speaker 2 (20:20):
I would also say they are victims. These people are
doing some of the most important work in the world
while san Dhar Pashai gets two hundred million dollars a year.
That I see these people as heroes and victims at
the same time.
Speaker 1 (20:36):
Yeah, and I agree, I think. I think what I
mean by that is they would not want to be
seen as either the hero or the victim in the piece, right.
They don't. They don't they have I think to be
involved in open source software more generally, you have to
have a very kind of uh it's almost quaint, right,
(21:01):
You have to be very, very community minded, very kind
of I am doing this for the good of everybody,
but I also don't want the praise for it, and
so either kind of being presented as like this hero
defending us against all of the bad stuff, or being
presented as the victim who is you need pity. I
think that's the thing they don't want. They don't want right,
(21:24):
but they just want money.
Speaker 2 (21:25):
And I think we get back to the systemic problem then,
because I don't see them as like any kind of
pathetic thing or indeed, I mean that I think that's
something heroic about what they're doing. But I think what
they're doing is cool. I think what the problem is
the systemic lack of support for them. We blame these things,
we don't but people may blame these projects for breaking
(21:46):
Oh it didn't work as well as it should, But
it turns out that it's just we we put all
of this pressure and these requirements on these people and
on these projects and then don't give them the sport
at all. So naturally, I'm going to say to my listeners,
please go and fund your open source movements. Brought on
Molly with Wikipedia. Molly Molly White of course about Wikipedia
(22:07):
very early on in the show. Fund these projects because
they deserve it. But the funny thing is is that
some of the sometimes I've seen very stupid idiots say
things like, well, if they were fun, if they were
private and corporate entities, they'd be fined and nothing would
go wrong. Except we look at CrowdStrike and it's the
complete opposite.
Speaker 1 (22:26):
Yeah, And the worst thing is with CrowdStrike is it's
they thought they knew better, right, like this is this
is the key thing. They thought that they were doing
everything perfectly. They kind of crowed about how good their
tools were, how well they could protect people, and then
they didn't. And this is kind of the endemic thing
(22:49):
is you you can't introduce profit to the equation because
if you do your kind of you're looking to cut corners.
Now we still don't really fully you know why this happened,
and crowdstrikeer is still taking a hit to its business.
(23:09):
It still has the threat of legal action from those
customers who were affected, and they were an awful lot.
You know, airlines were knocked off for basically an entire
week in the United States, you couldn't get anywhere on
some airlines because the systems were just so completely broken.
But the idea that you can just kind of throw
(23:30):
money at it through a big tech lens doesn't really
work because you need that idea, that kind of ethos
of I'm doing this not for profit, not for myself
and not for the company that I work for, but
I'm doing it for kind of the greater good. And
I think the problem if you brought this into a
(23:51):
Google or even a crowd strike or whatever, is that
you end up looking at the bottom line and realizing, actually,
I need to acquire customers. I need to keep them.
That's going to be my focus, not just making good
stuff and making it work.
Speaker 2 (24:05):
So onto profit seeking entities. One of the others you
mentioned is Fastly, so fastly. Let's why do you walk
me through Fastly? Because I know there are other companies
in this realm too.
Speaker 1 (24:17):
Yeah, so fastly is it's kind of what you would
call an edge cloud provider. So that is basically an
attempt to try and bring the internet speeds up a
little bit, make them a bit quicker. So the idea
of bringing files out are commonly used, or websites that
they're commonly used closer to where the users want to
(24:40):
request them. The thing that people often overlook is that,
you know, the web is essentially still a data transmission system,
and so you have to If I was to pull
up a YouTube video from my home in the UK,
it would be very silly for me to put that
request through YouTube servers in United States because I would
(25:02):
have to send the request to the United States, the
request would have to be fulfilled, YouTube would have to
go looking for the video, It would then have to
send the video back to me, and then it would
have to be played. Now we're really talking about kind
of a fraction of a second there, but it can
be done quicker by serving it closer to me physically.
Speaker 2 (25:18):
A content delivery network like vastly exactly.
Speaker 1 (25:21):
So that is what Vastly does. The problem is that
it went wrong around about three years ago, again like
crowd strike, a misconfigured file got pushed out of the
company systems. Because fastly is used by Amazon, by read,
by Twitch, by the UK government, by PayPal, all of
(25:43):
those platforms were affected, which is kind of a big issue.
This guy you know who runs Fastly, hugely wealthy man
and you know, has done an awful lot of good
makes the Internet faster. But the problem is a game.
It is a private company. It is a single point
(26:05):
of failure for many many platforms and many websites that
we use day in, day out, and so if something
goes wrong, it goes really really wrong.
Speaker 2 (26:14):
And what's weird about that is you'd think that Amazon,
for example, would have their own CDN.
Speaker 1 (26:22):
And they do have their own CDNs in some ways,
but they still the part of the thing is these
companies are so sprawling and these services that they provide
are so huge that they tend to try and bucket
them in different ways. And so while you know, the
fast the element went down, they still had other bits,
(26:43):
but it was kind of very much concentrate on Fastly.
So that's why we had those outages there.
Speaker 2 (26:49):
And there are other companies like this, like Achimaize the
other one, where if they buckle or fall, just chunks
of the Internet fall offline.
Speaker 1 (26:58):
Yeah, and again it goes back to around about maybe
got the late nineties early two thousands. We took a
series of decisions that essentially decided we are going to
take this thing that was previously like a kind of
hobbyist's home developed by frankly amateurs but actually kind of worked.
(27:24):
And we're going to turn this into like a massive
profit making machine, and we're going to privatize large parts
of it, and we're going to simultaneously have you know,
big business and also kind of you know, public goods
and services being transacted on it. And we've kind of
existed in that awkward space forever, and you've done episodes
(27:47):
in the past about loads of parts of social media
and the fact that there is this kind of challenge
of this is as Eezylon Musk's favorite raisor de facto
public square, but it is based on essentially private land.
And as soon as you kind of take what was
initially kind of like an educational base communications network and
(28:09):
you turn it into something that is for profit, you
really complicate things in a way that means you have
single points of failure and a lot of banks on
those things working, and when they don't, it causes big.
Speaker 2 (28:25):
Issues and it's it. It is a bit worrying, and
I try not to do too much fud on this show,
(28:46):
but this is the stuff that actually keeps me up
at night. This is the thing, especially as we have
the increasing electricity use of AI as, especially as we
have any basic strain on these companies that hold up
the Internet. The other thing I think about is what
if there are problems with I mean, we've seen this
tons of times with Amazon Web Services, with Microsoft as
your Google Cloud and so on and so forth. They
(29:08):
feel like also a huge point of failure.
Speaker 1 (29:12):
Yeah, and you see kind of rumblings of this right,
Like down detector is constantly pinging with things. Down Detector
is kind of the website that everybody goes to whenever
something either isn't working or isn't responding, to see whether
or not other people are noticing these sorts of issues.
It's strange, right because it's like we have and it
(29:35):
happens every month or two. We have kind of like
pretty significant tremors that put cracks in our walls, and
we kind of go, oh, you know what, let's just
plaster them up and I'll be okay, well okay, and
we kind of overlook it, and it's it's I suppose
the question is to what extent are those tremors kind
(29:55):
of like the pre warnings of like a massive rupture,
A huge kind of that is going to affect things
or are we able to just kind of keep it
ticking over and we have occasional outages and we fix
it and that's okay. Yeah.
Speaker 2 (30:09):
I was speaking with Burial in the other day. He's
an anti monopoly expert and he kind of made this point. Though.
We also have absolutely no public kind of measurement of
success or efficacy or indeed safety with any of these
cloud providers. We have it for power plants, we have
it for sewerage, we have it for water. We don't
(30:30):
treat despite those utilities, but really cloud services are utilities too,
and we just don't. We have no idea, We don't know,
and we have no quality standards. So who even knows
as they push these massive data centers whether they stay up.
And it terrifies me, it really does.
Speaker 1 (30:52):
Yeah. I think what's interesting is any data that we
do have is also provided by them, and it's kind
of it's bundled into marketing materials. Right They say we
have like six digit up time, which is that kind
of six nines after ninety nine points to highlight how
how well they maintain their services and how likely it
(31:13):
is that you will never encounter an outage. But the
reality is even that kind of point, not not one percent,
over a long course of time, can be quite a
significant outage. And if it's the thing, you know, if
it's if it's an outage that happens that is, you know,
keeping a hospital online or keeping your banking system online
(31:34):
at a time when everybody needs it. Even the smallest
outage on these kind of too big to fail services
can be huge, and we don't we don't realize them because,
as you say, there is no centralized record of this
is when we've had outages. This is when we've had issues.
They just come along every couple of months. They kind
of grab the attention. In the case of crowd strike,
(31:56):
it grabbed the attention because it was quite so massive
and quite so visual and visceral. But then we move
on and we forget about it, and actually we're only
ever reminded the next time, and by that point we're
so far beyond it that we forget actually how significant
it was. I mean, people couldn't go to work on
that Friday because they couldn't use their computers.
Speaker 2 (32:18):
Yeah, it's just really terrifying. This is the actual crisis,
and I feel as if it's almost it feels like
screaming into the void at times. One of the reasons
I wanted to do this episode was because of this,
because I don't think most people realize how brittle everything is.
You've got, oh the way that most transactions are in
crypto on the Internet, that's by two steves and everything
(32:40):
holding everything up is like a patchwork of a few
companies that are pretty much do not have They don't
get held accountable until something breaks. It's it's very bad,
but let's get it. Let's make it worse. So the
last two you brought up in your article, I can
and vera sign when you talk about why, they're also
(33:00):
very worrying.
Speaker 1 (33:02):
Yeah, so I currently is it's how do I describe this? Basically,
I can't at its heart run to what are called DNS,
the domain name system, which is kind of the address book.
So you type in a URL to your web browser
(33:22):
that is not machine readable, so it gets converted into
an IP address, which is a bunch of digits essentially,
and that gets rooted through what is called the DNS,
which is essentially a massive address book, and it's run
many of them, not all of them, three of the
kind of dozen or so that exists are run by
(33:42):
i can, which is a sort of nonprofit that is
one of the kind of earliest major organizations and involved
in kind of the early web, and also VeriSign, which
is kind of a private company. So if these things
(34:03):
go offline, then like everything breaks, because if the DNS,
if the kind of the address system of the Internet
of the Web is corrupted in some way. I don't
know about you, but I don't remember the IP address
of like the BBC News website.
Speaker 2 (34:21):
I remember my own phone number and nobody else's. That's
that should tell you everything. And just to be clear,
every website you visit, without exception, is actually just the
an IP address which has then gone through DNS. That's
that's good.
Speaker 1 (34:36):
Yeah. And so unless you are I don't know, some
sort of amazing memory, powerful individual who can remember every
single IP address.
Speaker 2 (34:48):
Who's also who also knows them because we don't get
exposed to them by the nature of the DNS system. DNS.
Speaker 1 (34:55):
Yeah, you just you just type it and it works,
and that is you know, it's one of those things.
It's we've we've traded off convenience for actually understanding how
this technology works, which is great because it works, but
if it doesn't work, then we're in real trouble. And
I think that is if you think about kind of
the economic impacts of crowd striking, the outages because they
(35:15):
couldn't people couldn't get onto their devices. Think about what
happens if people can get onto their devices, but they
don't know how to access their bank or they don't
know how to access the websites that they need for
day to day working. That is the really interesting thing.
And you know, I CAN is nonprofit. It has around
about four hundred staff, so like it is well staffed.
(35:38):
This isn't too Steves and a dog, but it is.
I suppose four hundred seems fewer people than you need
for something as important as this, right when you consider
the huge numbers that are employed by big tech companies,
you would think that I CAN would have.
Speaker 2 (35:56):
More than Well, the un has thirty six thousand people
working for it, and this is probably the size that like,
this is a probably a little bit more important than
the u N if you really think about it.
Speaker 1 (36:06):
Yeah, the website would not work without these things, and
so that is the.
Speaker 2 (36:10):
Email would email break as well?
Speaker 1 (36:13):
That's a good question. I think it would be.
Speaker 2 (36:16):
If you were accessing through through the webit world.
Speaker 1 (36:19):
Yeah, and also presumably I would I don't fully know
the answer, but I would presume that actually, yeah, because
you're putting in a a kind of domain name something
at something dot com or dot co, dot UK or
dot net or whatever, that it would still be routed
through the same systems.
Speaker 2 (36:36):
So yeah, And a quick Google says that that's the
case too. This is how I learned things. And also
another website I wonldn't be able to access the DNS
was down. That's that's so good.
Speaker 1 (36:46):
Yeah, exactly, this is the thing you would sometimes It's
happened when I used to work prior to journals, I
used to work in an office and sometimes the like
the actual router would fail and you would just kind
of be stuck there twiddling your thumbs and things, Well,
what can I do? Like imagine that but everybody in
the world, all at once unable to do the most
(37:09):
basic stuff. And think about how reliant we are on
all of the the internet connected services and tools that
we use, and then think about what would be the
impact if all of those stopped suddenly and we didn't
know what to do afterwards.
Speaker 2 (37:24):
And it says here in your article. There's thirteen of
the largest DNS servers a run by i CAN. So
three of the three of the thirteen largest run by
i CAN. So if you took if someone took out
I CAN, it would still function, but I imagine there'd be
a massive outage just kind of connecting the bits.
Speaker 1 (37:42):
Yeah, so they all have different route service, which is
kind of like the they have kind of the original
phone books as it were. You can get copies of
copies of copies of copies, which are increasingly less reliable.
It kind of generally seems to work geographically, so it
would probably affect parts of of the world rather than
the entire world, depending on which way you were served
(38:05):
through in terms of which quote unquote phone book you got,
which root server. But it's it's kind of a huge issue,
and the problem is we don't fully understand and wouldn't
fully understand until it happened, what the impact could be
because we know, okay, if it affected those three servers,
(38:27):
those three root servers, fine, but is there something on
the other root servers, or the websites or the back
ends of the organizations that operate the other root servers
that relies on those root servers to get access to
Like it's kind of could the domino effect start to
(38:48):
play out here where actually one pretty significant error anyway
could actually spread further and further and further.
Speaker 2 (38:56):
It's yeah, it's almost as if everyone It's would be
like if they one forgot how to speak. Yeah, you
could perhaps write letters, but speaking was off the table.
It's terrifying. And again, three of them are held up
by nonprofits. It's which is good, but all of them
should be. It's so strange. As countries we can all
(39:18):
get together to go to war or help support a
war perhaps, but we genocide, I guess in that case.
But we are in this situation where it's fucking we
can't put the money together to support the literal way
that people communicate online.
Speaker 1 (39:36):
Which is because we get through Yeah.
Speaker 2 (39:40):
It must works right now.
Speaker 1 (39:42):
Yeah, it works right now, and when it goes wrong,
it hasn't gone completely wrong one hundred percent all over
the world, and so we kind of go, well, that's
a whoopsie, Okay, we can deal with it and move on.
Hopefully it won't happen again. Fingers crossed. Let's hope that's
all okay, and that the way that it's kind of working.
That's the status quote a minute.
Speaker 2 (40:04):
I guess there's nothing. It's one of these I like
to end episodes by being like, what can regular people do?
It doesn't feel like we can.
Speaker 1 (40:10):
And you have read some of my journalism, you know
that I'm one of the most pessimistic people. I do
a radio slot here in the UK where I introduce
tech stories to people who don't necessarily know lots about tech,
and every single week I get harangued by the hosts
because I always end with depressing notes, and unfortunately that
is the case here.
Speaker 2 (40:29):
I think, Yeah, I think that something I like to
come back to though, is knowledge is power. I think
that I wonder if there is This is one of
the dumber things I've thought up, but I wonder if
there is actually a way of most people downloading the
phone book of DNA with DNS phone book just distributed.
Speaker 1 (40:46):
It could be crypto far off. I can be on
a blockchain.
Speaker 2 (40:50):
Bit of crypto. And it is funny as well, because
you see all of this AI bullshit and you got
the crypto bullshit and they're like, yeah, this is the future.
This is so cooled important. Objectively, DNA is cool, like
this stuff is that it's actually insane. The Internet works
at all.
Speaker 1 (41:04):
Yeah, Like I wrote a book called The History of
the Internet in bite sized Trucks, And as I said,
I'm thirty five. I kind of I joined the Web
when I was about ten or eleven, and yeah.
Speaker 2 (41:19):
I'm thirty eight, by the way, so we're right there.
Speaker 1 (41:21):
They go kind of got interested in it and found
it incredible, but forgot that, like I lost that wonder
because we don't see how it works anymore. You don't
see the crankshafts, you don't see the gears working in
the way that you used to. Kids nowadays don't know
how to store files on a computer because they just
(41:42):
have cloud storage. It's just always accessible easily there. They
don't have to structure a file system or something like that.
And so we take it for granted that these things
work and we just assume that like it's all okay.
But actually, yeah, knowledge is power. And knowing that there
is a person behind this, Knowing that there is a
system behind this and kind of getting a sense a
(42:03):
little bit of how it works means that you understand
more perhaps when things go wrong, and importantly, you can
kind of advocate maybe for how to make sure that
it doesn't go wrong again in the future.
Speaker 2 (42:17):
Chris, thank you so much for joining me. Where can
people find you?
Speaker 1 (42:20):
They can unfortunately find me on X that stokel that
is my am. I'm going down with that ship long
li SD Okay.
Speaker 2 (42:30):
Yeah, all right, thank you so much. Chris. You've been
listening to Better Offline. You know where to find me.
There's the same thing that comes on after it that
you'll complain because they haven't changed it in a wow,
thank you for listening everyone, and then it's gonna say
thank you for listening again. Thank you for listening to
(42:51):
Better Offline.
Speaker 3 (42:52):
The editor and composer of the Better Offline theme song
is Metasowski. You can check out more of his music
and audio projects at Metasowski dot com, M A T
T O S O W s ki dot com. You
can email me at easy at Better Offline dot com,
or visit Better Offline dot com to find more podcast
links and of course my newsletter. I also really recommend
(43:14):
you go to chat dot where's youreaed dot at to
visit the discord, and go to our slash.
Speaker 2 (43:18):
Better Offline to check out our reddit. Thank you so
much for.
Speaker 3 (43:21):
Listening Better Offline is a production of cool Zone Media.
For more from cool Zone Media, visit our website cool
zonemedia dot com, or check us
Speaker 1 (43:30):
Out on the iHeartRadio app, Apple Podcasts, or wherever you
get your podcasts.
Host
Ed Zitron
Popular Podcasts
24/7 News: The Latest
Today’s Latest News In 4 Minutes. Updated Hourly.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.