230: Maybe Gentoo Was Right All Along

00:00SPEAKER- Okay, Brad, yesterday we went to Great America

00:02SPEAKERand I finally did the thing I've been meaning to do

00:03SPEAKERfor a long time.

00:05SPEAKER- Took a bath in cotton candy?

00:07SPEAKER- Well, okay, there was a little bit of cotton candy

00:09SPEAKERand there were churros consumed.

00:11SPEAKER- Good.

00:12SPEAKER- They were dipped in goo.

00:14SPEAKER- What kind of goo?

00:14SPEAKERHang on.

00:15SPEAKER- Strawberry today.

00:16SPEAKER- Okay.

00:17SPEAKER- Sometimes it's Bavarian cream,

00:18SPEAKERsometimes it's strawberry.

00:19SPEAKERRight now, because it's Peanuts Festival for some reason,

00:22SPEAKERthey have a red velvet/sour cream cheese icing churro

00:29SPEAKERthat they make, but only apparently on weekday food

00:32SPEAKERat the park, not anywhere near as wild

00:34SPEAKERas weekend food at the park, we learned.

00:36SPEAKER- I believe that.

00:37SPEAKERI'll take whatever goo I can get.

00:38SPEAKERAnyway.

00:39SPEAKER- So for people who don't know,

00:40SPEAKERGreat America is like a theme park.

00:42SPEAKERIt's down south towards San Jose

00:46SPEAKERon the bay side of the peninsula.

00:48SPEAKERAnd it's like, I don't wanna call it small,

00:51SPEAKER'cause it has some absolute banger roller coasters,

00:54SPEAKERbut it's like the small theme park.

00:56SPEAKERIt's like the roller coasters are mostly fairly short.

01:00SPEAKERTheir intensity and excitement levels may be high,

01:03SPEAKERbut it's like, if you were playing Roller Coaster Tycoon,

01:06SPEAKERit's like, how much stuff can you pack

01:07SPEAKERinto this 20 acre area?

01:09SPEAKERNot, we're going to Ohio and land is cheap and free,

01:12SPEAKERso we're gonna expand out as big as we can get.

01:14SPEAKER- It's your classic regional park.

01:16SPEAKER- Yeah, it's a good one of those.

01:19SPEAKERAnd it's been around for a long time.

01:21SPEAKERIt's like tickets are cheap.

01:23SPEAKERIt's a fun thing to do on like a weekday,

01:26SPEAKERthe last day of spring break, so we went down.

01:28SPEAKERAnd often when we go, we go in the evenings

01:30SPEAKERor on like the holiday days and not all the rides are open,

01:33SPEAKERincluding, it's worth mentioning,

01:36SPEAKERit's right next door to EA's Redwood Shores offices too.

01:39SPEAKER- Oh, I know where that is, yeah.

01:41SPEAKER- Yeah, one of the things that they have there

01:44SPEAKERis the Mass Effect ride, which has been there for,

01:48SPEAKERI assume a fairly long time now.

01:50SPEAKERIt's like a 4D, 5D theater thing where like,

01:53SPEAKERyou put on the glasses and you sit down

01:56SPEAKERand there's chairs that like shake and move and rattle

01:59SPEAKERand the screen's pretty good size.

02:01SPEAKERIt's not like IMAX big, but it's like 60 or 70 feet.

02:03SPEAKERIt's pretty big.

02:04SPEAKERAnd like kids today, yesterday, I've never pushed them to go

02:08SPEAKER'cause like my kids are, my kids young enough

02:10SPEAKERand her friends are young enough that they don't,

02:11SPEAKERlike they don't know what Mass Effect is.

02:13SPEAKER- It's been a minute and a half since Mass Effect

02:16SPEAKERhas been what I would describe as relevant, but anyway.

02:18SPEAKER- Yeah, like, I was like, hey, this is a video game.

02:22SPEAKERShe's like, what kind of video game?

02:22SPEAKERIt was like, 'cause it's an Xbox 360 video game.

02:24SPEAKERShe was like, was that the last one?

02:26SPEAKERI was like, no, it's the one before that one.

02:28SPEAKERShe's like, oh, unimpressed.

02:30SPEAKERSo, but they were, today they were like,

02:31SPEAKERhey, let's go ride the Mass Effect ride.

02:33SPEAKERThat would be fun.

02:34SPEAKERAnd I was like, okay, we can go because we were walking by

02:36SPEAKERand they were letting people in so we could just walk

02:37SPEAKERright on.

02:38SPEAKERWe walk in, we get in the theater.

02:40SPEAKERThe seats are pretty, it's like big deal.

02:43SPEAKERWhen we talked about Great America last time,

02:45SPEAKERI got multiple questions from people that were like,

02:46SPEAKERhey, did you ride the Mass Effect thing?

02:48SPEAKERIt's supposed to be pretty cool.

02:49SPEAKERSo you go into this big theater.

02:51SPEAKERThere's like big super D-box chairs

02:53SPEAKERwhere like they're clearly like gonna rock and rattle

02:56SPEAKERand move around like crazy.

02:57SPEAKERThere's underneath in front of all the seats,

03:00SPEAKERthere's like the thing that you used to step on

03:02SPEAKERto go into the supermarket door

03:03SPEAKERthat would open the automatic door for you.

03:06SPEAKERSo I assume if somebody like falls out of their chair,

03:08SPEAKERthen it stops moving the seats so they don't get crushed.

03:11SPEAKERIt has little puffers that blast air

03:14SPEAKERon the backs of your legs and like the back of your neck.

03:17SPEAKERAnd it has like full on actuators in the seat

03:20SPEAKERthat like pound you in the back, like a massage chair almost.

03:23SPEAKERAll right, I like a good neck puffer.

03:25SPEAKERThis sounds good.

03:26SPEAKERYeah, I mean, who doesn't like in their neck puffed?

03:28SPEAKERSo there's also a guy who's standing

03:31SPEAKERat the front of the auditorium in like a Mass Effect costume,

03:35SPEAKERnot armor, but like a civilian on the Citadel.

03:38SPEAKEROh, a little jumpsuit.

03:40SPEAKERYeah, like a jumpsuit-y kind of vibe.

03:42SPEAKERHe's a little bit of a jacket on, he's wearing glasses.

03:44SPEAKERHe's a headset mic that looks like a normalized headset mic

03:48SPEAKERthat maybe put a little bit something clothes on

03:49SPEAKERso it looks a little science fiction-y.

03:52SPEAKERAnd he's there to get you hyped up

03:53SPEAKERabout the universe of Mass Effect.

03:55SPEAKERAnd how hyped did you get?

03:57SPEAKERI was, I, look.

03:58SPEAKERWas he into it?

03:59SPEAKERHe was, he was very, he was working hard.

04:02SPEAKEROkay.

04:03SPEAKERAnd it was, it was a,

04:03SPEAKERit was the most unfriendly crowd I've ever seen.

04:06SPEAKER'Cause he was like, we're getting on,

04:07SPEAKERwe're getting on our spaceship.

04:08SPEAKERIt's a decommissioned blah, blah, blah class.

04:11SPEAKERJust like Commander Shepard.

04:14SPEAKERPause for cheers.

04:17SPEAKERNo reaction from the audience.

04:18SPEAKERWow, wow.

04:20SPEAKERI would guess, if I had to make a guess,

04:22SPEAKERI would say I was the only person

04:23SPEAKERin the theater of 45 or 50 people

04:26SPEAKERthat was aware of Mass Effect as a franchise.

04:29SPEAKERAnd you were screaming your head off.

04:30SPEAKERNo, I was just like,

04:31SPEAKERI was looking around to see if anybody else

04:33SPEAKERwas reacting at all.

04:34SPEAKERMan, that's rough.

04:35SPEAKERSo, but it, but it like, it did not deter him.

04:38SPEAKERHe's like, so he's standing there, he has like consoles,

04:40SPEAKERhe's like handles to pull and buttons to press.

04:43SPEAKERAnd he's like, you know, we're,

04:45SPEAKERtoday we're gonna use the mass relay system.

04:47SPEAKER'Cause the whole story is,

04:48SPEAKERspoilers I guess for this 10, 15 year old theme park ride

04:52SPEAKERthat's only in California.

04:54SPEAKERYou're getting on a ship

04:55SPEAKERand you're gonna go to a resort planet

04:58SPEAKERthrough the mass relay at Cheron.

05:00SPEAKERYou know, the one that you use at the start of the game.

05:02SPEAKERAnd then you're gonna like, just as you get there,

05:06SPEAKERthere's an invasion happening.

05:08SPEAKERThe bad invasion is happening

05:09SPEAKERand things are going down.

05:10SPEAKERIt seems like Mass Effect 3 timing.

05:12SPEAKEROkay, were they Reapers or the Geth?

05:14SPEAKERI think they were Reapers.

05:15SPEAKERIt was the big tripod guys.

05:17SPEAKER- Yes.

05:18SPEAKER- Those seem like Reapers, right?

05:19SPEAKER- Space cuttlefish.

05:21SPEAKER- Yeah, kind of. - Yes, yes.

05:22SPEAKER- So we're like, we're doing the whole thing

05:25SPEAKERand we fly around and the Normandy zoops out in front of you.

05:30SPEAKERYou get like, there's like an Eevee lady up there

05:33SPEAKERthat's telling you what's going on.

05:36SPEAKEREverything of course is super 3D

05:37SPEAKERbecause you have your glasses on.

05:40SPEAKERHe shouts out the Normandy, he shouts out the Citadel,

05:43SPEAKERhe's trying real hard, like side characters come in

05:45SPEAKERand pop up on our radio HUD as like the ship is going,

05:48SPEAKERas the flight's going bad.

05:50SPEAKERAnd the whole time, no reaction from anybody in the room

05:54SPEAKERabout like, it's always the pause, no reaction.

05:57SPEAKER- That's harsh man. - Consistently.

05:59SPEAKERLike usually people are primed to be excited

06:01SPEAKERabout stuff at theme parks, right?

06:02SPEAKERLike you're hopped up on junk food,

06:04SPEAKERyou've been riding exciting rides all day.

06:06SPEAKERLike it's hot, sweaty.

06:08SPEAKERYou're suddenly in a cool air conditioned room

06:11SPEAKERand like, just dead stares every time.

06:16SPEAKERBut like to his credit, he did the pause every time, right?

06:21SPEAKERHe had a whole bit about how he had to wear glasses

06:23SPEAKERbecause he stared at the sun

06:24SPEAKERwithout his special protective glasses on.

06:26SPEAKER- Topical.

06:27SPEAKER- Yeah, and like he was going for it

06:30SPEAKERand just no love from anyone.

06:33SPEAKERI'm gonna say, fell flatter than Mass Effect Andromeda.

06:36SPEAKER- Ouch, that's, boy, that was both too low and too soon.

06:41SPEAKER(laughing)

06:43SPEAKER- So anyway, Mass Effect ride,

06:45SPEAKERI'm gonna give it a skip probably.

06:46SPEAKERI'd rent that one maybe.

06:48SPEAKER- Were there any, I have to know,

06:49SPEAKERwere there any my favorite blank on the Citadel jokes?

06:53SPEAKER- 100%, he does say he 100% had a,

06:57SPEAKERit's Commander Shepard's favorite blah, blah, blah

07:00SPEAKERon Terra Nova. - Great.

07:02SPEAKER- He shouted out the Citadel.

07:04SPEAKERHe shouted out, not grunt, what was the first Krogan?

07:07SPEAKER- Oh, Rex. - Rex.

07:08SPEAKERRex shows up. - Shepard.

07:10SPEAKER- Yeah, what's his name, the short-lived guy,

07:15SPEAKERthe scientist guy doesn't show up, the genocider.

07:17SPEAKER- Oh, Morden.

07:18SPEAKER- Morden, Morden doesn't show up.

07:20SPEAKER- Oh, well, he's gone.

07:21SPEAKER- But yeah, it was, I'm glad we did it.

07:23SPEAKERThe kids were completely underwhelmed

07:26SPEAKERand I was like, hey, did you enjoy that?

07:28SPEAKERThey were like, no, there were scary monsters

07:30SPEAKERon the screen all the time.

07:32SPEAKERAnd also they didn't like getting punched in the back

07:36SPEAKERby the seats.

07:39SPEAKERThey expect the seats to be a passive part of the ride

07:41SPEAKERand not an active part of the ride.

07:43SPEAKERAnd they were like, the guy kept waiting

07:44SPEAKERfor reactions from the audience.

07:47SPEAKERLike they picked up on the guy dying slowly,

07:50SPEAKER30 times a day as he pitched this to the audience.

07:54SPEAKERSo I'm gonna say, if you wanna ride the Mass Effect ride

07:57SPEAKERat California's Great Adventure, Great America,

08:00SPEAKERprobably you should go soon.

08:03SPEAKER- It's a ride whose days are numbered.

08:06SPEAKER- Yeah, it's not long for the world.

08:08SPEAKERThe Reapers are coming.

08:10SPEAKER(upbeat music)

08:38SPEAKER- Welcome to Brad and Will Made a Tech Pod, I'm Will.

08:40SPEAKER- I'm Brad, hello.

08:42SPEAKER- Hi, hello.

08:43SPEAKERSo after I got back from New Terra,

08:45SPEAKERwe went and rode Flight Deck,

08:46SPEAKERwhich used to be the Top Gun roller coaster.

08:49SPEAKERIt's a hanging coaster, so your feet dangle

08:51SPEAKERand you like go the outside of the loop.

08:53SPEAKERThat was pretty good.

08:54SPEAKERIt washed the bad Mass Effect taste out of both their mouths.

08:56SPEAKER- That's good.

08:57SPEAKERSpeaking of regional theme parks,

08:58SPEAKERI watched a documentary on the plane out here

09:01SPEAKERcalled Closed for Storm.

09:03SPEAKER- Oh.

09:04SPEAKER- About a regional theme park in the New Orleans,

09:07SPEAKERgreater New Orleans area called Jazzland.

09:10SPEAKER- Oh, that sounds extremely regional.

09:12SPEAKER- It has been abandoned since 2005.

09:15SPEAKER- Wow.

09:16SPEAKER- Abandoned, gone to sea, just like sitting there.

09:19SPEAKER- Was this a defunct land thing?

09:20SPEAKER- No, it's just a documentary made by some filmmaker.

09:23SPEAKERI don't know, didn't seem to be affiliated with anything,

09:27SPEAKERbut I felt kind of like an idiot.

09:29SPEAKERIt was 20 plus, maybe 30 minutes into the movie

09:32SPEAKERbefore I realized why it has been abandoned since 2005.

09:36SPEAKER- It was Hugo, right?

09:38SPEAKER- Or do you remember something?

09:39SPEAKERDo you remember a little weather event

09:40SPEAKERcalled Hurricane Katrina?

09:41SPEAKER- Yeah, it was when we flooded all of New Orleans

09:44SPEAKERand millions of people had to evacuate

09:46SPEAKERand people were fighting alligators and stuff.

09:48SPEAKER- A lot of people died and lost their homes

09:49SPEAKERand all kinds of other horrible stuff.

09:50SPEAKER- It was real bad.

09:52SPEAKER- Well, the park was another casualty of that storm

09:55SPEAKERand the levees breaking all that stuff.

09:56SPEAKERAnyway, the documentary is okay.

09:58SPEAKERIt's a little slow and uneven.

10:00SPEAKERI think it could have been tightened up in places,

10:02SPEAKERbut it's still kind of interesting.

10:03SPEAKERAnd they do a lot of urban exploring

10:04SPEAKERof what's left of the park,

10:05SPEAKERwhich has been sitting like I said for the better part

10:07SPEAKERof 20 years now.

10:08SPEAKER- Sounds like a real good video game setting.

10:11SPEAKER- Yeah, sure.

10:12SPEAKERMaybe like a Left 4 Dead campaign or something

10:14SPEAKERin a theme park.

10:17SPEAKER- Did you ever go to Carowinds when you were a kid?

10:19SPEAKER- Yes, Carowinds was the place.

10:21SPEAKERIt was like two hours away.

10:23SPEAKER- It was a longer drive for us,

10:24SPEAKERbut we went to Carowinds a couple of times when I was a kid.

10:26SPEAKER- Was that still the closest park for you even in Tennessee?

10:29SPEAKER- No, the one with Tweetsie Railroad at the mile high

10:34SPEAKERon 40 between Knoxville and Asheville.

10:38SPEAKER- Okay.

10:39SPEAKER- I can't remember what that one's called.

10:40SPEAKER- It feels like Tennessee has more parks

10:43SPEAKER'cause you've also got Dollywood.

10:45SPEAKER- Yeah, Dollywood was two hours away for us,

10:47SPEAKERbut it was Silver Dollar City when I was a kid still.

10:50SPEAKER- Okay.

10:51SPEAKER- Once it became Dollywood, they put more rollercoasters

10:54SPEAKERand like log flumes and stuff in.

10:55SPEAKERWhen I was a kid, it was just like a log flume

10:57SPEAKERand a kind of shitty rollercoaster.

10:59SPEAKER- Can't have enough log flumes.

11:01SPEAKER- Look, my favorite-

11:02SPEAKER- That's my stance.

11:03SPEAKER- My favorites are those ones that it's like a giant

11:05SPEAKERinner tube with like five chairs in that you strap in,

11:08SPEAKERyou're all spinning around like crazy.

11:09SPEAKERThose are great.

11:10SPEAKER- That's a good time.

11:11SPEAKER- Yeah.

11:12SPEAKER- But yeah, I didn't have a frame of reference

11:13SPEAKER'cause like I said, we didn't go to other parks,

11:15SPEAKERbut Carowinds was absolutely the thing

11:17SPEAKERto as far as going to theme parks for us.

11:20SPEAKER- Yeah, I think we only went when I was really,

11:22SPEAKERreally little and I don't remember a whole lot of it,

11:27SPEAKERbut yeah, the East Coast theme park

11:30SPEAKERand the West Coast theme park,

11:31SPEAKERlittle bit different, I'm gonna say.

11:33SPEAKER- Yeah, sure.

11:34SPEAKER- Food's still bad everywhere.

11:36SPEAKER- But like a lot of other things between the two coasts.

11:38SPEAKER- Yeah, I think if Tweetsie Railroad

11:39SPEAKERis the thing I was thinking about.

11:41SPEAKER- Yeah, I had like saw commercials and heard that name a ton

11:43SPEAKERwhen I was a kid, but we never went.

11:45SPEAKER- I think that's the place that they had a rollercoaster

11:48SPEAKERlike break off of the rail,

11:51SPEAKERmaybe that was at Carowinds actually.

11:52SPEAKER- Oh boy, oh boy.

11:53SPEAKER- Somebody recently had a rollercoaster,

11:55SPEAKERlike a kind of scary rollercoaster break loose

11:58SPEAKERand nobody was killed and they didn't,

11:59SPEAKERthey caught it before it became a problem, but like, yeah.

12:03SPEAKER- I used to, as a kid,

12:04SPEAKERI would go on any thrill seeking kind of ride.

12:06SPEAKERI'm not so sure I would do that at this point.

12:09SPEAKER- Do you look at the number of people that ride those rides

12:11SPEAKERand the number of injuries that they have

12:13SPEAKERand as a general rule, you're pretty safe.

12:16SPEAKER- Yeah, okay, fine, statistics.

12:18SPEAKER- The math is good there.

12:20SPEAKER- Statistics only means so much though

12:21SPEAKERwhen you draw the short straw.

12:24SPEAKER- I mean, look, I operate my final destination rules, man.

12:27SPEAKERIf it's my time, the Reaper is gonna get me regardless.

12:32SPEAKERIt's just how hilarious will it be when it comes?

12:35SPEAKER- Yes. - I put it off.

12:36SPEAKER- Yes, indeed.

12:37SPEAKER- Speaking of the Reaper coming,

12:39SPEAKERthis week we're talking about the XC Utils backdoor.

12:42SPEAKER- Yes, the open source world was rocked recently.

12:46SPEAKER- Yes. - I think it's fair to say

12:49SPEAKERby the insertion of a backdoor into a very common,

12:52SPEAKERI mean, very common doesn't begin to describe it.

12:54SPEAKER- Super ubiquitous.

12:56SPEAKER- This is beyond ubiquitous.

12:57SPEAKERIt is in literally everything.

12:59SPEAKER- Yeah, like the head unit in your car very likely

13:02SPEAKERhas this piece of software in it,

13:04SPEAKERnot the affected one, just to be clear.

13:06SPEAKER- Yes, to maybe to put it in common,

13:08SPEAKERthis is kind of analogous to like when,

13:12SPEAKERit's not a great analogy,

13:14SPEAKERbut WinZip or the zip format in Windows,

13:18SPEAKERit's just another type of compression.

13:20SPEAKERIt happens to be a type of compression

13:21SPEAKERthat's used more on the backend

13:23SPEAKERin various open source contexts.

13:25SPEAKERAlthough actually to put it in even more context,

13:27SPEAKERthe same compression routine that XC uses

13:31SPEAKERis also used in the 7-Zip format.

13:33SPEAKERAnd I'm sure plenty of people listening to this

13:34SPEAKERhave used 7-Zip before.

13:36SPEAKERThat is not to say that 7-Zip is affected in any way here.

13:39SPEAKERThat's not at all the case.

13:41SPEAKERI'm just trying to paint a picture of how common

13:43SPEAKERthis underlying technology is that has been attacked.

13:46SPEAKER- Yeah, so just to lead with the important stuff,

13:50SPEAKERit's only in very early, very new, very current versions

13:55SPEAKERof the XC utils package.

13:58SPEAKERSo if you're running, even if you're running Linux,

14:00SPEAKERif you're not running one of the unstable

14:03SPEAKERdevelopment branches, you're probably fine.

14:05SPEAKERAnd updating very regularly.

14:07SPEAKER- Yeah, I don't say this to be flip at all,

14:11SPEAKERbut if you have to ask if you're affected by this,

14:13SPEAKERyou are almost certainly not affected by this.

14:16SPEAKERLike you would know, like you would be so deep

14:17SPEAKERinto bleeding edge open source software

14:20SPEAKERthat you would already know basically

14:21SPEAKERthat you're at risk of this thing.

14:24SPEAKERAnd it's like barely been used in practice

14:27SPEAKERas far as we know, because it was discovered quite quickly

14:29SPEAKERafter it was fully inserted.

14:32SPEAKERSo yeah.

14:33SPEAKER- So yeah, there's that.

14:34SPEAKERIt's specifically the 5.6.0 and 5.6.1 versions of XC utils.

14:40SPEAKERIt provides lossless compression

14:42SPEAKERthat's analogous to gzip and zip.

14:45SPEAKERIt's not like a lossy compression.

14:46SPEAKERIt's not for images, stuff like that.

14:47SPEAKERIt's mostly used to like distribute packages on Linux

14:52SPEAKERin a compressed format, stuff like that.

14:54SPEAKER- Yeah.

14:55SPEAKER- They hadn't made it into the mainline distros yet.

14:57SPEAKERSo like Brad said, you would know if you have this problem.

15:03SPEAKERWe don't really know what it did,

15:05SPEAKERwhat it was supposed to do.

15:06SPEAKERWe know that it gave access,

15:08SPEAKERonce it was fully activated on affected systems,

15:11SPEAKERit would have potentially given attackers access

15:14SPEAKERto the SSH-D process if it was running in the right folder.

15:19SPEAKERAnd it gives them the ability

15:22SPEAKERto bypass normal authentication on the machine

15:24SPEAKERso that people could remotely execute code,

15:27SPEAKERwhich just to be clear, that's bad.

15:29SPEAKER- That's extremely bad.

15:30SPEAKERCan I just stop for a second

15:32SPEAKERand reflect on the incredibly clandestine nature

15:35SPEAKERof this whole thing?

15:36SPEAKERLike the fact that you can sit there and say,

15:38SPEAKERwe don't know exactly what this thing does.

15:41SPEAKERLike people are investigating and figuring it out,

15:43SPEAKERbut as of right now,

15:44SPEAKERwe don't know the entire scope of like the attack surface

15:47SPEAKERor like what it was capable of.

15:49SPEAKERWe don't know if there are other types of back doors

15:51SPEAKERthat this malicious attacker or other,

15:54SPEAKERthere could be other exploits elsewhere

15:56SPEAKERthat have been discovered as of right now.

15:59SPEAKERThere might be other instances of this happening

16:02SPEAKERperpetrated by entirely different attackers

16:04SPEAKERthat also have not been discovered yet.

16:06SPEAKERProbably not, but maybe.

16:08SPEAKER- It's a threat surface that I think security people

16:11SPEAKERhave been aware of and have considered,

16:12SPEAKERbut haven't really, since nobody's exploited it

16:15SPEAKERprior to this that we're aware of,

16:18SPEAKERI think people haven't treated it as a serious threat

16:21SPEAKER'cause it's a fairly involved,

16:23SPEAKERand as always is the case, it's like technical stuff

16:26SPEAKERand social engineering stuff all working together.

16:28SPEAKERSo it's really interesting to kind of unpack what it does.

16:31SPEAKERThe other, to your point though,

16:33SPEAKERthat we don't really know what it did,

16:34SPEAKERonce you get to the point that you can remotely execute code

16:37SPEAKERon a machine, it kind of doesn't matter.

16:38SPEAKER- Yeah, yes, you've already got full access.

16:41SPEAKER- Yeah, you essentially have full access to the machine.

16:43SPEAKERYou just wget or whatever software you want, run that,

16:47SPEAKERand then you can do whatever you want

16:49SPEAKERwith somebody else's machine,

16:50SPEAKERwhether that's install a key logger,

16:52SPEAKERor join a giant botnet,

16:55SPEAKERor make the centrifuges spin in such a way

17:00SPEAKERthat they'll permanently damage themselves, right?

17:02SPEAKER- Yeah, or covertly dump all data

17:05SPEAKERthat the machine has access to,

17:07SPEAKERlike some giant database of sensitive information,

17:09SPEAKERlike the sky is probably the limit,

17:11SPEAKERas long as you don't get caught by something

17:14SPEAKERlike increased CPU overhead or latency,

17:16SPEAKERor network activity or something,

17:18SPEAKERwhich we'll get to in terms of how this was discovered, but.

17:21SPEAKER- Yeah, but presumably being able to remotely execute code

17:24SPEAKERopens up the entire realm from ransomware

17:27SPEAKERto destroying hardware to like,

17:29SPEAKEReverything is possible at this point.

17:31SPEAKER- Yeah, yeah, we'll get into what sort of slice

17:34SPEAKERof the computing world this was targeting,

17:35SPEAKERbecause they went after like probably the most high value

17:38SPEAKERwhen they could have, if they had to pick one.

17:40SPEAKER- Yeah, so the good news though,

17:42SPEAKERis that we found a Microsoft employee

17:45SPEAKERthat works on Postgres, the SQL, their SQL server,

17:51SPEAKERdiscovered it and like the reaction was pretty good,

17:56SPEAKERI think from the community and from the developers

17:59SPEAKERand from the distros and from the US government

18:01SPEAKERand everybody involved.

18:03SPEAKERBut what would have happened if it hadn't been found

18:08SPEAKERwhile it was still in this kind of early development distros,

18:11SPEAKERthe development testing distros,

18:13SPEAKERwhen it made it over to like a mainline

18:15SPEAKERor a long-term support version of say Debian

18:18SPEAKERand Fedora Red Hat and all that stuff.

18:20SPEAKER- Yeah, that was absolutely the intention here

18:22SPEAKERwas to slip it in.

18:23SPEAKERI mean, that's how you have to do it,

18:24SPEAKERbecause only bleeding edge distros

18:27SPEAKERare gonna be pulling in bleeding edge software.

18:29SPEAKERYou're gonna have to wait potentially years

18:30SPEAKERfor this to trickle down to the much more widely used stuff.

18:33SPEAKERSo that's why they went after what they did.

18:35SPEAKERSo before we get further into what this is

18:38SPEAKERand how people found it

18:39SPEAKERand all the kind of downstream repercussions of this,

18:41SPEAKERI think it's worth explaining for people who don't know

18:44SPEAKERlike what SSH is, for example.

18:46SPEAKER- Sure, SSH stands for secure shell.

18:50SPEAKERIt is a much more modern

18:51SPEAKERand as the name implies secure version of Telnet.

18:53SPEAKERIf you remember that from way back in the day,

18:55SPEAKERTelnet and then secure shell are just ways

18:57SPEAKERto remote into a server or some machine,

19:00SPEAKERnot at your location and get a command line there,

19:03SPEAKERget a command line session where you can run software

19:07SPEAKERat the command line as if you were sitting at that machine.

19:09SPEAKER- With elevated privileges to let you install software

19:11SPEAKERand do all sorts of other stuff as well, right?

19:13SPEAKER- If you've got the right credentials to log in

19:15SPEAKERas some kind of super user remotely,

19:16SPEAKERthen yeah, you can do the exact same things you could

19:18SPEAKERas if you were on that machine.

19:19SPEAKERAnd so SSH, the server that runs remotely

19:23SPEAKERthat you console into or session into,

19:27SPEAKERthat is the point of ingress of this backdoor.

19:30SPEAKERLike that's the thing that is actually compromised

19:32SPEAKERultimately that lets you, or lets attackers

19:35SPEAKERor would have let attackers use this backdoor,

19:38SPEAKERbut it's not SSH that was compromised.

19:39SPEAKERThat's the, it's kind of, if you don't really understand

19:42SPEAKERhow software is kind of put together hierarchically,

19:44SPEAKERespecially in the open source world,

19:45SPEAKERit might be hard to conceptualize,

19:47SPEAKERbut SSH is the thing people would have backdoored through

19:50SPEAKERin terms of network connectivity,

19:51SPEAKERbut it's not the software that was compromised.

19:54SPEAKER- That gave them access.

19:55SPEAKER- Right.

19:56SPEAKER- This xzutils package is what gave people access.

19:59SPEAKERIt just affected SSH on the affected systems.

20:01SPEAKER- Right.

20:02SPEAKER- And that's 'cause software is linked together.

20:03SPEAKERLibraries are linked together.

20:04SPEAKERYou know, like software is, that's what dependencies are.

20:07SPEAKERThat's the classic xkcd thing with the Jenga stack.

20:10SPEAKERYou know, like the dependencies are the little bricks

20:13SPEAKERat the bottom, which are the component software

20:15SPEAKERthat other programs like SSH might rely on

20:18SPEAKERin different contexts.

20:19SPEAKER- So for example, in the Windows world,

20:22SPEAKERwe have similar ideas 'cause often you have to install

20:24SPEAKERlike Visual C runtimes or something

20:25SPEAKERlike that to make an application work.

20:27SPEAKERBut additionally, if you're, say,

20:29SPEAKERif you wanna use cURL functionality in a Windows program,

20:33SPEAKERand it's like a full installed desktop Windows program,

20:36SPEAKERyou probably just embed cURL in your executable program

20:41SPEAKERthat you have built there.

20:42SPEAKEROn Linux, instead of doing that, you actually just say,

20:45SPEAKERhey, before this is all the way installed,

20:48SPEAKERyou have to have cURL on this machine,

20:50SPEAKERand then your application just reaches out to cURL

20:54SPEAKERwhen it needs to do cURL stuff,

20:55SPEAKERand you don't have to have separate versions.

20:57SPEAKERYou don't have to have a version of cURL

20:59SPEAKERembedded in the application.

20:59SPEAKERIt's one of the reasons why often Linux applications

21:02SPEAKERare really, really tiny compared to, say,

21:04SPEAKERa giant Windows application.

21:05SPEAKER- Right.

21:06SPEAKER- And there's pros and cons to each solution,

21:08SPEAKERand we'll talk more about this in a little bit,

21:10SPEAKERbut it's important to understand that if you use xzip

21:15SPEAKERon your application on Linux,

21:16SPEAKERyou probably don't package up a branch of xzip

21:19SPEAKERinside your distributable file.

21:21SPEAKERYou're probably just saying, hey, make sure that,

21:23SPEAKERyou're telling the OS, hey, make sure xzip is installed

21:26SPEAKERso that I can run 'cause I need that to work.

21:28SPEAKER- Yes.

21:29SPEAKER- Okay, so what happens here if this programmer

21:35SPEAKERdidn't discover this problem?

21:36SPEAKER- Yeah.

21:37SPEAKER- If two years down the line,

21:39SPEAKERsome bad actor has low-level access to millions

21:43SPEAKERor billions of Linux and potentially Mac OS and iOS

21:47SPEAKERand even maybe Windows in some cases, machines,

21:51SPEAKERlike, what does that mean?

21:52SPEAKER- It means a lot of really bad things

21:54SPEAKERfor the computer security world.

21:56SPEAKERWhere they started though,

21:57SPEAKERI think we should talk about where they started

21:58SPEAKERbecause like practically every computer on the planet

22:00SPEAKERprobably has some amount of XE code in it.

22:02SPEAKER- Yeah.

22:03SPEAKER- Maybe the exception of some windows,

22:04SPEAKERbut it's like we said, it's everywhere in the Unix world.

22:07SPEAKERBut as of now, as far as we know,

22:09SPEAKERand there are gonna be a lot of as far as we knows

22:11SPEAKERin this episode, because this is all based on like

22:13SPEAKERwhat has been uncovered in the last like two weeks

22:14SPEAKERsince this was found.

22:15SPEAKER- Three now, yeah.

22:16SPEAKER- Or three.

22:17SPEAKERSo far as we know right now,

22:19SPEAKERthe actual backdoor could only actually be activated

22:22SPEAKERon a subset of a subset of a subset

22:24SPEAKERof Linux machines out there.

22:25SPEAKERSo what I mean is like Mac OS has XE code and a FreeBSD,

22:29SPEAKERlike all the Unix likes out there, absolutely.

22:30SPEAKERWindows probably as well, it's everywhere,

22:33SPEAKERbut it wouldn't work on, for example, Mac OS

22:36SPEAKERbecause Mac OS lacks the other components they used

22:39SPEAKERin this kind of chained attack.

22:41SPEAKER- Yeah.

22:42SPEAKER- So like malicious code was in that library.

22:45SPEAKERIf it, and again, it probably that this bleeding ed version

22:48SPEAKERnever actually got installed in Mac OS.

22:49SPEAKERI don't know why I take that back.

22:51SPEAKERIt was on my laptop.

22:52SPEAKERHomebrew, which is the Homebrew is the most common package

22:55SPEAKERmanager for Mac OS absolutely had rolled out.

22:58SPEAKER- Oh, XE?

22:59SPEAKER- Yes.

23:00SPEAKER- Wow.

23:01SPEAKER- 5.6.0 version of XE, went and updated that quickly,

23:04SPEAKERbut the malicious code was in that version, for example,

23:07SPEAKERin Mac OS, but it was inert.

23:08SPEAKERYou know, it wouldn't actually do anything.

23:09SPEAKER- It needed the other part to connect to, right?

23:12SPEAKER- So in short, and I might be leaving a couple of pieces

23:15SPEAKERout here, but in short, the environment where this actually

23:17SPEAKERwould have been able to be activated is limited

23:20SPEAKERto Linux distros, Debian and Fedora, basically.

23:24SPEAKERFedora is part of Red Hat.

23:26SPEAKERFedora is kind of like the upstream version of Red Hat.

23:28SPEAKERLike if you've heard of Red Hat,

23:29SPEAKERit's huge in the enterprise world.

23:30SPEAKER- Yeah.

23:31SPEAKER- Debian is like the root of the probably the most common,

23:35SPEAKERpopular group of community-based distros

23:37SPEAKERin the Linux world.

23:38SPEAKERAnd so like, they basically went to like,

23:39SPEAKERif you would envision Jenga towers of Linux distros,

23:42SPEAKERthey went right to the bottom of like,

23:43SPEAKERit's probably the two most common ones, right?

23:45SPEAKER- Yeah, or if your Linux world is a tree,

23:47SPEAKERthen as a forest and each main distro branches is a tree.

23:52SPEAKER- Yes.

23:53SPEAKER- This is the root of the two biggest trees.

23:55SPEAKER- So I went and pulled a list on a distro watch,

23:57SPEAKERwhich is a very fun, very old fashioned site.

24:00SPEAKERThere are, they list 127 Linux distros

24:03SPEAKERthat are derived from Debian,

24:05SPEAKERjust to put it in perspective.

24:06SPEAKERAnd one of those is Ubuntu.

24:07SPEAKER- Yeah.

24:08SPEAKER- Which is certainly one of the most popular distros

24:10SPEAKERout there, but also MX, Linux Mints, Pop!OS,

24:15SPEAKERElementary OS, like those are all like big, widely used

24:18SPEAKERLinux distros that are all based on Debian.

24:20SPEAKERAnd then, like I said, what happens in Fedora

24:23SPEAKERtrickles down to Red Hat,

24:24SPEAKERwhich is used by tremendous numbers of corporations,

24:27SPEAKERacademic institutions, like-

24:28SPEAKER- Cloud services run on Red Hat, like it's, yeah.

24:31SPEAKER- And entities that can afford big

24:33SPEAKERRed Hat service contracts are gonna be using Red Hat.

24:36SPEAKERAnd there are Red Hat rebuilds

24:38SPEAKERbecause of the way the licensing works.

24:41SPEAKERYou can take all of Red Hat's code and rebuild it,

24:43SPEAKERwhich a couple of distros do.

24:45SPEAKERSo basically you're using Red Hat

24:46SPEAKERwithout Red Hat's corporate support, right?

24:48SPEAKERBut those are also very widely used

24:50SPEAKERin a lot of server contexts.

24:51SPEAKERSo long story short, they picked the two distros

24:54SPEAKERthat are going to encompass, like,

24:56SPEAKERI couldn't even tell you how much of the Linux world.

24:58SPEAKERLike there are other distros out there,

25:00SPEAKERlike Arch, that are unaffected by this,

25:02SPEAKERbut especially in like server contexts,

25:04SPEAKERthey picked, you know, like a massive attack surface

25:09SPEAKERby targeting these two distros.

25:11SPEAKER- It's a really small, it's a big initial snowball

25:14SPEAKERthat makes an even bigger avalanche,

25:16SPEAKERI think is the way to look at it.

25:18SPEAKER- Yeah, so the way these really buttoned up

25:20SPEAKERLinux distros work is that they do a lot of development

25:23SPEAKERand kind of like unstable versions

25:26SPEAKERthat are not really meant for end users

25:28SPEAKERor servers especially.

25:30SPEAKERBut down the line, as the packages get tested

25:32SPEAKERand, you know, bugs are fixed and whatever,

25:34SPEAKERthey declare like, okay, this version of this distro

25:37SPEAKERis now ready to go out wide as like a stable release.

25:39SPEAKERSo the ultimate intent was for this backdoor

25:41SPEAKERto make it into these stable releases

25:43SPEAKERthat then would be used on like every server

25:46SPEAKERon the planet practically, you know what I mean?

25:48SPEAKERSo if this had gotten out there

25:50SPEAKERand been undiscovered for years, like the damage,

25:53SPEAKERthe damage would have been incalculable,

25:54SPEAKERlike who knows what this could have been used for.

25:57SPEAKER- So just so that people have an idea

25:59SPEAKERof what the timeline is here,

26:01SPEAKERthere's more stuff that happened before we were talking,

26:03SPEAKERbut the first hidden backdoor binary merge

26:07SPEAKERhappened on the 23rd of this year,

26:10SPEAKER23rd of February of this year.

26:12SPEAKERAnd that was, to be clear, that was the first point

26:14SPEAKERat which this could actually be used, right?

26:16SPEAKERBecause I think this malicious actor

26:18SPEAKERhad been laying the groundwork for this for some time,

26:21SPEAKERbut my understanding is that the final piece

26:23SPEAKERwas only put in place around then.

26:24SPEAKER- This is the first code, yeah, this is the first code.

26:28SPEAKERThe end, the discovery by the Microsoft researcher

26:33SPEAKERwas on March 27th or 29th.

26:37SPEAKERYeah, March 28th, he discovers the bug,

26:39SPEAKERprivately notifies Debian and the distros,

26:43SPEAKERthe distros at open wall mailing list,

26:46SPEAKERand Red Hat assigns a ticket and the whole thing.

26:50SPEAKER- Yeah, it got a CVE, the CISA, the government agency,

26:54SPEAKERgot out there immediately talking about it.

26:56SPEAKER- The backdoor warning gets posted on the 29th of March,

27:00SPEAKERso it's been in the world for a month.

27:02SPEAKERIt's posted about on the 28th and 29th,

27:05SPEAKERand everything from that point on,

27:09SPEAKERlike Debian immediately rolled it back to,

27:12SPEAKERrolled the affected thing back to 5.4.5.1,

27:16SPEAKERwhich was the last unaffected version,

27:19SPEAKERand it becomes not an issue at that point

27:23SPEAKER'cause people are aware of it,

27:25SPEAKERword gets out incredibly fast, yada yada.

27:28SPEAKERShould we talk about the discovery?

27:30SPEAKER'Cause it's kind of interesting how the discovery happened.

27:32SPEAKER- Yeah, so it was Anders Freundt, I guess,

27:35SPEAKERis the name of the engineer at Microsoft

27:36SPEAKERwho works on Postgres, which is database software.

27:40SPEAKERI've kind of got a bone to pick

27:41SPEAKERwith the way that some people have talked about this,

27:43SPEAKERnot really people who are like in the know

27:45SPEAKERin the computing world,

27:46SPEAKERbut I feel like more like people

27:47SPEAKERwho were just sort of like peanut gallery

27:49SPEAKERchiming in on Twitter.

27:50SPEAKERWell, there are a lot of people,

27:51SPEAKERoh, sorry, let me back up.

27:52SPEAKERThe way he found this was doing some benchmarking,

27:54SPEAKERsome routine benchmarking,

27:56SPEAKERand noticed that SSH logins were taking a long time

27:59SPEAKERon his test machine,

28:00SPEAKERwhich was running the unstable development version of Debian.

28:02SPEAKER- It was like he noticed a lot of CPU utilization

28:05SPEAKERon the SSH D, which is the SSH server test, the daemon,

28:09SPEAKERwhich usually is really low CPU utilization.

28:12SPEAKER- Right, so he saw higher CPU usage

28:15SPEAKERand higher latency on these logins.

28:17SPEAKERAnd I saw some people out there

28:18SPEAKERwho were just going like,

28:19SPEAKERpeople were sort of like exoticizing this guy.

28:22SPEAKERIs that a word?

28:23SPEAKERKind of going like, oh, this is the internet's final boss.

28:25SPEAKERHe's the silverback gorilla.

28:27SPEAKERHe cared about 500 milliseconds of latency.

28:29SPEAKERAnd now the world has been saved.

28:31SPEAKERLike 500 milliseconds of latency is a long time

28:34SPEAKERin enterprise contexts and server operations.

28:37SPEAKERLike in the heavy computing,

28:40SPEAKERlike any amount of latency that shouldn't be there

28:42SPEAKERis where it bears investigation.

28:44SPEAKERIt's not like this guy was like so obsessive

28:47SPEAKERthat it was not some like independent stay situation, right?

28:50SPEAKERWhere he's the mad scientist who saved the world

28:52SPEAKERbecause he cared about something

28:54SPEAKERno person in their right mind would ever care about.

28:55SPEAKERLike this, as far as I know,

28:57SPEAKERis like pretty routine thing to look at.

28:59SPEAKER- Yeah, I mean, it's just normal bug fixing

29:03SPEAKERand like looking for, hey, here's,

29:05SPEAKERI'm having a problem with something else

29:06SPEAKERand happened to notice something else

29:07SPEAKERthat was weird at the same time.

29:09SPEAKER- Yeah.

29:10SPEAKER- It's the same as when you are like,

29:13SPEAKERhey, my video card is, my game is running slow.

29:16SPEAKERAnd then you realize that your SSD is running

29:18SPEAKERon PCIe 2.0 speeds instead of PCIe 4.0 speeds, right?

29:25SPEAKERSo on the 29th of February,

29:29SPEAKERthe assumption is that this attack was accelerated

29:31SPEAKERbecause of some changes that were happening

29:33SPEAKERto this libsystemd that would have probably

29:37SPEAKERdefeated the attack.

29:38SPEAKERSo that's possible.

29:39SPEAKERIt's possible that's why everything went fast

29:41SPEAKERand why there were some bugs and stuff

29:45SPEAKERthat caused like the systemd CPU utilization

29:47SPEAKERto go up and all that.

29:49SPEAKERThe reaction was, I think, pretty good from everybody.

29:53SPEAKERThe person who maintains this actual package

29:56SPEAKERwas on vacation.

29:58SPEAKERLassie Collin is the person's name.

30:00SPEAKER- Yeah, who was also the creator of the XC format.

30:03SPEAKERIt should be made clear.

30:05SPEAKERLZMA is the compression routine that XC uses.

30:08SPEAKERBut like I said, 7-Zip also uses that.

30:10SPEAKERLike XC is not the only format

30:11SPEAKERto use that type of compression.

30:13SPEAKERBut he created the XC format and is its sole maintainer.

30:16SPEAKER- Yeah, and has been since like 2008.

30:18SPEAKER- Right, like it is the classic example

30:21SPEAKERof that XKCD comic of literally one person

30:24SPEAKERmaintaining incredibly fundamental, important infrastructure

30:29SPEAKERwithout any help or pay.

30:30SPEAKER- For the entirety of the open source world.

30:32SPEAKER- Yes.

30:33SPEAKER- Okay, so the person discovered the attack

30:37SPEAKERat the end of March.

30:38SPEAKERHow did it actually happen?

30:40SPEAKERHow did we, like there's code reviews in process.

30:42SPEAKERThere's a bunch of stuff that's supposed to happen

30:43SPEAKERto prevent attacks from making it into distros

30:47SPEAKERthat are back doors like this, right?

30:50SPEAKER- You know, I can't say for sure

30:53SPEAKERdid the process break down here.

30:54SPEAKERI'm like, not that, certainly not that knowledgeable

30:56SPEAKERor authoritative here.

30:57SPEAKERBut I don't know if that's always the case.

30:59SPEAKERLike, I don't know that there is always

31:01SPEAKERactually somebody there reviewing everything

31:03SPEAKERperhaps as closely as it should be.

31:05SPEAKERThis was, you know, this vulnerability was slipped

31:07SPEAKERinto essentially source code releases

31:10SPEAKERthat were made specifically for these distros,

31:13SPEAKERnot what is publicly available on the Git repository.

31:16SPEAKER- Yeah, that's correct.

31:17SPEAKER- But at the root, really, this just comes down

31:20SPEAKERto one person maintaining this critical piece of software

31:23SPEAKERwho clearly looking now at mailing lists

31:24SPEAKERfrom the past two, three years,

31:26SPEAKERlike was incredibly burned out

31:28SPEAKERand became a target of opportunity for whoever went out

31:31SPEAKERlooking for a way to do something like this.

31:34SPEAKER- For me, the how did it happen wasn't somebody figured out

31:36SPEAKERa way to bypass the stuff.

31:39SPEAKERThe way it happened was one person was responsible

31:43SPEAKERfor this, an unpaid volunteer developer was responsible

31:47SPEAKERfor a piece of critical code,

31:49SPEAKERwhich is the way it happens every time.

31:51SPEAKERThe Heartbleed stuff was the same thing.

31:52SPEAKERIt wasn't, you know, a social exploit like this,

31:57SPEAKERbut like what happened was this malicious person,

32:00SPEAKERthe user whose name is supposedly Jia Tan.

32:03SPEAKERThat's, I mean, that's their name on GitHub.

32:04SPEAKERThat's their name in mailing lists and stuff like that.

32:07SPEAKERIt's unclear that that's almost certainly

32:09SPEAKERnot their real name.

32:12SPEAKEROver a period of like two and a half years,

32:15SPEAKERbuilt a relationship with the existing developer

32:18SPEAKERof this piece of software and started contributing

32:20SPEAKERin a way that literally no one else ever had

32:23SPEAKERin the history of this project.

32:24SPEAKER- Right, yeah, I think we should stress

32:26SPEAKERthat there were no inherent flaws in any piece of software

32:28SPEAKERinvolved in this entire chain that made this possible.

32:32SPEAKERIt was not existing bugs.

32:33SPEAKERIt was not like security flaws or poor design

32:35SPEAKERor anything wrong with any software that was out there.

32:38SPEAKERIt was this person infiltrating this project,

32:41SPEAKERgaining the trust that ultimately allowed them

32:44SPEAKERto make contributions to the project without any oversight.

32:47SPEAKERAnd then they slipped malicious code in there.

32:49SPEAKERSo again, this was not taking advantage of like poor work

32:52SPEAKERor anything like that on the part of anyone involved here.

32:56SPEAKERIt was entirely something they socially engineered

32:58SPEAKERtheir way into, not software engineered,

33:00SPEAKERI guess is the best way to put it.

33:02SPEAKER- Well, I mean, their software engineering as well,

33:04SPEAKER'cause the way they distribute it to the distros

33:06SPEAKERis fascinating.

33:08SPEAKER- Yeah, and a lot of the stuff is tremendously over my head,

33:11SPEAKERbut a lot of security researchers out there I have seen

33:13SPEAKERrefer to this as an incredibly clever attack.

33:15SPEAKERLike whoever perpetrated this

33:17SPEAKERextremely knew what they were doing on an engineering level.

33:21SPEAKERIt's interesting when we talk about code reviews

33:23SPEAKERand how this was distributed

33:24SPEAKERand who they targeted and et cetera, et cetera.

33:26SPEAKERLike there's a lot of conflicting views out there

33:28SPEAKERon is open source inherently weak to this kind of thing?

33:33SPEAKERIs it because of the way open source works

33:35SPEAKERthat this happened?

33:36SPEAKEROr is open source also uniquely positioned

33:38SPEAKERto respond to something like this

33:39SPEAKERin a way that closed source is not?

33:41SPEAKER- I think this is a situation

33:43SPEAKERwhere both of those things can be true, right?

33:45SPEAKERLike it's not impossible for a malicious actor

33:48SPEAKERto go get a job at Microsoft, right?

33:51SPEAKERAnd get on the Windows kernel team and do bad things.

33:56SPEAKERIt's probably easier for a malicious actor

33:59SPEAKERto sit and be paid to make good contributions

34:02SPEAKERto a project for two or three years

34:05SPEAKERand then start slipping in the bad stuff

34:07SPEAKERonce they've gained the trust

34:09SPEAKERand the appropriate permissions, right?

34:11SPEAKER- Yeah, it definitely seems to me it's the case

34:12SPEAKERthat this was both easier to perpetrate

34:14SPEAKERand also easier to discover and defeat

34:17SPEAKERin the context of open source

34:18SPEAKERthan it would have been in like a corporate setting, right?

34:20SPEAKERLike you said, it's gonna be a lot harder

34:22SPEAKERto get hired at a major corporation probably as a bad actor.

34:25SPEAKERBut then if you pull that off, the code is closed.

34:28SPEAKEROnly people at the company can look at it

34:30SPEAKERand see that there's something wrong

34:32SPEAKERand especially see what it is that's wrong, right?

34:35SPEAKERWhereas here, once people knew that this was a problem,

34:37SPEAKERthe code is out there

34:38SPEAKERand literally everybody who knows what they're doing

34:40SPEAKERcan all look at it together and figure it out quite quickly.

34:43SPEAKER- Right, and so the interesting thing about this

34:45SPEAKERis that the, like you said, the GitHub repository

34:47SPEAKERdoesn't actually have the malicious backdoor code.

34:50SPEAKERIt was only distributed in the source tarballs

34:54SPEAKERthat go up to the respective distribution--

34:59SPEAKER- Distro-maintainers, right. - Yeah, distro-maintainers.

35:01SPEAKERAnd it was also, they did a bunch of clever things

35:06SPEAKERthat made it look like it wasn't actually C code

35:10SPEAKERthat would compile so that it would bypass

35:13SPEAKERthe automated checks for the kind of things

35:15SPEAKERthat it was doing.

35:17SPEAKERAnd it worked shockingly well, right?

35:21SPEAKERLike they did this thing that shouldn't have been possible.

35:25SPEAKERThe other interesting thing is like,

35:26SPEAKERin one of the articles I read,

35:27SPEAKERI think it might've been the big long fact

35:29SPEAKERthat kind of breaks down the whole thing,

35:31SPEAKERthe developer was like,

35:33SPEAKERI should have been able to find this 'cause I run Gentoo

35:35SPEAKERand I was seeing problems reported from users on Debian

35:39SPEAKERand Red Hat that I wasn't able to replicate

35:42SPEAKERon my Gentoo test machine,

35:43SPEAKERpresumably because the Gentoo test machine

35:45SPEAKERwas pulling the source code directly from the Git repo

35:48SPEAKERand not from some upstream tarball,

35:51SPEAKERso they weren't affected by the problem,

35:53SPEAKERwhich is like a super interesting idea

35:56SPEAKERthat maybe the Gentoo people all along were right,

35:58SPEAKERwe should just all be compiling all of our stuff from source.

36:02SPEAKER- Sure.

36:04SPEAKER- So, and it does mean that theoretically,

36:06SPEAKERI mean, at this point, I think the consensus

36:09SPEAKERis that no code submitted by this person

36:11SPEAKERshould probably be regarded as safe

36:13SPEAKERand most of the distros, we'll talk about that in a minute,

36:16SPEAKERbut the distros are taking action to expunge

36:19SPEAKERor validate everything.

36:21SPEAKER- Scorched earth action in some cases.

36:24SPEAKERBut I guess we should talk about or finish talking about

36:27SPEAKERkind of how this happened from a social standpoint.

36:29SPEAKERI mean, people have trolled the XC development mailing lists

36:33SPEAKERfrom the last like two years very extensively at this point

36:35SPEAKERand kind of unearthed like very conspicuous,

36:38SPEAKERlike smoking gun kind of threads, email threads,

36:41SPEAKERmaking it quite clear that Colin was very burned out

36:43SPEAKERand struggling to maintain this important software

36:46SPEAKERby himself with no support or pay.

36:48SPEAKERAnd you see threads and it's very difficult to tell

36:51SPEAKERif these are just legitimate encounters

36:54SPEAKERin the open source world versus

36:56SPEAKERa bunch of sock puppet accounts

36:57SPEAKERthat we're all collectively trying to make this happen.

36:59SPEAKERBut threads where he's being pressured

37:02SPEAKERabout how slowly he's maintaining and updating software,

37:05SPEAKERhere comes this knight in shining armor

37:07SPEAKERwho just happens to be very eager

37:09SPEAKERto contribute to the XC project.

37:11SPEAKERHere are some other accounts chiming in saying,

37:13SPEAKERyou've got a prospective maintainer right here,

37:15SPEAKERlike what's taking you so long?

37:16SPEAKERWhy not get the help you need?

37:17SPEAKERI mean, it's again,

37:19SPEAKERlike could absolutely be real people out there

37:22SPEAKERwho just stumbled into poor behavior on their part

37:25SPEAKERthat aided this attack,

37:26SPEAKERor it could have been an orchestrated attempt

37:29SPEAKERto make this, to push this through.

37:30SPEAKERIt's hard to say.

37:31SPEAKER- There was a blog post by this guy named Rob Minching

37:34SPEAKERwho goes through the whole thing,

37:37SPEAKERthe whole series of threads.

37:39SPEAKERAnd yeah, like when we were doing FOSPOD research,

37:42SPEAKERI remember finding, there were multiple episodes

37:44SPEAKERwhere we found threads that were like,

37:47SPEAKERokay, one of the common threads we talked about is,

37:50SPEAKERthere's a fair number of software packages

37:55SPEAKERthat are really, really crucial to the functioning

37:57SPEAKERof the technological world at this point

38:00SPEAKERthat are the XKZD example.

38:02SPEAKERThey're maintained by one or two people

38:04SPEAKERor really small number of people who are mostly unpaid

38:07SPEAKERor very lightly paid volunteers.

38:09SPEAKERAnd they have trillion dollar companies,

38:14SPEAKERmarket cap companies that are,

38:16SPEAKERthat depend on their software that are like,

38:19SPEAKERhey, like when the Microsoft engineer

38:22SPEAKERdiscovered this exploit,

38:23SPEAKERhe posted a high priority message

38:26SPEAKERto the mailing list for it, right?

38:27SPEAKERLike for all of these things,

38:29SPEAKERwhere most of the people working there

38:31SPEAKERare developer volunteers.

38:32SPEAKERThey're not people who are being paid.

38:34SPEAKERIt's not their job.

38:34SPEAKERThere's no pager duty or whatever.

38:37SPEAKERFor these folks.

38:39SPEAKERAnd it's the same thing with this.

38:42SPEAKERAnd it happens a lot where somebody who comes in

38:45SPEAKERand relies on this piece of open source software

38:47SPEAKERcomes into the maintainer and is like,

38:48SPEAKERhey man, I need you to put this bug fix in.

38:51SPEAKERIt's been in for three weeks.

38:53SPEAKERWhat's taking you so long to get it integrated?

38:55SPEAKERAnd it's usually just like,

38:57SPEAKERI wanted to go to my kid's soccer game

38:59SPEAKERor I was having a mental breakdown

39:02SPEAKERfrom the amount of pressure I'm under

39:04SPEAKERor any number of other things.

39:06SPEAKERIt's not sustainable in a lot of cases for these people.

39:10SPEAKERAnd one of the things that's good is it's another,

39:14SPEAKERone of the good things that will come of this

39:16SPEAKERis another example like Heartbleed was a few years ago

39:20SPEAKERof this really crucial piece of software

39:22SPEAKERthat's being developed by a team

39:24SPEAKERthat's too small and completely unfunded

39:26SPEAKERand not supported in any way, shape or form

39:28SPEAKERby the distros or by the corporate users

39:30SPEAKERor by literally anyone.

39:33SPEAKERSo yeah, but this thread literally has a user come in

39:38SPEAKERand say, hey, is this version of this for Java

39:44SPEAKERstill maintained?

39:44SPEAKERI asked a question here a week ago

39:46SPEAKERand haven't heard back, right?

39:47SPEAKER- Yeah.

39:48SPEAKER- Like, yeah, and then the maintainer responds

39:51SPEAKERand says, yes, it's behind, I'm behind.

39:54SPEAKERIt'll get fixed if somebody supports a bug.

39:56SPEAKERJust there's no new features right now 'cause I'm too busy.

39:58SPEAKER- And then this one conspicuous account

40:00SPEAKERthat a lot of people have looked at is like,

40:02SPEAKERokay, if there's any bad actor in this thread,

40:04SPEAKERit's probably this person.

40:06SPEAKERProgress will not happen until there is a new maintainer.

40:09SPEAKERDennis, you're better off waiting

40:10SPEAKERuntil new maintainer happens or fork it yourself.

40:12SPEAKERSubmitting patches here has no purpose these days.

40:15SPEAKERThe current maintainer lost interest

40:16SPEAKERor doesn't care to maintain anymore.

40:18SPEAKERIt is sad to see for a repo like this

40:20SPEAKERand they pop up several more times

40:22SPEAKERwith some pretty scathing language.

40:25SPEAKERYou ignore the many patches bit rotting away

40:27SPEAKERon the mailing list.

40:28SPEAKERRight now you choke your repo.

40:30SPEAKERWhy wait until 5.4 to change maintainer?

40:32SPEAKERWhy delay what your repo needs?

40:34SPEAKERLike it's, again, could just be a shitty asshole

40:38SPEAKERpressuring somebody doing free work

40:40SPEAKERbecause they want something faster.

40:42SPEAKERCould have been part of this coordinated attack to push.

40:45SPEAKER- Brad, I'm sorry about your mental health issues,

40:48SPEAKERbut it is important to be aware of your own limits.

40:50SPEAKERI get this is a hobby project for all contributors,

40:53SPEAKERbut the community desires more.

40:56SPEAKER- Yeah, yeah, it's hard to read this

40:59SPEAKERand not see some kind of like orchestrated ill intent

41:02SPEAKERin some of these messages.

41:03SPEAKERAt any rate, whatever the nature of that pressure,

41:06SPEAKERit was successful.

41:07SPEAKERAnd this person who, like I said,

41:09SPEAKERhad magically shown up with a lot of interest

41:11SPEAKERin contributing to this project,

41:12SPEAKERultimately was made a co-maintainer of the project,

41:15SPEAKERwhich meant they had Car Blanche

41:16SPEAKERto do literally whatever they wanted with it.

41:18SPEAKER- But at this, like literally at the same time

41:21SPEAKERon this kind of project,

41:22SPEAKERif somebody comes in and starts making code submissions

41:25SPEAKERand helping, like the maintainer's gonna say yes.

41:28SPEAKER- The maintainer did nothing wrong here

41:29SPEAKERas far as I'm concerned, right?

41:31SPEAKERLike somebody wants to help on an open source project,

41:33SPEAKERthe whole point of the open source is that,

41:35SPEAKERyeah, you accept help.

41:37SPEAKERAnd the fact that like this person

41:40SPEAKERwas able to get in and do this is,

41:44SPEAKERit's kind of a bug, also kind of a feature.

41:47SPEAKER- Yeah.

41:47SPEAKER- So I don't know,

41:51SPEAKERlike, I mean, they actually,

41:52SPEAKERlet's talk about what's next.

41:54SPEAKERLike that's where we're at, right?

41:55SPEAKERWhat happens next?

41:56SPEAKER- Yeah.

41:57SPEAKER- GitHub almost immediately suspended

41:59SPEAKERboth main contributors to the XCTLs,

42:01SPEAKERColin, LaCie, and Jiten.

42:04SPEAKER- Colin has been, was reinstated pretty quickly, obviously,

42:07SPEAKER'cause it was pretty clear

42:08SPEAKERthat he didn't do anything wrong here.

42:10SPEAKERHe was just made a victim, essentially.

42:12SPEAKER- If you're running the unstable rolling update versions

42:15SPEAKERof Linux, you should absolutely update your systems.

42:17SPEAKERLike that's important.

42:19SPEAKER- Yeah, like we said earlier,

42:20SPEAKERthis malicious code got out there to everywhere.

42:23SPEAKERLike I said, it was on my MacBook, it's in Arch,

42:25SPEAKERit's in a bunch of places that have bleeding edge software.

42:28SPEAKERBut again, it was only those specific contexts,

42:31SPEAKERDebian and Fedora-based distros

42:33SPEAKERwith specific, all these different pieces in place

42:35SPEAKERwhere it could actually be used.

42:38SPEAKERBut, and the caveat constantly to that is,

42:43SPEAKERas far as we know, like for now, this is what we know now,

42:46SPEAKERbut this person was so smart and so malicious

42:50SPEAKERthat the running assumption is like,

42:51SPEAKERthere could be anything else out there

42:52SPEAKERthat we don't know about.

42:53SPEAKERSo it doesn't matter if you're on

42:55SPEAKERwhat we think are unaffected systems right now,

42:56SPEAKERyou absolutely should update immediately.

42:59SPEAKER- Well, and the more paranoid security researchers

43:03SPEAKERthat I follow are now looking at this as a potential other,

43:07SPEAKERlike this method as a vector for other attacks

43:10SPEAKERon other parts of the software ecosystem,

43:15SPEAKERin addition to what we've seen here with XeUtils.

43:19SPEAKER- Right, yeah, this is what is,

43:22SPEAKERI don't think I'd heard this term before,

43:23SPEAKERlike this is what is being referred to as a supply chain

43:25SPEAKERattack because it attacks something, again,

43:28SPEAKERnot the thing that ultimately is to be exploited,

43:30SPEAKERwhich in this case was SSH-D,

43:32SPEAKERbut something else that it depends on in specific cases.

43:36SPEAKERSo like that's kind of where the supply chain part comes in.

43:39SPEAKERYou can imagine this chain of other software

43:41SPEAKERthat you need in place to run things in a normal context

43:45SPEAKERalso being exploited here.

43:46SPEAKER- But then the other thing to think about is

43:49SPEAKERif you're a nation state and you want access to something,

43:53SPEAKERit's not like your person who,

43:55SPEAKERyou know, your people who maintain these high value targets

44:01SPEAKERare like, you know, they're not signing security clearances

44:05SPEAKERand they're not getting annual,

44:07SPEAKERthey're not getting checkups from the security agent.

44:10SPEAKERYou know, they're not working at CISA

44:12SPEAKERor the FBI or the NSA or whatever and secured.

44:16SPEAKERSo if I'm a maintainer of this high profile thing

44:19SPEAKERand I have financial problems and somebody comes to me

44:22SPEAKERis like, hey, we'll take care of your credit card problem

44:25SPEAKERif you just put a little code in here for us.

44:29SPEAKERLike it's a kind of nasty situation as interdependent

44:32SPEAKERand interlocked as the open source world is.

44:35SPEAKER- Yeah, I mean, you find a lot of open source developers

44:38SPEAKERare very principled and you would hope that the principles

44:40SPEAKERthat bring people to this kind of work in the first place

44:42SPEAKERwould be counter to some kind of self-serving action

44:46SPEAKERlike that, but you never know, like it's impossible to say.

44:48SPEAKER- Well, I was gonna say one of the things that is happening

44:50SPEAKERas a result of this is that other software in the tool chain

44:53SPEAKERis being investigated to see how to prevent

44:56SPEAKERthis type of attack in the future.

44:57SPEAKERLike OpenSSH is considering implementing library-less

45:00SPEAKERversions of its functionality so that it doesn't rely

45:04SPEAKERon external software as much, right?

45:06SPEAKER- Yeah, like I've seen a lot of those kind of policy changes

45:09SPEAKERand so forth that people are discussing, which is good,

45:12SPEAKERbut also seem like very reactive sorts of responses to me

45:15SPEAKERwhere it's like, hey, this particular vulnerability is here

45:18SPEAKERand so we're gonna try to close that avenue off,

45:21SPEAKERbut there are assuredly there are countless other avenues

45:24SPEAKERfor something like this that people aren't thinking

45:25SPEAKERabout right now.

45:26SPEAKERIt feels like the better approach to solving this

45:28SPEAKERis better support for the people who make

45:31SPEAKERthis really important software on the part of,

45:33SPEAKERyou tell me, on the part of corporations,

45:35SPEAKERon the part of governments, I don't know,

45:37SPEAKERon the part of the community, like it just,

45:40SPEAKERit seems like an unsustainable state of affairs

45:41SPEAKERto have such critical infrastructure be maintained

45:44SPEAKERby people who are putting a huge amount of work in

45:47SPEAKERand getting nothing out of it.

45:49SPEAKERThat's just not sustainable forever

45:51SPEAKERand it seems like this is just going to happen again

45:53SPEAKERif there isn't some kind of better support put in place.

45:56SPEAKER- I think that's a larger conversation, right?

46:00SPEAKERRight now we're in the, hey, one horse got out of the barn,

46:02SPEAKERbut there's still 24 horses left in the barn.

46:04SPEAKERShould we shut the door to the barn

46:06SPEAKERor should we build a better, put a fence around the barn?

46:11SPEAKERI don't know, metaphor broke down there, but.

46:15SPEAKEROne of the conversations I saw that was really interesting

46:18SPEAKERout of this is that those distro-specific tarballs

46:21SPEAKERhave been, are essentially un-peer reviewable

46:25SPEAKERbecause there's just, it's a massive amount of code

46:27SPEAKERto go over and it's more or less the same

46:30SPEAKERand people are starting to look at ways

46:33SPEAKERthat that can be evaluated systematically or,

46:40SPEAKERI hesitate to say this, it might be a machine learning,

46:43SPEAKERit might be a machine, this might be a machine learning

46:46SPEAKERproblem, there's other, there's a whole bunch

46:49SPEAKERof really interesting conversations happening

46:51SPEAKERabout that stuff.

46:52SPEAKER- Yeah, for sure, I would love to hear from like working

46:54SPEAKERopen source developers, it was a little bit over my head

46:57SPEAKERwhy exactly code is packaged up separately

47:00SPEAKERfor these different Linux distros and distributed directly

47:02SPEAKERthat way as opposed to pulling directly from kind

47:05SPEAKERof the upstream, I mean, distros patch stuff all the time,

47:08SPEAKERlike we've talked about here, it's specifically

47:11SPEAKERbecause of the way Debian and Fedora do things

47:13SPEAKERwith this chain of software that this was even possible.

47:16SPEAKER- It's because they have unique make files often

47:19SPEAKERand they have unique, like you said, the patches

47:22SPEAKERthat come in on a distro-specific basis

47:25SPEAKERfor individual libraries have to be integrated

47:27SPEAKERand they, so they set them up so that's easy to do,

47:30SPEAKERbasically, is my understanding.

47:32SPEAKER- Yeah, if there's any more nuance to that,

47:33SPEAKERI would love to hear like on our Discord

47:35SPEAKERif people wanna chime in.

47:36SPEAKER- Yeah, that'd be great.

47:37SPEAKER- There.

47:39SPEAKER- On the funding side though, there was a good CSA post

47:44SPEAKERof the Cybersecurity and Infrastructure Security Agency,

47:46SPEAKERwhich is a US government agency tasked

47:49SPEAKERwith finding this out.

47:52SPEAKERThere's obviously a lot of law enforcement work going

47:54SPEAKERon this right now, but they had a blog post

47:56SPEAKERthat explicitly says, it was bolded, this text.

47:59SPEAKERI'm gonna read it exactly.

48:00SPEAKERIt says, tech manufacturers who profit

48:03SPEAKERfrom open source software must do their part

48:05SPEAKERby being responsible consumers of

48:07SPEAKERand sustainable contributors

48:09SPEAKERto the open source packages they defend on.

48:11SPEAKERAnd then they said, that explicitly means

48:13SPEAKEReither putting money in,

48:16SPEAKERdedicating developers to things or both.

48:19SPEAKER- Interesting, this, okay, this, yeah,

48:20SPEAKERthis blog post just went up the day before recording this.

48:22SPEAKERI hadn't seen this yet.

48:23SPEAKERThat's, I mean, that's good.

48:25SPEAKERThings like that need to be said more often

48:26SPEAKERand more loudly for sure.

48:28SPEAKER- Yeah, they, in that blog post, it's worth reading,

48:30SPEAKERthey had talked about having a tabletop exercise

48:33SPEAKERthat basically mimicked that the week,

48:36SPEAKERliterally the week before this happened,

48:38SPEAKERthat, you know, where they basically get together

48:42SPEAKERand war game out this whole scenario of,

48:44SPEAKERhey, a backdoor has been discovered,

48:45SPEAKERhow do we deal with it with a bunch of big contributors

48:49SPEAKERand open source distro maintainers

48:51SPEAKERand software developers and all this.

48:53SPEAKERThat stuff's public, so you can go download it if you want,

48:55SPEAKER'cause it was built by the US government.

48:59SPEAKERThe FFmpeg blog that you linked that called out Microsoft

49:02SPEAKERfor posting high priority security alerts

49:05SPEAKERto a unpaid volunteer developer mailing list, feels icky.

49:10SPEAKER- Yeah.

49:11SPEAKER- They make a lot of money.

49:12SPEAKER- For sure, I mean, yeah, this started on Twitter.

49:15SPEAKERI didn't actually see the blog version of this,

49:17SPEAKERbut the FFmpeg Twitter account says,

49:21SPEAKER"The XZ fiasco has shown how a dependence

49:23SPEAKER"on unpaid volunteers can cause major problems.

49:25SPEAKER"Trillion dollar corporations expect free

49:27SPEAKER"and urgent support from volunteers."

49:29SPEAKERLike this message they posted from their mail,

49:31SPEAKERfrom this bug that was filed from a Microsoft engineer.

49:35SPEAKERHi, this is a high priority ticket

49:37SPEAKERand the FFmpeg version is,

49:39SPEAKERand this FFmpeg version is currently used

49:41SPEAKERin a highly visible product at Microsoft.

49:44SPEAKERWe have customers experiencing issues with captions

49:46SPEAKERduring Teams live events, please help.

49:50SPEAKER- Yeah.

49:50SPEAKER- Think about that, this is a $3 trillion company

49:53SPEAKERgoing out there and saying,

49:54SPEAKERhey, volunteers who make this free product

49:57SPEAKERthat we use in a ton of our software,

49:59SPEAKERthis is high priority, please fix this immediately.

50:02SPEAKERLike what?

50:03SPEAKERAnd then the FFmpeg account goes on,

50:05SPEAKER"After politely requesting a support contract

50:07SPEAKER"from Microsoft for a long-term maintenance,

50:10SPEAKER"they offered a one-time payment of a few thousand dollars

50:13SPEAKER"instead, this is unacceptable."

50:15SPEAKER- Oh.

50:17SPEAKER- The lesson from the XZ fiasco is that investments

50:19SPEAKERin maintenance and sustainability are unsexy

50:21SPEAKERand probably won't get a middle manager their promotion,

50:24SPEAKERbut pay off a thousand fold over many years,

50:27SPEAKERbut try selling that to a bean counter.

50:30SPEAKERWhoever runs this account has strong opinions

50:32SPEAKERon this matter.

50:33SPEAKER- I mean, look, if I made FFmpeg,

50:35SPEAKERI'd probably have some real strong feelings about it too,

50:37SPEAKERgiven all the places it shows up.

50:39SPEAKER- It's insane, I mean, it's like maybe not

50:40SPEAKERthe most common software, but it's up there,

50:42SPEAKERespecially in the user-facing context.

50:44SPEAKER- It's like, yeah, in terms of stuff people see,

50:47SPEAKERevery TV in the world.

50:49SPEAKER- FFmpeg is probably used in damn near every device

50:52SPEAKERthat handles audio and video in some fashion

50:54SPEAKERor their library versions like libavcodec.

50:57SPEAKER- Yeah.

50:57SPEAKER- But it's weird because in some cases this does happen.

51:01SPEAKERLike for example, last I checked Intel

51:03SPEAKERis the biggest contributor to the Linux kernel.

51:05SPEAKERNetflix is a massive contributor to FreeBSD.

51:07SPEAKER- Twitch and I think Google contribute to OBS.

51:11SPEAKER- Right, like yes, like AV1 came out of a consortium

51:15SPEAKERof corporations, like there are a lot of cases

51:18SPEAKERwhere big companies do pump tons of money

51:20SPEAKERinto and development efforts into open source stuff,

51:23SPEAKERbut it's not global and kind of like that,

51:25SPEAKERthe FFmpeg account said, like in a lot of cases like this,

51:28SPEAKERit's not sexy to devote money

51:30SPEAKERto maintaining a compression library, you know?

51:33SPEAKERIt seems like they target things

51:35SPEAKERwhere they are going to very tangibly benefit

51:37SPEAKERin the short term.

51:38SPEAKER- But I mean, so it's good that like YouTube

51:40SPEAKERand Logitech and Twitch and Facebook and Nvidia

51:43SPEAKERand AMD and Intel sponsor OBS, right?

51:47SPEAKERBut OBS also has a boatload of downstream contributors

51:50SPEAKERin other open source projects like FFmpeg and all that.

51:53SPEAKERAnd like, if you don't support the whole chain,

51:59SPEAKERit's not gonna like, yeah, it just doesn't work.

52:03SPEAKER- Yeah, like this example should make it abundantly clear

52:05SPEAKERthat you have to, everything has to be covered

52:08SPEAKERor nothing is ultimately.

52:10SPEAKER- Yeah, so let's see there.

52:13SPEAKERI'm sure law enforcement people are doing this.

52:15SPEAKERSomebody did a time zone analysis of all the commits

52:17SPEAKERthat the bad actor did.

52:20SPEAKER- Yeah, it's been funny watching people

52:22SPEAKERtry to sleuth that out or not.

52:23SPEAKERYou know, some people are like it's pointless

52:25SPEAKERto waste our time like trying to dissect

52:28SPEAKERwhat time commits were made

52:29SPEAKERor what this person's username is.

52:31SPEAKERLike other people will figure that out.

52:33SPEAKERBut then like you said, a lot of other people

52:35SPEAKERhave been saying like, well, the majority of the commits

52:38SPEAKERwere made from like nine to five

52:41SPEAKERin this particular time zone in Eastern Europe

52:43SPEAKERor in China or the commits seem to miss

52:47SPEAKERthese specific holidays in this specific country.

52:49SPEAKERYou know, a lot of like what nation state

52:51SPEAKERwas this person working for?

52:52SPEAKER- Hardcore internet detective work is what I would call it

52:55SPEAKERwith all the good and bad that that implies.

52:58SPEAKER- Right, sometimes I yield results.

53:00SPEAKERA lot of times it does a lot of damage.

53:02SPEAKERBut the one thing that I feel like

53:04SPEAKERjust about everybody agrees on here

53:06SPEAKERis that it was almost certainly somebody affiliated

53:08SPEAKERwith a nation state.

53:09SPEAKERBecause of the level of sophistication

53:11SPEAKERand time and dedication involved here,

53:14SPEAKERthat it seems almost inconceivable

53:16SPEAKERthat this could have been anything else

53:17SPEAKERother than somebody who was supported

53:19SPEAKERby a large amount of resources.

53:22SPEAKER- I don't know enough about this to have an expert opinion.

53:27SPEAKERI think that it's pretty easy to compromise people.

53:31SPEAKERWhether they're working on the behalf of a nation state

53:34SPEAKERfrom the beginning or they started working

53:35SPEAKERfor a nation state after they had some sort of problem

53:38SPEAKERthat got them in trouble that made them easy to compromise.

53:42SPEAKERLike there's a reason that if you have too high

53:44SPEAKERa debt to income ratio or if you have

53:47SPEAKERa longstanding drug use problem,

53:48SPEAKERyou can't get a security clearance, right?

53:50SPEAKERAnd it's like we're in the same space here.

53:55SPEAKERAnyway, but yeah.

53:57SPEAKERTwo years to build this relationship seems like a lot

54:01SPEAKERfor somebody who's not getting paid by someone.

54:03SPEAKER- For sure.

54:04SPEAKER- Especially given that they did mostly get commits happen

54:08SPEAKERwithin a working day type schedule.

54:12SPEAKERIt is worth mentioning that the commit times

54:15SPEAKERcan be changed on GitHub.

54:16SPEAKERThere's a whole host of reasons

54:18SPEAKERthat that is weird and difficult to do.

54:22SPEAKERMostly around the conversations around the commits

54:24SPEAKERcoming onto message boards and stuff like that

54:26SPEAKERand those times not matching up.

54:27SPEAKER- Right.

54:28SPEAKER- But yeah.

54:29SPEAKER- It was kind of weird.

54:30SPEAKERI'd see a good number of people out there going like,

54:32SPEAKER"Oh, look, this malicious actor has a Chinese name.

54:36SPEAKERLike look at that."

54:37SPEAKERAnd I'm like, "That seems like more evidence to me

54:39SPEAKERthat this person was not affiliated with China."

54:41SPEAKERLike that's the easiest thing to fake.

54:44SPEAKER- If you're a Russian hacker,

54:47SPEAKERmalicious hacker doing bad things to people,

54:50SPEAKERyou don't make your username Russia number one.

54:52SPEAKER- Right.

54:53SPEAKER- Yeah.

54:54SPEAKER- Yes.

54:55SPEAKER- It's unfortunate this keeps happening.

55:00SPEAKERWhether it's vulnerability or a back door,

55:03SPEAKERwe've seen this happen again and again.

55:05SPEAKERAnd we've said it four times, I think,

55:07SPEAKERalready in this episode, but it's true.

55:10SPEAKERIt's either Atlas holding up the world

55:12SPEAKERor the XKCD Image Magic comic

55:14SPEAKERwhere it's the one dependency

55:15SPEAKERon which the entire internet rests.

55:18SPEAKERAnd it feels like this is the open source

55:21SPEAKERdeveloper's version of going viral,

55:22SPEAKERwhich seems horrible for everyone involved.

55:25SPEAKER- Yes.

55:26SPEAKER- So I don't, I hope that the person

55:30SPEAKERwho maintains XE Utilities, Colin Laskas,

55:33SPEAKERrecovers from this and it doesn't make their life worse.

55:36SPEAKER- They seem to be taking it somewhat hard,

55:38SPEAKERwhich I mean, how could you not when this first broke,

55:41SPEAKERwhich really makes you feel for them.

55:43SPEAKERBut all we can hope for is that this leads to change

55:47SPEAKERand institutional infrastructural change

55:50SPEAKERin how a lot of this stuff is supported

55:52SPEAKERand propped up and kept going.

55:54SPEAKER'Cause like everybody depends on this.

55:56SPEAKERLike literally the entire world depends on the sanctity

55:59SPEAKERof this stuff at this point.

56:00SPEAKERAnd surely, surely this time things will be different.

56:05SPEAKER- Well, let me tell you a little story

56:07SPEAKERabout Lucy and Charlie Brown and the football.

56:10SPEAKER- Oh boy.

56:12SPEAKER- I don't know, I mean, in the time

56:14SPEAKERthat I have been following this,

56:16SPEAKERI have never seen this community react

56:18SPEAKERto something like this, not even close.

56:20SPEAKER- This feels like, it feels a lot like Heartbleed, honestly.

56:22SPEAKER- This feels like a watershed moment

56:24SPEAKERwhere people are like, oh my God,

56:25SPEAKERwe have got to fix some things

56:27SPEAKERor we're all going to be in a lot of trouble.

56:30SPEAKER- I think each time it happens

56:32SPEAKERand the people who are smart and tell them,

56:35SPEAKERtell the people that have decisions,

56:36SPEAKERmake decisions and have money that this is going to happen

56:39SPEAKERand it keeps happening again and again,

56:41SPEAKERit means at some point.

56:43SPEAKERLike I personally, I am fine with my tax dollars

56:47SPEAKERgoing to fund technological infrastructure like this, right?

56:53SPEAKERLike this is a thing, if the people who maintain

56:57SPEAKERthis kind of software and stuff like Curl or Wget

56:59SPEAKERor FFmpeg or the things that are the foundational pieces

57:04SPEAKERof the technological stack that make a massive portion

57:08SPEAKERof the economy work, if they can't make money

57:12SPEAKERmaking those things and those things aren't necessary,

57:15SPEAKERwe should be funding them.

57:16SPEAKERThere's no reason not to.

57:17SPEAKERIt doesn't cost very much compared to the,

57:20SPEAKERlike compared to the financial impact

57:21SPEAKERof something like this going wide

57:23SPEAKERand being installed on every computer in the United States.

57:26SPEAKERYeah, no, there's no reason not to give this person

57:28SPEAKER50 grand a year to keep doing the thing that they're doing

57:31SPEAKERso they don't have to sweat it all the time.

57:33SPEAKER- Or even a healthy, like a healthy living

57:35SPEAKERfor like two, three, five people.

57:38SPEAKER- Yeah.

57:39SPEAKER- Is absolutely nothing in large budgetary contexts, right?

57:43SPEAKERIt's absolutely nothing for the payoff.

57:46SPEAKERI mean, that's what FFmpeg said, right?

57:47SPEAKERLike it's the payoff is beyond reckoning for the investment

57:52SPEAKERbut somebody has got to make the investment first.

57:54SPEAKERAnyway, that's probably about it.

57:57SPEAKERI do want to mention just the response to this.

58:00SPEAKEREverybody's purging it in their own ways.

58:02SPEAKERLike you said, like the underlying libraries

58:05SPEAKERand software are putting up extra guard rails

58:08SPEAKERto prevent things like this happening again.

58:09SPEAKERAnd the distros are all taking care of it.

58:10SPEAKERI did think it was funny.

58:11SPEAKERI followed the, I run Debian on my server at home.

58:15SPEAKERSo I followed the Debian bug mailing list

58:17SPEAKERas this was all happening.

58:19SPEAKERThey initially rolled back to the version

58:22SPEAKERbefore like this final piece

58:23SPEAKERof all the malicious code was put in,

58:25SPEAKERbut then very quickly said like,

58:26SPEAKERhow can we trust any commit this person ever made?

58:29SPEAKERYou know, and pretty quickly decided like,

58:32SPEAKERhey, let's just roll back to before this person ever had

58:35SPEAKERany kind of commit access whatsoever.

58:36SPEAKERLike what is the value of keeping any of their contributions

58:40SPEAKERagainst the potential harm that there is more lurking in here

58:43SPEAKERwe don't know about.

58:44SPEAKER- Yeah.

58:45SPEAKER- So that's what's happening there.

58:46SPEAKERBut I did read Lassa Collins page.

58:50SPEAKERLike, I don't know how recently this got updated.

58:51SPEAKERThey are pretty quickly working toward a new clean version,

58:55SPEAKERwhat they consider a fully clean version,

58:58SPEAKERwhich they say is going to be 5.8.0.

59:00SPEAKER- Yeah.

59:01SPEAKER- And once that's out, they say this,

59:03SPEAKERat least in this particular case,

59:04SPEAKERthis danger hopefully will have passed

59:06SPEAKERbecause they will have audited everything at that point.

59:09SPEAKER- This is, we talked about this with Jeremy Allison

59:11SPEAKERon the FOS pod last year or year before last, I guess now.

59:14SPEAKERJeremy was the maintainer for Samba for a long time.

59:19SPEAKERAnd he described it as the plumbing of the computer world.

59:23SPEAKERLike, you're making the sewer and the water lines

59:27SPEAKERand it's stuff that everybody relies on

59:29SPEAKERand nobody really thinks about except for when it breaks.

59:32SPEAKERAnd that's where we're at.

59:33SPEAKER- It's broken and somebody's got to put it back together.

59:36SPEAKERSo thanks to the people who do that work.

59:39SPEAKERAnd on that note, I think that'll do it for us this week.

59:44SPEAKERBrad, as always, you know this, the listeners may not,

59:47SPEAKERBrad and Will Made a Tech Pod is a listener supported show.

59:49SPEAKERThat means we would not be here

59:50SPEAKERwithout your generous support.

59:53SPEAKERFor five bucks, you get access

59:54SPEAKERto the fabulous Tech Pod Discord,

59:56SPEAKERwhich is full of beautiful nerds who talk about stuff like,

1:00:00SPEAKERit turns out it was a useful resource,

1:00:02SPEAKERlearning about this back door.

1:00:06SPEAKERTalk to several members to ask some questions

1:00:09SPEAKERthat clarified things and all that

1:00:11SPEAKERas we were putting the notes together for this.

1:00:13SPEAKERAny errors in the show are ours and not theirs, of course.

1:00:17SPEAKERBut you can go to patreon.com/techpod

1:00:20SPEAKERand like I said, for $5 a month,

1:00:21SPEAKERyou get access to the Discord,

1:00:23SPEAKERyou get our patron exclusive episodes,

1:00:25SPEAKERand you get to be a part of the Tech Pod community,

1:00:27SPEAKERwhich is growing and awesome every day.

1:00:30SPEAKER- Great resource.

1:00:31SPEAKERI have to say the Linux channel has never been

1:00:33SPEAKERas active as it was in the days after this broke.

1:00:36SPEAKERIt was fun to watch people who work in this world

1:00:40SPEAKERkind of stunned by everything that was happening.

1:00:43SPEAKER- And as always, we have a,

1:00:45SPEAKERwe want to thank everybody who supports the show,

1:00:47SPEAKERbut we want to give a very special thanks

1:00:48SPEAKERto our executive producer tier patrons,

1:00:50SPEAKERincluding Paddle Creek Games, Makers of Fractured Veil,

1:00:53SPEAKERAndrew Slosky, Jordan Lippett, Bunny Crimes, Just Wedge,

1:00:57SPEAKERJoel Krauska, Twinkle Twinkie, David Allen, James Kamik,

1:01:00SPEAKERand Pantheon, Makers of the HS3 High Speed 3D Printer.

1:01:05SPEAKERThank you all so much for listening.

1:01:06SPEAKERWe will be back next week

1:01:08SPEAKERwith another episode of the Tech Pod.

1:01:09SPEAKEROh, and just remember, I think not next week,

1:01:13SPEAKERbut maybe the week after,

1:01:14SPEAKERis that what we're thinking now?

1:01:15SPEAKERPirates of Silicon Valley?

1:01:17SPEAKER- Cool.

1:01:18SPEAKERLet's say it could be any time.

1:01:19SPEAKER- Could be any time.

1:01:21SPEAKERIt's on archive.org.

1:01:23SPEAKEREverybody can watch it around the world.

1:01:25SPEAKER- Yes, I'll link it in the show notes yet again this week.

1:01:28SPEAKERIt could be a bright glass

1:01:29SPEAKERand we're gonna do it next week,

1:01:31SPEAKERdepending on, I'm traveling again this week,

1:01:33SPEAKERso it could be a, we need to-

1:01:35SPEAKER- See how that goes, okay.

1:01:36SPEAKER- But it could be the week after.

1:01:39SPEAKERYes, make sure to watch that.

1:01:41SPEAKER- So watch Pirates of Silicon Valley.

1:01:43SPEAKERLook, it has the guy from ER playing Steve Jobs.

1:01:45SPEAKERHe can't beat that.

1:01:46SPEAKER- Yeah, no.

1:01:47SPEAKER- We'll be back next week with another Tech Pod.

1:01:49SPEAKERThanks for listening.

1:01:49SPEAKERSee you all then.

1:01:50SPEAKERBye.

1:01:51SPEAKER(upbeat music)