00:00SPEAKERI love this mug you sent me.
00:01SPEAKERI'm glad you like it. It was the biggest handle I had.
00:05SPEAKERLook, you told me - so you gave me a mug in January when I was in Chicago and saw you and you said,
00:10SPEAKER"Hey, put this in your hand and feel it." And it was literally the first time anybody had ever
00:14SPEAKERsaid that I should think about how a mug fits in my hand and I didn't - like, I'd never considered
00:20SPEAKERit before. I'd always just bought the mug that had the picture on it that I wanted and I was
00:27SPEAKERabsolutely heartbroken when I dropped a pole on it a few months ago and it broke. So I was stoked to
00:32SPEAKERget a fresh big handle bad boy here with a lovely blue glaze. So...
00:38SPEAKERA lot of potters hate handles. They hate making them because they can be tricky. And I always,
00:44SPEAKERwhen I first started, focused on form. I focused on, like, what looks pretty or has a nice negative
00:51SPEAKERspace or whatever the other - and never really thought about, like, how it felt in my hand.
00:56SPEAKERAnd then I was studying under this potter a year and a half ago, two years ago, and she's like,
01:01SPEAKER"All right, we're going to do handles. We're going to make handles for an hour straight and just
01:05SPEAKERattach them to a bucket. Like, just do anything to keep making handles." And I think I made 43
01:10SPEAKERhandles in an hour. Like, I just motored through them. And you could clearly look at the first five
01:16SPEAKERwhich was garbage, but the last five were just like, yeah, it's starting to get there. It's kind
01:20SPEAKERof neat. And then I started to think about, like, really what makes it feel like it's hugging your
01:24SPEAKERfingers. The mug hugs your fingers.
01:54SPEAKER- Welcome to Brad and Will Made a Tech Pod. I'm Will. Brad, unfortunately, well, he's here.
02:02SPEAKERHe's listening. But he had some surgery this week and his head is packed full of gauze. So
02:08SPEAKERrather than talk to us and make us hear the horrible non-dulcet tones of Brad's gauze-packed
02:15SPEAKERnose, I'm here with my pal, Brian Fitzpatrick, of TOC, but formerly of Google and a bunch of
02:23SPEAKERother stuff that we'll get to. And we're gonna talk about some old technology. Fitz,
02:27SPEAKERwelcome to the show. Hello. - Thanks. Thanks for having me. Good to see you. Good to catch
02:31SPEAKERup a little. - If you're a long-time listener to Still Entitled, you've heard me talk about an
02:38SPEAKERevent that happens in Chicago in January called OrdCamp that Fitz, along with a team of really,
02:44SPEAKERreally brilliant people, has been running for, what, like 12 years now?
02:47SPEAKER- 12 years. We've done 12 of them, yeah. - Yeah. And it is the event that he runs
02:56SPEAKERis a collection of brilliant, interesting, fascinating, fun people that come together and
03:03SPEAKERkind of share the things that they're interested and passionate about. And it's a really incredible
03:10SPEAKERcross-disciplinary event. And it's one of my favorite things that I get to do when I get to
03:15SPEAKERgo. So thank you for that. But you've also been doing technology stuff for... You're one of the
03:22SPEAKERfabled Greybeards, Fitz. I don't want to out you. - Yeah. No, no worries. I've been doing this for
03:27SPEAKER26 years now. Yeah. - You started with one of the early code management systems, right?
03:33SPEAKER- Well, I started before that. I mean, that was sort of the thing that helped me, I think,
03:39SPEAKERreally get out there. I was doing open source stuff, working on... I did work on Subversion
03:43SPEAKERwhen it first started in 2000. And Subversion launched in 2003. I have not been using Subversion
03:49SPEAKERfor longer than it's actually existed, which is kind of funny. For longer than I actually used it.
03:56SPEAKERBut yeah, and I started doing CGI programming with Perl back in '94 when it was like Mosaic
04:03SPEAKERand Netscape had just come out. And I don't know, I just loved it. I think at a late start,
04:08SPEAKERI didn't start till I was 24. - Oh, so did you... Were you like a CS guy? - No, I majored in Latin,
04:16SPEAKERminored in Greek and pottery. - Wow. - So my mom thought I was gonna be homeless. - I mean,
04:23SPEAKERlook, it worked out okay for you, I guess. - I always have a hobby. - That's funny. I did Latin
04:29SPEAKERand ancient Greek and those contributed to me being able to write really well.
04:35SPEAKER- Yeah. - Because you can't... You're not gonna do well at Latin if you don't... Doing well at Latin
04:42SPEAKERmakes you really, really, really understand English grammar in a way that is fascinating,
04:48SPEAKERI always found. But yeah, so okay. So you worked on Subversion, which was... I mean, I guess we
04:54SPEAKERcould explain what Subversion is, but it's basically a predecessor to Git or one of the
04:59SPEAKERmore modern code management systems at this point, right? - I'd agree. Yeah, it was a centralized
05:03SPEAKERversion control system built to replace CVS, which was the original open source version control
05:10SPEAKERsystem that everyone used back in the '90s and 2000s up until Subversion basically supplanted
05:16SPEAKERthat and then Git supplanted that. - And then that took you to Google, right? - Yeah, I started at
05:24SPEAKERGoogle, I don't know, a year and a year and a half after Subversion launched. - And you were part of
05:31SPEAKERa crowd that seemed like they didn't wanna move to California but wanted to work for Google and
05:37SPEAKERyou ended up somehow staying in Chicago? - Yes, somehow. I don't know how that worked
05:41SPEAKERout. That was kind of fascinating. I basically said, "Yeah, I'm staying here and I'd love to
05:45SPEAKERwork for Google." And wound up with me starting Google's engineering office in Chicago along with
05:50SPEAKERmy friend Ben, who you know very well. - Yeah, I love Ben. Ben's great. And Google... So when we
05:57SPEAKERget to Google, we're gonna get to the interesting part, I think the really interesting part of the
06:00SPEAKERstory because Google... In addition to day job stuff that you did, you started the takeout,
06:09SPEAKERwhat is now I think called the Google takeout program which lets you extract all of your data
06:15SPEAKERfrom the giant faceless multinational tech company so that you can move it to other places or
06:21SPEAKERwhatever. But I mean, that initiative wouldn't... I was at Maximum PC when you all started doing
06:27SPEAKERthat and I saw that and I was like, "Oh, this is something I had never thought these enormous tech
06:30SPEAKERcompanies were gonna allow." And it kind of set a really positive tone for data portability in a way
06:38SPEAKERthat Google was giving us all these free services. There was absolutely no... There was no need for
06:45SPEAKERGoogle to release the data that we'd given to Google. So it was a refreshing thing and I'm
06:51SPEAKERcurious, that process, how much of that was... How much of a fight was it to get that to happen
06:59SPEAKERinside the company at the time? Were people on board or was it a challenge?
07:03SPEAKERIt was actually effortless and so far as I went and talked to... I basically went to Eric Schmidt
07:10SPEAKERat the time and said, "Hey, I really wanna do this thing." And he's like, "Well, why are you asking me
07:14SPEAKERpermission? Just go, you should be doing this. Of course you should be doing this." Because he had
07:18SPEAKERbeen... One of the things that... There are two things that inspired me to start it. One was Eric
07:24SPEAKERkept giving talks internally and externally where he talked about the fact that Google doesn't lock
07:28SPEAKERin your data. Like other companies where it's impossible to get your data out or export it,
07:34SPEAKERwe make it easy to go. And I thought like, "Yeah, we kinda do. The door's open, but in some cases,
07:40SPEAKERyou really gotta bang your shoulder up against it and I think it could be better." And then I
07:44SPEAKERwound up having a conversation, there's a long story behind this, but due to an article about
07:48SPEAKERme in ValleyWag that misquoted me, I met this Google PM named Jess, who was the one that
07:55SPEAKERinitially threw the idea about the Data Liberation Project. She called it Data Portability. And we
08:00SPEAKERwere just getting ready to roll it out and then dataportability.org came out. And I looked at this
08:05SPEAKERand I'm like, "This is never gonna go anywhere." And this was like 2008, 2009. And so we decided
08:11SPEAKERto go Data Liberation, sort of like Guerrilla Army type marketing. And I'm glad we did 'cause
08:17SPEAKERdataportability.org is still there and it sort of hasn't done a whole lot, frankly.
08:21SPEAKER- Well, but I mean, if you've never used the takeout service, it's really neat. You go in,
08:26SPEAKERyou log into your Google account and then you say, "Okay, I wanna download all this stuff." And it
08:29SPEAKERsays, "Okay, we're gonna zip this up and we'll let you know when the download's ready." And then
08:35SPEAKERyou can get everything from email, stuff that's uploaded to photos, stuff that's uploaded to
08:40SPEAKERYouTube. - Your blog, anything.
08:43SPEAKER- Yeah. All of it. And it works.
08:47SPEAKER- The really cool thing about it is we initially launched with zip because that was the easiest
08:51SPEAKERthing to get done and build the pipeline. But now you can point your stuff straight at Dropbox. You
08:57SPEAKERcan push your stuff right at Microsoft OneDrive. And Google started the data transfer project in
09:03SPEAKERthe last, I don't know, year and a half, two years ago, which is an open source project to allow
09:07SPEAKERanyone basically to build an endpoint for takeout. Not just for takeout, but for this sort of
09:13SPEAKERprotocol of transferring data from one place to the other so that it doesn't turn into a technology
09:18SPEAKERproblem to integrate with other services. It turns into basically a sort of a procedural problem
09:23SPEAKERwhere you vet the service and make sure that they're legit before you hook up to hook up to
09:26SPEAKERthem. - So I guess shortly after that,
09:30SPEAKERyou left Google, right? Or maybe- - Shortly, I mean, six years
09:34SPEAKERafter that, after we started it, yeah. - But along the way, one of my favorite
09:42SPEAKERthings at Org Camp every year is you take a lunchtime Saturday tour of the Chicago Google
09:48SPEAKERMuseum, which is a little nook on one of the floors in the Google office. I can't remember which one,
09:54SPEAKERbut it's got all this weird science and tech ephemera from the Chicago. It seems like it's
10:00SPEAKERmostly from the Chicago area, but also from Google's history and a bunch of other things.
10:05SPEAKERAnd there is some stuff there that I've never seen in actual real museums. For example,
10:12SPEAKERthere's an old, I think you said second generation Google rack, maybe?
10:15SPEAKER- Yeah, there's a second generation rack. A first generation rack is in the Computer Science History
10:21SPEAKERMuseum or Computer History Museum in Mountain View, right by Google. I tried to get one of
10:27SPEAKERthose, but they don't ship. They fall apart because they're literally held together with zip ties.
10:32SPEAKER- So those are the ones that when we would hear stories about Google
10:36SPEAKERengineers building machines and stacking them on, I think, what was it? Pieces of
10:43SPEAKERparticle board or something? - It was a cork board. They call them the cork board racks,
10:47SPEAKERthe first generation, because they would just basically had these bits of sheet metal and they
10:51SPEAKERwould put a thin layer of cork board on just basically to keep the motherboard from shorting
10:56SPEAKERout against the metal rack. - Yeah, and they're not screwed in. They're just rested and zip tied
11:01SPEAKERin the corner holes, right? - They weren't even zip tied. They were just rested on there.
11:05SPEAKERThe back wall of the rack was a zip tied together wall of fans, basically.
11:11SPEAKER- Wow. - And the whole thing was just like, it's just super, super sketch. If you're ever in the
11:16SPEAKERmuseum in Mountain View, go check it out. You're like, it's sort of canted a little bit to the side
11:22SPEAKER'cause it's like gonna fall over in any moment. - It reminds me of the, I can't remember what the,
11:28SPEAKERwhat the, it's the people in one of the Douglas Adams books that the supercomputer is pushing
11:34SPEAKERspace flight on and they build the jankiest spaceship in the world in the universe. And you
11:40SPEAKERlook at it and everybody's like, I can't believe this thing flew. And you kind of look at those
11:43SPEAKERold racks and you're like, I really, this is what made search good. I can't, I don't, I don't know
11:48SPEAKERif I buy this. - Well, I mean, there's a couple of things that were groundbreaking for that. First of
11:52SPEAKERall, they didn't have pretty bezels. They didn't have blinking lights. They didn't come, everyone
11:56SPEAKERdidn't have a CD drive. Everyone didn't have all these, you know, all this stuff that you expected
12:01SPEAKERin a server. And also they weren't designed to be sturdy and last. The premise was hardware will
12:07SPEAKERfail no matter what, especially as you get into larger numbers. So let's just plan on it failing
12:11SPEAKERand go as cheap as we can. The other thing was they were optimizing for square footage and the
12:16SPEAKERuse of electricity because at the time in late nineties, early two thousands data centers were
12:21SPEAKERcharging you per square foot, not per watt used. And so by maximizing this, and then by the time
12:27SPEAKERGoogle got to second gen rack, which is the one that I have in the museum here, they would base,
12:33SPEAKERthey were still renting data center space at the time instead of building their own data centers.
12:37SPEAKERBut the really cool thing is they would roll all these racks into a data center and they would use
12:43SPEAKERso much electricity and cooling that half the data center would be left empty. - Wow. - Because they
12:48SPEAKERjust couldn't put anything else in there. - So like you, you guys were trying to build, they were
12:52SPEAKERtrying to build Comtronium basically there rather than... Brad asks, what kind of systems are we
13:00SPEAKERtalking about in those zip tied racks, architecture, OS, stuff like that? - Well, it was all
13:05SPEAKERbasically a version of Linux. I don't know about the gen one rack, but the gen two rack that I have
13:12SPEAKERwas dual Pentium threes. And they were just like, they had two different kinds of racks. Okay. So
13:20SPEAKERlet me back up a second. So I decided I wanted to get a rack for the office. The whole reason this
13:25SPEAKERmuseum thing exists is because I was like, we're out in Chicago. We should have cool nerd stuff
13:31SPEAKERlike they have out in Mountain View because in Mountain View building 42, they had a first gen
13:34SPEAKERrack in the lobby and they've got all this sort of cool stuff out there. And I'm like, there's
13:38SPEAKERno reason we shouldn't have this here. So you may remember early days, Google had this thing called
13:43SPEAKERthe geo display, which is this rotating earth with little pillars of light showing where searches
13:47SPEAKERwere happening. So I built one of those machines and had that in the office, which was cool. And
13:52SPEAKERthen I'm like, we need to get a rack. And so I tried to get the gen one rack and failed because
13:56SPEAKERthey just said, we can't ship them. They'll fall apart. Then I managed to sort of wheeze on my way
14:01SPEAKERinto getting a gen two rack shipped. And so there's two of them. One is what they call a serving rack,
14:07SPEAKERwhich would have 40 machines, 20 per side. And it was basically CPU, just for serving up search
14:13SPEAKERqueries, handling, whatever types of data ads, et cetera. Then they had a rack that they affectionately
14:20SPEAKERcalled a disk full rack because it was full of disks that was for storage. And so the one that
14:25SPEAKERI wound up getting was a disk full rack, and that's 40 machines. Each machine's a Pentium 3,
14:30SPEAKERdual Pentium 3, and each machine had eight hard disks in it. Now there were originally 20 megabyte
14:37SPEAKERhard disks and they all like plugged into these little off-board daughter cards. At some point,
14:43SPEAKERhold on, 20 megabyte or 20 gigabyte disks? I'm sorry. It would have had to be 20 gigabyte,
14:49SPEAKERright? No, this was early 2000s. This was 2003 era. So I don't know that that would have been 20,
14:59SPEAKERbut yeah, it would have been 20 gigabyte. Yeah, 20 megabyte would have been like mid 90s. Yeah,
15:02SPEAKER20 megabyte would have been... See, this is the problem with starting with computers in the 80s,
15:07SPEAKERis that I get my megabytes and gigabytes confused all the time. Oh, it's easy to be off by an order
15:12SPEAKERof magnitude on the reg, yeah. Yeah, so 20 gigabyte disks. And Google was desperate for
15:18SPEAKERstorage space because they were running on a straight basis or a log. So at some point,
15:22SPEAKERthey went through and converted all these to 400 gigabyte drives. And as you well know,
15:28SPEAKERthe problem with making your basement 20 times larger, but still having the same wee little
15:33SPEAKERbasement door, gave you all sorts of IO problems. And these racks were legendary for just being
15:40SPEAKERsuper difficult to get data into and off of. But that's the rack that we have at Google Chicago.
15:47SPEAKERAnd the cool thing about it is I was so ignorant going into this, I had no idea how much this thing
15:53SPEAKERweighed. Yeah, there's 40 machines and each one has eight hard drives. So that's 320 hard drives.
16:00SPEAKERYes, and you've got the rack and you've got all the other metal and stuff. The weight of this
16:05SPEAKERrack is just under 1500 pounds. And so the delivery guy showed up and he was super confused.
16:12SPEAKERHe's like, where's the data center? And I'm like, oh, there's no data center. This is just an
16:16SPEAKERoffice. He's like, I can't deliver this. I'm like, why not? He's like, well, your ramp has a
16:20SPEAKERone inch lip at the bottom and we can't get over that. And I'm like, well, can't you just tilt it
16:24SPEAKERback? And he just looked at me like I had three heads and we wound up somehow not killing ourselves
16:31SPEAKERwith like seven or eight people getting this rack up and rolling it down the hallway to where it
16:36SPEAKERlived. And it was one of those things where it was running a standing wave through the floor
16:41SPEAKERof the office because it was so heavy. And when the building management people find out what I
16:46SPEAKERbrought into the building, they were furious because the building wasn't rated for this.
16:51SPEAKERAnd so we wound up having to literally like put it right next to a load bearing
16:54SPEAKERcolumn. Couldn't move it, had all these rules and restrictions around it.
16:58SPEAKERIt's a good thing you're in earthquake territory, I guess.
17:01SPEAKERRight. Right.
17:02SPEAKERIs there anywhere online people can see pics of this stuff? Like are these available online
17:08SPEAKERplaces or are they still kind of semi-secret?
17:10SPEAKERI don't think they're semi-secret. This was also known as a breadboard rack because it looked like
17:17SPEAKERgiant breadboards, these big metal sheets that were in there. And so I bet you if you search
17:23SPEAKERaround for Google breadboard rack, you could find something or other. They're just super ghetto
17:28SPEAKERlooking, especially now that it's a 17-year-old piece of hardware. And it's another funny story
17:34SPEAKERabout this. I actually created a sticker that I wound up giving out like 30,000 of at Google
17:40SPEAKERthat said my other computer's data center.
17:45SPEAKERSo, oh, go ahead.
17:47SPEAKERNo, I'd say this sticker was, lots of people reuse it. My other computer's a book. The
17:53SPEAKERsecurity team made my other computer's your computer and the hardware ops people made.
18:00SPEAKERMy other computer is made of razorblades and hate because these machines had so many sharp edges
18:07SPEAKERthat they were required to wear gloves after a while to handle them because people were
18:10SPEAKERjust getting their fingers sliced to ribbons.
18:13SPEAKERI was going to say, I looked at this and it reminded me of the worst of the 90s PC cases
18:18SPEAKERin that it is like the die stamped metal with the super sharp edges. This is before the
18:27SPEAKERpoint. I mean, now obviously Google buys enough hardware that they just say, hey, we want,
18:32SPEAKERthis is the giant motherboard manufacturer. Here's the motherboard we want with this stuff
18:37SPEAKERon it. And this configuration, this many slots and this many drive ports and all that stuff.
18:41SPEAKERBut back then I assume this was mostly off the shelf hardware or not really.
18:46SPEAKERYeah. Completely off the shelf, bulk stock crap. And there was a switch built into the middle of
18:54SPEAKERthe rack too. So everything basically just plugged into there. You would roll in and basically plug
18:59SPEAKERin three or four things into the wall and then plug in one ethernet cable and off you went.
19:05SPEAKERSo like you can turn up a data center really quickly and burn it in really quickly. There's
19:09SPEAKERtwo other things that merit note about this rack. And one was very controversial is that
19:15SPEAKERall the drives were just basically instead of screwing them all in because drives, drives feel
19:19SPEAKERall the time, especially when you have hundreds of thousands of them, they were all Velcro down.
19:24SPEAKERAnd go ahead. I was just saying mechanical hard drives of that time didn't have accelerometers
19:30SPEAKERand stuff if they were jostled. So like moving the rack when it was, I guess moving a 1500
19:35SPEAKERpound rack was probably challenging. So. Right. I mean, well, you wouldn't, you obviously,
19:40SPEAKERyou know, wouldn't bump it around a lot, but and that the heads would be parked when it was moving
19:44SPEAKERand you wouldn't move it when it was on. But there were two things that were interesting.
19:47SPEAKEROne is that, you know, software engineers were like, oh, this is fine. You know, these
19:51SPEAKERracks, these drives can handle this sort of, you know, vibration, et cetera, without being mounted,
19:56SPEAKERyou know, screwed down. Well, it turned out they would do fine on reads, but on writes,
20:00SPEAKERthey would fail like 50% of the time because of the vibrations. And it just got worse as they
20:05SPEAKERadded more drives. So, you know, eventually the hardware engineers did the, I told you so thing.
20:10SPEAKERThe other thing, the other thing fun to note about this is that these, these racks were just
20:16SPEAKERso poorly built and so what slammed out so quickly that it was sometimes very difficult to get the
20:22SPEAKERlower servers pushed in because the machine would, the sides would bow in or out. And so the, the,
20:29SPEAKERthe documented protocol for getting the last bottom machine back in all the way was to lay
20:35SPEAKERon your back in front of it and kick it until it went in. - The early days of tech, pretty,
20:45SPEAKERpretty janky, it turns out. What, what was the, like, how many of these were we, are we talking
20:53SPEAKERabout? Like, were these things that they were just building as fast as they could, and there were
20:56SPEAKERtens of thousands of them eventually, or was this something that there were a few hundred of,
20:59SPEAKERand it would, and that was all that was needed to run the Google of the time.
21:04SPEAKER- I'd say it's somewhere between those two or I don't know the exact number,
21:08SPEAKERbut it was definitely in the five digits range at least. I mean, yeah, four to five digits.
21:14SPEAKER- And then they, eventually they transitioned to building custom enclosures and custom racks so
21:20SPEAKERthat they can increase the density even more. And now we're looking at like four use with 32 CPU's
21:25SPEAKERand a bazillion drives and all that stuff, right? - Yeah. I mean, everything now is super custom,
21:29SPEAKERincluding the data centers themselves. Like they're, they're really amazing. I've been,
21:33SPEAKERI've been to some cool places. I've been to, I've been inside the Fermi lab Tevatron,
21:38SPEAKERI've been to Argonne national labs, electron accelerator, and I've been in a Google data
21:43SPEAKERcenter. And the, the last generation that was there when I left in 2014 blew my mind,
21:50SPEAKERabsolutely blew my mind. - The science of how can we pack more compute into the smallest amount of
21:55SPEAKERspace and cooling footprint has been, has been really, it's, it's gone a lot of, we've got a
21:59SPEAKERlot of places in the last 20 years, right? - Sure has. Yeah. It's all about electricity
22:03SPEAKERat that point. That's your, that's generally your limiting factor. - Well, yeah, there's a reason we
22:07SPEAKERbuild them in places where there's a lot of hydroelectric power or a lot of sun and, and a
22:11SPEAKERlot of cool water, right? - Right. Yeah. There's one in Finland, I think, you know. - Yeah. I know
22:18SPEAKERthat there's a bunch on like the Columbia river in Oregon and Washington places like, you know,
22:23SPEAKERyeah, it's, it's like, what are the other things that matter for data centers? I guess speed of
22:27SPEAKERlight. So distance to the places that you need the resources or does that even matter really?
22:32SPEAKER- Intersection of electricity and good fiber is, is basically it. But yeah, I mean, that's the
22:37SPEAKERreason I used to, when, when I would do tours of the office and especially with school students,
22:42SPEAKERI would ask them like, why do you think Google has data centers all the way around the world?
22:45SPEAKERAnd they would come up with a lot of different answers, but no one ever got the right answer,
22:47SPEAKERwhich is that the light isn't fast enough. So if everything's in North America, people have way
22:53SPEAKERaround the globe, you're adding, you know, a seventh of a second to everything, which is a lot.
22:57SPEAKER- Wait, when the search query takes a third of a second, a sixth of a second, then a seventh of a
23:02SPEAKERsecond lag is a lot, right? - Everything starts to add up. Yeah.
23:04SPEAKER- Let's, let's switch tracks a little bit and talk. One of the other things that's fascinating
23:10SPEAKERabout the Google museum is you have a very large electromagnet there.
23:14SPEAKER- I do. I do. It's very large, yellow 850 pound superconducting electromagnet.
23:21SPEAKER- Everyone has one. - So, so, okay. So it's
23:26SPEAKERelectromagnet. It's from the Tevatron, right? - That's right. The Fermilab Tevatron. Yeah.
23:30SPEAKER- Which was? - Which is, it was a high speed
23:33SPEAKERexperiment to find particles, right? I mean, it was high speed, high.
23:39SPEAKER- High energy. - High energy. Thank you. Yeah.
23:42SPEAKERHigh energy physics. Yeah. That was worked on for years, if not decades.
23:45SPEAKER- Yep. - Yeah.
23:47SPEAKER- I was just going to say, put you a particle accelerator in the Chicago land area at Fermilab.
23:54SPEAKERAnd this thing, the stand, I don't know if you guys built the stand or if they built the stand,
24:00SPEAKERbut the table for it is, is pretty impressive. - I bought the stand.
24:05SPEAKER- Oh, you bought the stand. Okay. - Yeah. It's rated for 34,000 pounds.
24:09SPEAKERCause so I bought one. Well, let me, let's back up. We should talk about how we got it, right?
24:13SPEAKERSo on September 30th, 2011, they shut down the Tevatron because they have the large Hadron
24:19SPEAKERCollider now. And an interesting note, Fermi did a lot of the magnet research and a lot of the
24:23SPEAKERmagnet R and D for the LHC because they were the ones with the expertise. So anyway, it was
24:30SPEAKERannounced that it was shut down and I immediately emailed a friend of mine that worked over there.
24:35SPEAKERAnd I'm like, dude, it's the Fermilab Tevatron shut down. Can you get me a magnet from it?
24:39SPEAKERAnd he wrote me back and I don't know, can we swear on this podcast?
24:42SPEAKER- Yeah, fuck yeah. - He's like,
24:44SPEAKERyou have no fucking idea what you're asking for. And I'm like, what do you mean? He's like,
24:48SPEAKERthey're huge. I'm like, well, get me a small one. And so I have a small one that's 850 pounds and
24:55SPEAKERfour feet long. The large ones are multiple tons in like 20 to 40 feet long, something very long.
25:02SPEAKERBut there's two kinds of magnets in the Tevatron. One is a steering magnet, which is basically,
25:08SPEAKERit's a dipole, has two poles and basically turns the beam a little bit to the left or a little bit
25:12SPEAKERto the right as it goes around this mile ring, mile long ring or mile diameter. I think it's
25:17SPEAKERmight be mile diameter. - I think it's mile circumference.
25:19SPEAKER- Mile circumference. It's big, whatever it is. I mean, it's so big they have Buffalo or bison
25:24SPEAKERliving in the middle of it, right? - The comments will let us know.
25:27SPEAKER- Yeah. And then the small ones are quadrupoles. They're focusing magnets. There's four poles.
25:32SPEAKERThey basically squeeze the beam down every so often as it goes around. There's about 774
25:40SPEAKERof the dipoles in the ring. And then there's 250 of the quadrupoles. And so in any case,
25:47SPEAKERmy friend, he humored me and we went back and forth and basically just over a year later,
25:53SPEAKERI got the magnet. Like it had to go through so much red tape and everything. And I bought it
25:59SPEAKERoff of eBay. That's how I got it. They called me up and they said, "We finally figured this out.
26:04SPEAKERHow do you want to buy it?" I said, "What is easiest for you?" And they said, "Really eBay's
26:08SPEAKERthe easiest." And I said, "Well, can you make it a buy it now? Cause I don't want someone else to
26:12SPEAKERstack this up." Because they had to go to all this thing where they had to put it up for scrap. I
26:16SPEAKERmean, no one bought it for scrap. Like no one would buy it. And so I clicked the buy it now
26:20SPEAKERbutton off of eBay and had it shipped to the office. I bought a table for it that was rated
26:26SPEAKERfor one ton. And I'm like, "I figured 2X the rating, that'll be totally fine." And this table
26:32SPEAKERwobbled back and forth so much. It scared the heck out of our facilities people. And like,
26:37SPEAKERyou need to buy a bigger, better table. So the table we have now is like 350 pounds and you could
26:42SPEAKERpark a truck on it. It's bananas. - I have to ask how much an 800 pound Tevatron magnet costs on
26:52SPEAKERbuy it now on eBay? - Well, so when they said, "Can you buy it now?" They sort of did this sort
26:59SPEAKERof sucking in their teeth. "We're going to have to raise the price a good bit if you put it on
27:04SPEAKERbuy it now." And I said, "Well, okay. What is the price going to be with buy it now?" And they said,
27:09SPEAKER"It's going to be $150." - You're paying less than a buck a pound. That's pretty good. - It's
27:20SPEAKERbasically a scrap price, but only two of these magnets have ever left Fermi. One's at the
27:26SPEAKERSmithsonian, I believe it's in storage. And this one is on display at the museum in Google Chicago.
27:31SPEAKER- So Brad wants to know, is it safe to assume you've never tried to put power on the magnet?
27:36SPEAKER- It is very safe to assume that for two reasons. One, we don't have that kind of power. Two,
27:43SPEAKERwe don't have liquid helium to cool it. - Oh, man. Always a problem, it turns out,
27:51SPEAKERthe liquid helium. - Yeah, yeah. And it's got these cool sort of bendy connectors on the side
27:56SPEAKERwhere they use to connect everything together. And you can look down the beam line. It's really
28:02SPEAKERan amazing piece of high energy physics history. And I'm super happy to have that. But there was
28:07SPEAKERsome concern about people, is it going to affect laptops or phones walking by and that kind of
28:12SPEAKERthing? And they're like, "No, it's basically a giant hunk of iron at this point." Although the
28:17SPEAKERmagnet itself inside the iron yoke is niobium titanium. - You don't want to degauss every
28:23SPEAKERcredit card in the building when you fire that thing up. I think that you said the one that you
28:27SPEAKERhave was a spare. Is that right? - Yeah, it was a spare. As my contact at Fermilab put,
28:35SPEAKERyou're never going to remove any magnet from the Tevatron itself because they are imbued with
28:39SPEAKERadministradium, which is to say there's so much red tape involved in getting something out of
28:44SPEAKERthere because basically they're hot, having been shot through with protons, high-speed protons for
28:49SPEAKERso many years. There's a little level of radiation in them. So this was a spare. And they cleaned it
28:54SPEAKERall up and they repainted it and they re-stenciled it and it's all pretty when we got it. But I had
28:58SPEAKERto buy an engine hoist to move it off of the pallet onto the thing because how do you move
29:04SPEAKERsomething that heavy? - Somewhere at Google Chicago, there's an engine hoist just rolling
29:08SPEAKERaround in a back storage room. - That is correct. - Very good. Did you have all the stuff in the
29:13SPEAKERold Google office before they moved to the new place? - Yeah, the old space was the one that had
29:18SPEAKERthe problem with the weight of the rack. The new space building is in an old cold storage building.
29:23SPEAKERYou could park cars in there. It's no problem. - Okay. That makes more sense. Yeah. So one of
29:28SPEAKERthe other things that's really fascinating about the space in Google, and there's a bunch more
29:33SPEAKERstuff we could get into, but your collection of esoteric storage devices there has some stuff that
29:40SPEAKERI'd never seen before, which is, I mean, look, I've been doing this for a little bit. And you have
29:49SPEAKEReverything from old analog formats going all the way back to early record cylinders and things like
29:55SPEAKERphonograph cylinders and things like that, all the way up to micro, micro cassettes and weird disc
30:03SPEAKERformats. I think there's a German phonograph tape thing that's in there somewhere. - The teffy phone,
30:09SPEAKERyeah. - That one is the weirdest because it seems like it's a vinyl tape, right? Like a kind of
30:17SPEAKERthick vinyl tape in a cartridge. - Read by a needle. Yeah. It's the weirdest thing ever. But
30:25SPEAKERit's a beautiful red vinyl, whatever type of plastic type of thing. I don't know what it is,
30:30SPEAKERbut yeah, it's pretty cool. - That one, I assume you have to rewind. It's a record you have to
30:35SPEAKERrewind, but I don't know for sure. - No, I don't think so. I think it's like an A-track. I think
30:39SPEAKERit's got a loop. - So I don't know how A-tracks, so A-tracks are one of those ones I never bothered
30:44SPEAKERto spend the 20 minutes on Wikipedia to figure out how it works. What are the more interesting,
30:52SPEAKERlike, I mean, obviously there's like zip discs and side quest discs and all those big external
30:56SPEAKERhard drive things in there, but what are some of the, do you have some highlights and favorites
30:59SPEAKERfrom that section, Fitz? - Yeah. So the teffy phone is by far one of my favorites. I just,
31:06SPEAKERI love that idea. I love the film collection 'cause I've got everything from eight millimeter
31:12SPEAKERto 35, to 70, and then to IMAX. I think I let you pick up that IMAX trailer that I've got. It's like
31:18SPEAKERa 30 second IMAX trailer that weighs 15 pounds. - It's a disc of celluloid that is- - It's mylar.
31:27SPEAKER- Oh, sorry, yeah, mylar 'cause that celluloid would have caught on fire by now probably.
31:32SPEAKERIt's really, really heavy. And like, it's really fun when you have people and everybody's holding
31:39SPEAKERup the different sizes of the film. It's really nice to, it gives you a really firm representation
31:45SPEAKERof, hey, here's the difference in, you know, everything from, like you said, home eight
31:50SPEAKERmillimeter all the way through the IMAX and the CinemaScope and all the different projection
31:54SPEAKERformats. - Yeah. And to give you a feeling for like what you're talking about, take your pinky
31:58SPEAKERfingernail and look at it. That's about the size of an eight millimeter frame. So you can hold that
32:04SPEAKERto the light and you can maybe make out if it's a person or a house or something. The IMAX frame is
32:10SPEAKERhold your phone up, hold your extra large phone up, which I have one. That's probably two frames
32:15SPEAKERside by side of IMAX right there. So you can not only tell what's in it, but who's in it. Like,
32:21SPEAKERone of the frames I got is Robin Williams on a horse. And you can clearly see it's Robin
32:25SPEAKERWilliams standing on a horse because it's just that big. And there's just that much resolution
32:30SPEAKERthere. - Yeah, the IMAX is almost like, it's like the size of your palm almost, depending on how big
32:36SPEAKERyour hands are. - Yeah, it's ridiculous. - Go ahead. - I was gonna say that the other, going way back,
32:46SPEAKERthe giant digital disc pack that I've got, it's 175 megabytes. I think it's 12 or so,
32:53SPEAKER14 inch platters, and it weighs in the order of 40 pounds. And this was the other thing they would
32:59SPEAKERgo and put in those giant dishwasher size disc drives in a data center. And you'd spin the top
33:05SPEAKERoff and you close the lid of the dishwasher and the heads come in from the side and read the data
33:10SPEAKERoff of it. - It looked, those things looked like a tub washer, really. - Yeah, yeah. They're just
33:16SPEAKERenormous. - We did a video at Tested years ago with one of the, I guess it's one of the things
33:23SPEAKERthat followed that, but that IBM, I want to say it's like a two megabyte hard drive or something
33:27SPEAKERfrom the sixties that Adam got someplace. When we hung that from the ceiling in his shop, it took
33:36SPEAKERthree people and a chain hoist to get it up there. And I feel like it was in the early days of us
33:44SPEAKERworking with Adam at Tested. And I was standing there with this 80 or a hundred pound disc drive
33:50SPEAKERon my shoulder, on my hands, up above my head. And I was thinking, man, I hope his ceiling
33:56SPEAKERjoists are as strong as he said they are, 'cause otherwise I'm going to get flattened here. - That
34:00SPEAKERwas the first thing I thought when he pointed it out to me when I was over there a couple of years
34:04SPEAKERago. I was like, oh man. Yeah, I mean, that's like, that was big data, right? This stuff was
34:10SPEAKERheavy. But like the whole point around this was collecting things that for storing data over the
34:16SPEAKERyears, 'cause so many people collected old machines, but like that gets really bad really
34:21SPEAKERquickly. Like in, as in like build a warehouse because it's just so big. So as I sort of like
34:27SPEAKERwent forward and backward, different folks would send me different things. Different folks would
34:31SPEAKERrecommend stuff. I've got a deck of punch cards, which is really cool. Like I just adore punch
34:36SPEAKERcards 'cause I grew up with them 'cause my dad did computer stuff and they were everywhere in
34:40SPEAKERour house. They were for grocery lists, you know, you could cut them into strips and make little
34:44SPEAKERchains of like decoration, all sorts of stuff. - My grandparents' house had that as well. My
34:49SPEAKERgrandmother worked on early punch card stuff in the fifties and sixties and seventies, so.
34:53SPEAKER- It was so interesting. It was so tangible, you know? Yeah, million different kinds of tapes.
34:58SPEAKERTapes, I think is the most obnoxious category because, and I will back up and say, when I
35:04SPEAKERstarted doing this, I started collecting this because I'm like, oh, this is a nice finite thing
35:08SPEAKERto collect. I was wrong. It is absolutely infinite because there's so many- - Storage formats.
35:14SPEAKER- Yes, so there's so many different, really weird, not broadly used formats out there,
35:20SPEAKERlike the Tefiphone, you know? They probably made a thousand or a couple of thousand of those,
35:25SPEAKERI guess, you know? - Well, and especially in the early days of computing, there were so many
35:30SPEAKERpeople who had shifted from making, so many companies that shifted from making analog tape
35:34SPEAKERstuff to digital tape stuff. There were probably, I remember at one point in the nineties, I was
35:40SPEAKERspecking out a backup system for our server cluster at the place I was working. And there were
35:45SPEAKERprobably 20 different vendors making completely incompatible tape backup systems that could hold
35:52SPEAKERanywhere from what, maybe one gig to 10 gigs per tape or something like that. And that stuff could
35:59SPEAKERget out of control. The different disc formats is a fun part of your thing. You know, I did notice
36:06SPEAKERyou didn't have one of those little pocket zip discs. I don't know if you remember those,
36:12SPEAKERbut they were maybe 40 or 80 megs. They were mostly for MP3 players and they were two inches
36:19SPEAKERacross, maybe an inch and a half across. I think they only ever made one device that played them.
36:24SPEAKER- A removable zip? - It was a removable zip
36:27SPEAKERfor portable use. I might have one in the garage that I can bring out next time I see you.
36:32SPEAKER- Yeah, I've never actually been aware of those. So I've got a spreadsheet that,
36:37SPEAKERI should share this with you, that's got a list of all the known formats or particular types that I
36:43SPEAKERhave so that folks can go through and be like, oh yeah, I have that or add to it or remove from it.
36:47SPEAKERYou know, that's how Rob Pike actually, I was having dinner with him years ago and
36:52SPEAKERwe were talking about this and he was like, oh, do you have a 12 inch swarm disc in there? I'm like,
36:56SPEAKERwait, what? 'Cause I've got an optical, I've got a three and a half and a five and a quarter.
37:04SPEAKERI didn't know they made a 12, but I found one and it's like a laser disc in a cartridge. It's
37:08SPEAKERamazing. - Wow.
37:09SPEAKER- Yeah, they're huge. - Wow, that's bonkers. I guess it still
37:13SPEAKERfits in a rack, right? - Yeah.
37:15SPEAKER- The audio cassettes, the scale at which Sony made audio cassettes from very, very tiny spy
37:24SPEAKERtape size all the way up to, what's the big one, like five inches across or something?
37:29SPEAKER- Oh, it's at least five. Yeah, it's five, six, something like that. It's just enormous.
37:34SPEAKER- Yeah, but it looks like a normal, it looks like a micro cassette, but it's just huge.
37:39SPEAKER- Yeah, and the fun ones, again, it comes down to like weight and size. So I've got the tiny little
37:46SPEAKERmicro cassettes, which are smaller than the ones people are familiar with. Sony made this itty
37:49SPEAKERbitty one that's like an inch across. And then, you know, we've got the reel to reel stuff,
37:53SPEAKERgot a quarter inch. I don't actually have half inch, I don't know. I've got the nine track
37:57SPEAKERtape, which is all the old spinny tapes you'd see in the old data center pictures, early original
38:03SPEAKERTron, you know, type stuff like that. And then I've got one inch mastering tape, and then my
38:07SPEAKERfavorite two inch mastering tape is basically for 24 track analog recording. It's the kind of stuff
38:14SPEAKERthat the Beatles like, you know, recorded on in the very towards the Abbey Road days. But this,
38:21SPEAKERit's just so heavy. It's just, it's this, you know, 10 inch reel that weighs like
38:26SPEAKER15 pounds or something. - Oh yeah. I mean, yeah. When you think about
38:34SPEAKERthe density of data storage, what's happened to the density of data storage over the last 20 years
38:38SPEAKERand now, and the cost at the same time, like I, the other day I had to buy 128 gig micro SD card.
38:44SPEAKERIt's smaller than my pinky nail. It holds 128 gigabytes. And I think I paid 20 bucks for it,
38:50SPEAKER25 bucks for it, maybe like, yeah, it's, it's amazing. - Well, that's the thing that I point
38:55SPEAKERout to folks is I've got this display at Google with all this stuff in it and a whole lot more,
39:00SPEAKERwe've only really scratched the surface of it, but you could basically put all of this onto a single
39:06SPEAKERdrive that you could go out and buy commercially today. - And I think there's a, like the space is
39:12SPEAKERlimited that you have for the museum there, but there's a fascinating, like, you know, you know
39:18SPEAKERhow they present paleontological displays at natural history museums now, and they start,
39:22SPEAKERlike they show the bones through time. So you understand that the Tyrannosaurus and the,
39:26SPEAKERand the brontosaurus didn't live at the same time period because, you know, they were separated by,
39:29SPEAKERyou know, 30 million years or whatever. - Right. - There's an argument that like,
39:33SPEAKERthere'd be a neat way to put all that stuff on a timeline. So you can actually see it and see what
39:38SPEAKERhappened to the data density and the cost and the, and the weight and all that over, over the,
39:42SPEAKERyou know, the 40 years that that stuff has been, has been made. - I think I'd love to do something
39:48SPEAKERlike that. That's a great idea. - I think, I think that's a pretty good place to wrap it up, Fitz,
39:53SPEAKERif you, if you, unless you have, well, let's see, did we, oh, I got some questions coming in from
39:59SPEAKERBrad, hot text questions from Brad. I'll, I'll, he's typing. - Well, while he's typing, I'll throw
40:09SPEAKERa couple of other interesting things on there. I've got, we got in the museum, one is the,
40:13SPEAKERso we've got laser discs, which people have probably seen somewhere, 12, it's like a 12
40:17SPEAKERinch CD, right? We also have capacitance, capacitance electronic disc, which is a predecessor
40:22SPEAKERto the laser disc that used a very fine needle, like a record for tracking. And, but the problem
40:28SPEAKERis, is that they couldn't get any dust in there because it was, the needle track was still fine.
40:33SPEAKERSo RCA put them in a, in a case, a sleeve. So you would stick this long sleeve into the front of
40:39SPEAKERthis reader and then pull it out and it would, it would sort of grab the disc. - And would the sleeve
40:44SPEAKERlike, did have brushes to brush dust off as it was going in? - It did, it had a gentle little like,
40:50SPEAKERyou know, linear like brush at the end of it to sort of keep the dust out. But they still
40:55SPEAKERgot dusted and went, got destroyed. - So this is a little bit of a swerve back to the start of the
41:00SPEAKERconversation, but you said you started, you considered it a late start that you only got
41:04SPEAKERinto, into coding at age 24. This is from Brad, by the way. - Yeah. - What would you say to someone
41:09SPEAKERwho's developing an interest in the subject past the age of 40? - Go for it. Like, I don't think
41:14SPEAKERanything is, is too late. I mean, I got, I got a late start relative to most people that write
41:19SPEAKERsoftware. You know, I mean, most of the people I hire today are, you know, just out of college
41:24SPEAKERor whatever, not, and they've been doing it since high school. People are starting to write code.
41:28SPEAKERIt's, it's more accessible for people to not only write code, but get the resources to learn about
41:33SPEAKERit. Like in 94, when I was first starting all this, it was like save up money to buy an O'Reilly book.
41:38SPEAKER- Yeah. - Now it's just, use Google and Stack Overflow and you're halfway there, right? For
41:43SPEAKERwriting code. - Yeah. And like before, before Linux, if you wanted a code, like you had to
41:49SPEAKERsave money to buy your environment to write code and compile in, right? - To buy a compiler. Yeah.
41:54SPEAKER- So I would say that I don't think anyone's too old to learn it. Like just, if there's something
41:59SPEAKERyou're interested in or curious about, start digging in and start, find something you want
42:04SPEAKERto build, just like, or, or find something, some way of learning that works best for you.
42:08SPEAKERI'm not a classroom learner. I'm a, I'm a sort of hacker type of person. People would say I'm
42:13SPEAKERself-taught, but I say that I went and found a bunch of teachers when it came to this stuff. So
42:18SPEAKERif that's what works for you, go for it. If, if you're a classroom learner, take a class,
42:23SPEAKERbuy a book, whatever works for you. But I don't think there's any too late anywhere around
42:28SPEAKERlearning how to program. - Do you, do you like, does it matter what language, like what you start?
42:36SPEAKERLike, I know, like I learned to code. I took CS 100 when I was in college and was like, oh,
42:40SPEAKERthis probably isn't for me. And then 20 years later I got an Arduino kit and that information,
42:45SPEAKERthat knowledge translated straight ahead into, into how to write code for Arduinos.
42:49SPEAKERBut also at the same time, there was something really satisfying to me about making a change on
42:54SPEAKERmy computer that could affect something physical in the world, like turn a motor or light up a
42:58SPEAKERlight, make a buzz sound, whatever it was. And that, that made it a lot easier for me to get
43:03SPEAKERgoing. But I know like a lot of people teach Swift. A lot of people teach Python, learn Python
43:08SPEAKERnow. What's where, where would you recommend starting or do you think it matters? - I think
43:12SPEAKERa lot of it depends on what your interests are to your point. Like you, you found the interaction
43:17SPEAKERbetween software and the physical world interesting. And that, that sparked your interest,
43:21SPEAKERwhich is great. Like then Arduino is definitely the way to go with that. I, you know, when I was
43:27SPEAKERa kid, I had an Apple two C and my brother and I, you know, I learned basics and 6502 assembler.
43:33SPEAKERAnd then I got a Mac and I learned some Pascal and a tiny little bit of C, but it never really
43:39SPEAKERgrabbed me. I just, there was something about the client side stuff that I wasn't interested in.
43:44SPEAKERWhen my little brother, my little snot notes, brother who was in college at the time studying
43:50SPEAKERengineering he introduced me to Unix and Pearl. And that was my gateway drug. That was the thing
43:57SPEAKERthat caught my, because I was writing something that I could affect on a webpage that was just
44:01SPEAKERgo out there and other people could see. I thought that was, that graphs got me excited. So what is
44:06SPEAKERthe thing that gets you excited? I will say, I think you can make a mistake if you started like,
44:10SPEAKERyou know, doing X 86 assembler, you're probably going to quit before you get anywhere.
44:14SPEAKERSo, you know, I think Python's a great thing. I think Arduino is great. I think
44:19SPEAKERthings that are easier, I would avoid compiled languages. Frankly, I would start with C or Java.
44:26SPEAKERStart with something that can get you a return on investment pretty quickly. The reason I think
44:30SPEAKERPearl worked really great for me is because Larry Wall was a classicist. I was also a classicist,
44:35SPEAKERso I could learn Pearl much like I had learned Latin, Greek or Italian. And it was like little
44:41SPEAKERbits at a time have speak like baby Pearl and then get adolescent Pearl and then et cetera.
44:45SPEAKERSo, you know, find what you, find what works for you.
44:48SPEAKERWell, and now, now everybody has like, it's funny. Somebody, somebody asked me once what
44:54SPEAKERthey should do. Like what do they need to buy anything? I was like, yeah, maybe a second monitor
44:58SPEAKERbecause it turns out almost all of the programmers I know now almost always have like the complete
45:04SPEAKERreference guide up on the left side on the, on their other monitor, you know, because,
45:10SPEAKERyou know, when you're dealing with these more complex languages now, there's a lot to keep
45:13SPEAKERin your head and, and you kind of, you need to think about how the algorithms work, not
45:18SPEAKERnecessarily the syntax or whatever of what you're working on in all cases. So there's nothing wrong
45:23SPEAKERwith having that second monitor open with, with your, with your code examples and your,
45:27SPEAKERyour reference guides and all that stuff. Yeah. I mean, I think there is a,
45:31SPEAKERit is a mistake to think that people who have been coding for many, many years
45:36SPEAKERjust remember everything and just sit down and like, you know, start typing something out like
45:39SPEAKERthat right out of their head. Absolutely not. Like I still forget, like, what is a particular
45:45SPEAKERfunction to do something? What are the order of arguments? What's this magic number? Yada, yada,
45:49SPEAKERyada. Reference is the way it goes. Like I pulled up good. I've seen swordfish. I know how coding
45:55SPEAKERworks. Okay. Oh God. What, what's the worst movie representation of programming you've ever seen?
46:04SPEAKEROh, I don't know. I really don't know. Like it's not a there's, I can't pick out one particular
46:10SPEAKERmovie because there's, it's usually just the, like the silly stuff when something goes wrong
46:13SPEAKERand you hit a bug and like the keyboard sparks, like that's probably the thing that's, you know,
46:18SPEAKERor the other thing is the w when I was, when I was at Apple, we were creating these widgets for
46:22SPEAKERweb stuff that we could reuse. And one of the ones we, one of my colleagues created was called
46:27SPEAKERthe Hollywood login panel. And it was like little dancing lights around you, like your username and
46:31SPEAKERlogin. But if you just typed in like admin as the username and hacker, it just bypassed all login
46:36SPEAKERand got you into the app. Like, Oh God. Yeah. What, what, well, okay. So then what's the best,
46:42SPEAKERdo you think the one that was, was well done? It's not a movie, but I think Mr. Robot was really
46:48SPEAKERwell done. Okay. You know, I haven't seen a lot of, I haven't seen a lot of the other stuff like
46:53SPEAKERHalton catch fire, a bunch of friends like that. I haven't seen it yet. It's up in my queue. But like
46:57SPEAKERI watched the first season, Mr. Robot, and I was like, Oh, they definitely were talking to some
47:00SPEAKERpeople who were in computer security here. Yeah. That one, I remember watching the first
47:04SPEAKERseason of that and was like, Oh yeah, they have a, their consults are really good on this. Yeah.
47:10SPEAKERI've got another question from Brad. This one is in the context of Git eventually coming along and
47:14SPEAKERsupplanting subversion as someone who worked closely on a major project used throughout
47:18SPEAKERthe open source community, like SVN, what did it feel like to see what was basically
47:23SPEAKERthe next generation version of your work come along and become more prominent?
47:27SPEAKERI think my disappointment with it, I was certainly disappointed. Let's be honest.
47:35SPEAKERMy disappointed stemmed from a different place than most people would think.
47:38SPEAKEROne is that the user interface for Git is just terrifyingly bad. Like I often,
47:45SPEAKERI often say half jokingly only have maybe a third jokingly that you should never let kernel
47:49SPEAKERprogrammers write usual land software because it's, it's like a, it's like a Gatling gun that
47:54SPEAKERjust like randomly shoots at different people. And you're, you're going to blow your foot off
47:58SPEAKERlike once a week with it. And people have built sort of guard rails around it and wrappers and
48:02SPEAKERthings like that. But the other thing that really concerned me was the fact with distributed version
48:07SPEAKERcontrol, it's really easy to hide in a cave. It's especially for younger people who are learning.
48:12SPEAKERAnd, you know, Ben and I wrote about this in debugging teams. If you, if you do all your
48:19SPEAKERwork and then don't share it with someone, because you, you can just like commit it all locally,
48:23SPEAKERand then you can rebase and basically just burp out your perfect finished product and share with
48:28SPEAKERthe rest of your team. A you're more likely to not share more often, often enough so that you can
48:33SPEAKERlike get feedback, ongoing feedback. And two, you're likely to make mistakes that you don't
48:37SPEAKERhave to make. But three, like people make mistakes all the time. Like just don't be afraid to show
48:42SPEAKERyour work kind of a thing. And so those were the big concerns. But beyond that, like beyond a
48:47SPEAKERcertain point, I think subversion actually didn't meet the needs. It continued to be centralized.
48:53SPEAKERIt was still built on the repository that I haven't looked at it since 2007 is the last time
48:58SPEAKERI worked on subversion. But at the time that the repository is still built in this data structure
49:03SPEAKERthat I didn't think fit it fit what they really needed to do proper merge tracking and some of
49:08SPEAKERthe features that they needed to move forward. So I'm, I'm glad that something else, something
49:14SPEAKERbetter is out there tool wise. Personally, like for a lot of my pet projects, I've used Mercurial
49:21SPEAKERfor a long time because it actually had a very good interface and I liked it a great deal.
49:26SPEAKERBut like GitHub has done a great job of basically building an ecosystem and a support structure
49:32SPEAKERaround Git. And so I'm totally fine with it now. I'm glad that I was just, I will, I will say I'm
49:39SPEAKERglad I was part of what enabled the world to get to Git because I was, I was the person who finally
49:46SPEAKERshipped CVS to SVN, which was the thing that converted millions and millions and millions of
49:51SPEAKERlines of code from the old CVS system to subversion. And once, once you got it out of CVS in the
49:57SPEAKERsubversion, then you could take it anywhere. But CVS was such a janky piece of crap. I actually
50:04SPEAKERgave talks about this piece of software. It was called making apples from applesauce because
50:08SPEAKERCVS was such a mess that by analyzing it and running, I don't know, a dozen passes through
50:14SPEAKERit to get it into subversion. I could actually tell you information about your data that you
50:18SPEAKERcouldn't find out from CVS because it didn't version operations, but I could basically
50:22SPEAKERwindow box these things and say, Oh, well, this operation must have happened in here. So we'll
50:26SPEAKERjust bisect that and drop it in. So I'm glad we got people into that. We wound up in the end with
50:32SPEAKERa tool that's much more usable, but I still do worry about people sort of hiding their work and
50:36SPEAKERbeing afraid to share as much. Well, there's a, there's, there's a one universal for, we run
50:42SPEAKERfoo on Git and I'm going to GitHub with the LDS stuff and all that. One of the things that I've
50:50SPEAKERfound is that everybody who uses Git on the reg has a post-it note or a note document someplace
50:55SPEAKERwith like, here are the commands to use when you fuck it up really bad. And, and everybody's
51:02SPEAKERcommands are a little bit different. It's yeah, it's, it's, there's a little, there's a, there's
51:08SPEAKERa you can do so much and so many different ways that it's, there's an inconsistency that's
51:14SPEAKERchallenging for certain, certain types of people. So I totally get that. People say, you say a lot
51:22SPEAKERabout command line software, but I really, this is something that I actually spent a lot of time on
51:26SPEAKERwhen I was working in subversion for consistency and a smart command line syntax that people could
51:34SPEAKERactually re remember and get help with, et cetera. There there's, I'm no great designer when it comes
51:40SPEAKERto like web stuff, even though I've been doing web stuff for the past five years at talk.
51:44SPEAKERBut when it came to command line stuff, I really put a lot of thought into the design of that
51:48SPEAKERcommand line. Brad says last question, not to put you on the spot, but would you come back again
51:54SPEAKERsometime? I'd love to pick your brain for an awfully long time. Absolutely. I really enjoy
51:58SPEAKERchatting with you. This is super fun. It's just like having a chance to catch up with you and
52:01SPEAKERnerd out about stuff. Well, look, that's, that's the, you know, the, the, the joke I often see is
52:06SPEAKERthat it's, it's a, it's a, the podcast is an excuse for middle-aged white men to have meaningful
52:10SPEAKERconversations with each other. So there you go. Fitz, this is opportunity for you to plug some
52:18SPEAKERstuff. Talk obviously is awesome. Yeah, thanks. I that's what I've been doing for the last five
52:24SPEAKERyears. I left Google to start talk with Nick Kukonis and you know, we're, it's, it's a weird
52:29SPEAKERworld we're in right now with COVID-19 and everything. But we basically, when we started
52:35SPEAKERworking at home our team sat down and said, well, restaurants are closing globally and we're, we're
52:40SPEAKERa system that helps restaurants book. What can we do to help these restaurants? You should explain
52:44SPEAKERwhat talk does normally before I think, because I got a lot of people don't know. I was so talk,
52:50SPEAKERexplore talk.com. We are a booking system. Some people will say like open table and I would say,
52:55SPEAKERyeah, except it doesn't suck. We were basically, you know, a modern version of a booking system
53:03SPEAKERfor restaurants where we do everything a restaurant needs, which up until two months
53:07SPEAKERago included things like regular reservations, like cost zero dollars, events, prepaid dinners,
53:14SPEAKERpre-fixed dinners, where you pay for the whole meal before deposits, where you can pay part
53:17SPEAKERof the meal before add-ons where you can choose like a wine pairing or a truffle supplement, or,
53:23SPEAKERyou know, a cookbook, cheese course, cheese course, that kind of thing beforehand.
53:29SPEAKERAnd so that's, that's what we're doing. And also a whole lot of really good software to help the
53:34SPEAKERrestaurant manage everything better. A complex customer relationship management software that
53:40SPEAKERwould allow you to remember that, you know, Oh, this person's left-handed, they drink sparkling
53:44SPEAKERwater. Their spouse's name is Steve. Their anniversary is this, et cetera. So you basically
53:49SPEAKERkeep information about your regulars and give them a better treatment because the days of
53:54SPEAKERthere being a single maitre d that remembers everyone are long gone effectively. And so this
54:00SPEAKERis what we started doing in December of 2014. I've got a fantastic product team that is doing
54:05SPEAKERamazing work. It's a beautiful piece of software too, if you've ever used it. But when basically
54:12SPEAKERevery restaurant in the world shut down in the course of a week, we said, well, what do we do?
54:17SPEAKERWhat can we do to help these restaurants with the software we have? And my team worked literally
54:21SPEAKERfor like five days straight and burped out a thing called talk to go, which is also does delivery and
54:26SPEAKERpickup now. So we do all that stuff plus delivery and pickup. And the cool thing about that is most
54:32SPEAKERdelivery or pickup services are the kind of thing where you get on the phone or you pull up your app
54:37SPEAKERand you order it and it shows up as soon as possible. Right. And that might be 25 minutes,
54:42SPEAKERbut if it was super slammed, it might be an hour and a half, which tends to get into like, Oh,
54:46SPEAKERthis sucks. I'm really hungry territory. But our system's all about time slots. And so we basically
54:53SPEAKERallow a restaurant to say, yeah, I can do up to 20 meals every 15 minutes from five o'clock to
54:57SPEAKERnine o'clock. But between seven and eight, I can do 25 meals because we're really going to push hard.
55:02SPEAKERAnd so people can book in advance. So if it's like Friday morning and I'm like, oh,
55:07SPEAKERhey, I want to order from Loyalist tonight and get a burger. Yeah. What time do you want to pick it
55:12SPEAKERup? Oh, I'll pick it up at seven up. Seven's already sold out. Six thirties already sold out.
55:17SPEAKERI'll pick it up at six 15. That's cool. And so the restaurant, you talk about flattening the curve,
55:22SPEAKERthis peak that hits like at seven o'clock, you have to push that down and say, this is,
55:27SPEAKERthis is as much as our our kitchen can handle and then push it out to the shoulders so that you can
55:34SPEAKERactually get, give more people meals. You can do more business with fewer people in the restaurant,
55:40SPEAKERwhich is important because they're having to space things out right now. So anyway,
55:44SPEAKERwe launched that mid-March and it's been just a roller coaster ever since they're trying to get
55:48SPEAKERmore restaurants onto this. So yeah, it's, it's interesting because it's so when I don't remember
55:53SPEAKERif it was you or Nick that pitched me on the original idea for talk at OrdCamp five, four,
55:58SPEAKERfive years ago. But, but Nick is a restaurateur who owns the Alinea group in Chicago. And like,
56:07SPEAKERthis seemed, whereas the people I know that work with OpenTable describe it as something
56:11SPEAKERthat they have to do to remain in business. Like you guys took this from both a customer,
56:17SPEAKERyou know, diner perspective, but also from the restaurant perspective to, to build something
56:21SPEAKERthat's useful and good. And like you started doing, you started out with prefix, prefix menus
56:29SPEAKERso that, and, and I think it was Nick was telling me about this was the idea was that you could get,
56:32SPEAKERif, if you know that Wednesday night is always thin, you have the same number of people at the
56:36SPEAKERrestaurant on Wednesday night, working at the restaurant on Wednesday night as you do on
56:39SPEAKERSaturday or Friday. So, so you're paying those people regardless, your, your cost is the same.
56:44SPEAKERThe only thing you're not spending on is the extra food that that's going out on the tables that
56:48SPEAKERnight. But if you can get more people into the restaurants on those nights, even if it means
56:52SPEAKERreducing your price on the less demand, less in demand nights, then, then you can, you could do
56:56SPEAKERlike prefixes on Wednesday nights and make more money for the restaurant, have more people eating
57:01SPEAKERyour food. And, and it was a, it was a good experience for everybody all the way around.
57:05SPEAKERIt seems like this is, this is a natural extension of that from, you know, for, for the challenging
57:10SPEAKERtimes that we face right now. Absolutely. I mean, that's where Nick did actually have a talk about
57:14SPEAKERthis at our camp. I think it would have been six years ago because some of my current employees,
57:17SPEAKERsome of my current engineers actually had gone to that talk. And then they're working with me today.
57:22SPEAKERBut yeah, originally it came about because two things, one no-shows even restaurants that have
57:29SPEAKERmore demand than they have availability still have no-shows. And it's not just, you know a table of
57:36SPEAKERfour doesn't show up at all. It's a table of four shows up as three. And, and I will say this,
57:40SPEAKERthere are these people out there who I would call monsters who booked tables of four for two people,
57:44SPEAKERjust cause they want more space. Oh, you're basically depriving a restaurant of revenue
57:48SPEAKERbecause you want to be more comfortable in a restaurant. Well, it's not gonna be a problem
57:52SPEAKERgoing forward cause you're going to get more space. But the second thing is that variable
57:56SPEAKERprice. The thing you pointed out is that so many people want to go on a Friday or Saturday night,
58:01SPEAKERbut not as many people want to go on a Tuesday or Wednesday. So just like if you go to a ball
58:06SPEAKERgame and you're sitting right behind home plate, you know, paying 300 bucks, you know, people don't
58:10SPEAKERsit there and look back up in the, in the, you know, seats way up in the second upper deck and
58:15SPEAKERsay, God, you know, I can't believe this guy's only paid 70 bucks. Nobody's sitting there crying
58:20SPEAKERover that. And so the thought was, well, if, if the same way as if there's the same thing with
58:25SPEAKERdemand in a restaurant, what's raised prices a little bit on Friday and Saturday at seven and
58:30SPEAKEReight, but then lower prices on Tuesday and Wednesday. And what that came out with is the
58:36SPEAKERsystem where you, you start just again, that curve at peak curve, you spread it out and you fill the
58:41SPEAKERrestaurant more. And, and another thing you said is the real thing that brought me to want to create
58:46SPEAKERtalk with Nick wasn't that I was just wasn't happy at Google. I was having a blast at Google.
58:51SPEAKERThe fact is, is that the idea, the original concept of talk is something that's good for
58:56SPEAKERboth the guests and the restaurants. So we agreed early on that if we can align our business model
59:02SPEAKERwith keeping the guests and the restaurants happy, then we've got a recipe for success versus a lot
59:09SPEAKERof systems that we saw out there that were just like, Oh, we're going to build this concierge
59:12SPEAKERservice to help, you know, rich students that forgot to make a booking. They can pay an extra
59:15SPEAKER50 bucks and get a table that they wanted Friday at five or seven o'clock, which doesn't help the
59:20SPEAKERrestaurant at all. It doesn't help 99.9% of the world. And so that's what sort of brought us
59:27SPEAKERalong to this. And it's been really exciting to see how it's evolved because like, there's so
59:32SPEAKERmany different ways a restaurant could be booked because if you've got, if you've got high demand,
59:38SPEAKERseven days a week, great, then you can do all sorts of stuff. Most restaurants don't have that.
59:42SPEAKERYeah. Most restaurants that are open have high demand on a Friday and a Saturday. Great. How can
59:47SPEAKERwe push some of that off to non peak times and leverage like that demand to create something
59:52SPEAKERbetter for the guests. And the last thing is, but dealing with no shows appropriately, you can,
59:58SPEAKERyou wind up with this thing where people that want to get in the restaurant can, and the people that
1:00:03SPEAKERare going to screw you over can't, you know, the people that make two or three reservations
1:00:07SPEAKERon a Tuesday and then decide where to go last minute on a Friday. So people really do that.
1:00:12SPEAKERThat's horrible. Yeah. And the thing is that most restaurants treat every one of their guests like
1:00:16SPEAKERthey're a deadbeat when really it's only 15 to 20%. So the stuff that we have allows you
1:00:21SPEAKERbasically to keep your book open for the people that are committed to showing up and then weed
1:00:26SPEAKERout the deadbeats. So it's anti deadbeat software. Fits if people want to find you on Twitter,
1:00:31SPEAKERwhere can they do that? I am on Twitter at The Real Fits.
1:00:34SPEAKERVery good. This is the part of show where I usually talk to Brad about the Patreon.
1:00:39SPEAKERBut I'm just going to talk about a little bit myself. We have at great after extended
1:00:46SPEAKERconversation, we've added one more new channel to the Patreon to the discord, which you can get if
1:00:51SPEAKERyou subscribe at the $2 a month level. And it is called code code code. It's because the NAS
1:00:57SPEAKERchannel was getting clogged up with people talking about Linux programming. So that's there. We've
1:01:04SPEAKERgot the food channel has been popping off. I talked about my maker, the project that I,
1:01:10SPEAKERmy daughter and I made for Mother's Day tomorrow. Yesterday. I mean, we made it over the last week,
1:01:15SPEAKERbut it's been, we've been talking about it in the maker channel. It's an incredible resource.
1:01:19SPEAKERSomebody this morning came to me with a Mr. Question. Do you know what the Mr.
1:01:22SPEAKERis fits? You'll like this. I do not. It's a FPGA based emulation device that emulates
1:01:28SPEAKEReverything probably up to the PlayStation era. Not the PlayStation era is not there yet,
1:01:33SPEAKERbut it it's uses FPGAs to physically mimic, you know, actually mimic the circuitry in
1:01:39SPEAKERSNES is Genesis is a turbo graphics Capcom CFS arcade boards, some old arcade boards,
1:01:48SPEAKERstuff like that. So you can actually play, you know, as, as my favorite example is that the
1:01:53SPEAKERGenesis shipped with different audio chips in different regions and different versions of the
1:01:57SPEAKERhardware. So you can actually get the Genesis audio chip that you grew up with and have,
1:02:03SPEAKERhave the bleeps and bloops sound exactly like you want. I love it. So yeah, he messaged me with a
1:02:09SPEAKERMr. Question. I was like, I don't know the answer to this because I don't have mine yet, but you can
1:02:13SPEAKERhit the discord and someone there will have the answer. He's like, I was helped within like five
1:02:16SPEAKERminutes. So it's a wonderful community. It's like people are incredibly knowledgeable about a bunch
1:02:23SPEAKERof things. You can get access to them if you sign up for the Patrion and our first subscriber
1:02:29SPEAKERexclusive episode went up last week as well, where Brad and I talked about the plan for this month,
1:02:34SPEAKERincluding talking to people like Fitz. So you can find that at patrion.com/techpod.
1:02:41SPEAKERAnd if you don't want to sub to the Patrion, tell like three friends about the podcast. That's all
1:02:47SPEAKERwe ask. We want you to tell your friends about this thing that we make and appreciate everybody
1:02:51SPEAKERwho's listening, everybody who's backing the show. As always, we want to thank our executive
1:02:56SPEAKERproducer level patrons. These folks make the podcast possible every month along with everybody
1:03:02SPEAKERon the Patrion. Andrew Cotton, David Allen and Jacob Chappell. Thank you so much for supporting
1:03:07SPEAKERthe show and we will be back next week with another episode. Thanks for coming by Fitz.
1:03:11SPEAKERThis was awesome. It's my pleasure. Good to see you. Thanks a lot.
1:03:26SPEAKER[Music]