00:00SPEAKERSo Brad, this morning, as I was making coffee before we recorded, I was pouring the coffee
00:04SPEAKERand updating all of my iOS devices simultaneously.
00:07SPEAKERYes, as one does in the year 2023.
00:10SPEAKERJust a perfectly normal thing to do.
00:11SPEAKERAnd I was thinking, you know what would be nice?
00:13SPEAKERIf this update had happened last night automatically without me having to think about it, and also
00:17SPEAKERif I had a coffee refractometer, so I could see how well I was extracting my coffee.
00:22SPEAKERAgain, normal morning thoughts for 7.30 in the morning.
00:25SPEAKERThis is just the way my brain works.
00:27SPEAKERI can't, I feel like I want to dig into both of these, but I don't know where to start,
00:31SPEAKERso I'm going to leave it up to you.
00:32SPEAKERIt's a choose your own adventure, Brad.
00:34SPEAKERThat exploit was a rude awakening at 6.30 this morning when I first read about it.
00:38SPEAKERHoly.
00:39SPEAKERYou mean, hey, let me send you an image and it will totally pwn your phone.
00:43SPEAKERIt's 12 o'clock.
00:44SPEAKERDo you know where your iPhone is?
00:46SPEAKERDid you know?
00:47SPEAKERI learned, I didn't know about lockdown mode on iPhones until then.
00:50SPEAKERNo, I didn't either.
00:51SPEAKERWe should step back and just talk about, there's a very bad iOS exploit in the wild now.
00:56SPEAKERIt's I think it affects everything iOS, Mac OS, all Apple stuff.
01:01SPEAKERSo update your Apple devices today.
01:03SPEAKERThe distinction is, and it's right here in italics.
01:08SPEAKERI've not heard of the citizen lab.
01:09SPEAKERAre you familiar with them?
01:10SPEAKERYeah, they do.
01:11SPEAKERThey do like they're, they're a public service.
01:14SPEAKERThis is how they build themselves.
01:15SPEAKERI gather they're one of these, these kind of watchdog groups that identifies CVEs.
01:19SPEAKERWell, there's, there's money in bug bounties, right?
01:22SPEAKERLike big, big companies pay a lot of money for people who find new exploits and stuff
01:26SPEAKERlike that.
01:27SPEAKERYes.
01:28SPEAKERAnd you can fund a business that way.
01:29SPEAKERYes.
01:30SPEAKERSo even they, in their disclosure have this in all in italics, the exploit is capable
01:33SPEAKERof compromising iPhones, running the latest version of iOS italics without any interaction
01:38SPEAKERfrom the victim.
01:39SPEAKERYeah.
01:40SPEAKERLike that's the part, that's the part that makes it so crazy.
01:42SPEAKERI don't think I've ever seen an exploit where you didn't have to like click on or activate
01:46SPEAKERor run something malicious actively before you were exploited.
01:50SPEAKEROh, Brad, let me tell you about the great worm situation of the early 2000s, pre-Windows
01:56SPEAKERXP SP2.
01:57SPEAKERYou could just plug a computer on the, on the open network and it would be infected
02:01SPEAKERin like eight minutes.
02:02SPEAKERWe've talked about this before.
02:03SPEAKERIf you're going to run on the internet without a firewall, there's no helping you.
02:07SPEAKERIt was a gentler time.
02:08SPEAKEROkay.
02:09SPEAKERYes.
02:10SPEAKERBut yeah, like this is literally, if you get an infected iMessage, your phone is compromised.
02:14SPEAKERThat's it.
02:15SPEAKERLike all they have to do is send you the right payload through an iMessage and you're fucked.
02:19SPEAKERYeah.
02:20SPEAKERIf you listen to this on Monday, it's probably already too late.
02:22SPEAKERYou should probably just light your phone on fire at this point.
02:25SPEAKERHack the planet.
02:26SPEAKERNo, it's like, it's whatever.
02:27SPEAKERI mean, you'd have to be actively targeted.
02:29SPEAKERLike this is probably something that only public figures need to worry about, but really
02:32SPEAKERit's probably something mostly that only like people who run afoul of nation states have
02:37SPEAKERto worry about, but still it's bad.
02:39SPEAKERWell, I mean, you say that, but unlike most other exploits of this variety, this one they
02:44SPEAKERdid find in the wild.
02:45SPEAKERSo it's bad.
02:46SPEAKERLike, like it's out there.
02:47SPEAKERThey discovered this one because of someone's phone that they analyzed that had already
02:51SPEAKERbeen infected by it.
02:53SPEAKEROn the other side of this, the coffee refractor is a thing that you put a little drop of coffee
02:57SPEAKERin.
02:58SPEAKERIt tells you how many, like what the percentage of solids is it's in the device.
03:00SPEAKERDude, that's some Star Trek shit.
03:02SPEAKERI've just been thinking about that as I continue to dabble with cold brew coffee and I have
03:07SPEAKERcut caffeine mostly entirely out of my life.
03:10SPEAKEROh God, I'm sorry.
03:11SPEAKERWell, it's, it's for the best.
03:13SPEAKERIt's for the best.
03:14SPEAKERWell, let's say we have so much caffeinated coffee left that I'm still, I flouted your
03:20SPEAKERadvice and bought a five pound bag of decaf.
03:22SPEAKERYeah.
03:23SPEAKERHow's that going for you?
03:24SPEAKERThat's fine so far.
03:25SPEAKEROkay.
03:26SPEAKERAnyway, I'm cutting, now I'm cutting like decaf and caffeinated.
03:30SPEAKERSo, so it's just another variable in the chain of what am I getting out of this cold brew?
03:34SPEAKERYou know, it's like,
03:35SPEAKERSo it's funny because the last time I looked at this, getting a refractometer was like
03:41SPEAKERa better part of a thousand bucks.
03:43SPEAKERAnd now, now there are some, some that are like 80 bucks on Amazon from China.
03:49SPEAKEROkay.
03:50SPEAKERAnd I'm like, maybe I'm going to get one of these seems like what's, what is there?
03:55SPEAKERIs there a bad idea?
03:56SPEAKERWell, this one's like a visual one that you look through.
03:58SPEAKERWhat are we getting out of this?
03:59SPEAKERLike caffeine, caffeine content, like oil, solids, solids dissolved in the water, like
04:05SPEAKERhow much stuff is in the water.
04:07SPEAKERIt's just, it's just kind of how saturated is this liquid?
04:10SPEAKERIt lets you know how much you've extracted out of the beans and into the, into the, into
04:14SPEAKERthe coffee.
04:15SPEAKERWell, you don't know what compounds or anything like that.
04:17SPEAKERYeah.
04:18SPEAKERI mean, well, if you ground up coffee and put it in the water, you know that it's coffee
04:21SPEAKERcompounds.
04:22SPEAKEROkay.
04:23SPEAKERFine.
04:24SPEAKERIt's not like a spectrometer that'll tell you this is this much carbon and this much
04:29SPEAKERphosphate and this much look, I, I, I can't tell you that it's some sort of magically
04:35SPEAKERcorrosive argon gas, Brad.
04:37SPEAKERI don't understand that.
04:39SPEAKERI'm mainly just curious how much caffeine is ending up in there, but
04:43SPEAKERI'm sure that there's some sort of calculation you can do based on the, but, but that I think
04:46SPEAKERyou have to do know how much caffeine is in the beans and that varies from varietal to
04:50SPEAKERvarietal, I believe.
04:51SPEAKERSo anyway, there you go.
04:53SPEAKERRefractometers update your iOS devices.
04:56SPEAKERSounds like it's time for a podcast.
05:26SPEAKERWelcome to Brad and Will made a tech pod.
05:28SPEAKERI'm Will.
05:29SPEAKERI'm Brad.
05:30SPEAKERI'm Brad.
05:31SPEAKERI'm Will.
05:32SPEAKEROkay.
05:33SPEAKERJust, just, just in case it didn't take the first time.
05:37SPEAKERLook, school has started now.
05:40SPEAKERAnd as a result, like during the summer months, when we podcast, we often podcast early in
05:46SPEAKERthe morning and I get up and I have the house all to myself.
05:49SPEAKERI have an hour and a half usually before we go where I can like finalize show notes and
05:54SPEAKERI can get my stuff together and like have some, have a little bit of quiet Will time
05:59SPEAKERto prepare for a pod.
06:00SPEAKERSorry.
06:01SPEAKERIs that, I started to say, wait, are they like leaving for summer camp at 6 AM?
06:04SPEAKERBut then it occurred to me, Oh, they're probably still asleep.
06:08SPEAKERThe idea of sleeping past like six o'clock is hard for me to fathom at this point.
06:13SPEAKERAll the decent people in my house are asleep until nine during the summer.
06:17SPEAKEROh, that sounds good.
06:18SPEAKERYeah.
06:19SPEAKERIt's nice for them.
06:20SPEAKERSo now during the winter months, when school's on, everything starts at six 30 and it's full
06:28SPEAKERchaos for an hour and a half and they leave at about seven 50.
06:32SPEAKERSo that means that I have 10 minutes instead of the traditional 90 minutes to, to, to do
06:39SPEAKERmy pod peration.
06:40SPEAKERRight.
06:41SPEAKERYou've done some pretty good pod peration here in 10 to 15 minutes.
06:46SPEAKERYeah.
06:47SPEAKERSo anyway, there, there we go.
06:48SPEAKERThis week, we're going to talk about the kind of stuff that people don't talk about about
06:52SPEAKERmaking video games often working on a small studio.
06:55SPEAKERI've got to be engaged with some of this, not all of this.
06:58SPEAKERFor sure.
06:59SPEAKERYou make it sound so salacious.
07:00SPEAKERWhat?
07:01SPEAKERWhat do you mean?
07:02SPEAKERIt's the stuff nobody talks about the filthy underpinnings of the game industry.
07:07SPEAKERTalk about such secrets.
07:08SPEAKERWe don't want to hear about like per force servers and talk about, talk to talk about
07:11SPEAKERsuch scandalous subjects as your build server.
07:14SPEAKERYeah.
07:15SPEAKERYou want to come down to team city with me, Brad team city makes the build server software
07:21SPEAKERthat we use.
07:22SPEAKERI don't wait.
07:23SPEAKERI'm sorry.
07:24SPEAKERWhat is it called?
07:25SPEAKERTeam city.
07:26SPEAKEROh, I thought you were, I thought that was a Microsoft teams joke.
07:27SPEAKERNo, no.
07:28SPEAKERWe use teams too.
07:29SPEAKERWe love teams.
07:30SPEAKERIt's fine.
07:31SPEAKERSeattle studio.
07:32SPEAKERWe've team nothing better than a good Microsoft product.
07:33SPEAKEROh, is that a, Oh, team city is a jet brains thing.
07:36SPEAKERTeam city is a jet brains thing.
07:37SPEAKERThey make a, they it's the, that is literally, okay.
07:40SPEAKERSo anyway, we're going to talk about, we're going to talk about the things that are like
07:43SPEAKERthe backend stuff that makes game studios work and that makes it easy to publish games.
07:48SPEAKERAs I like to think of it, the, like, we're going to talk about how you build a video
07:52SPEAKERgame, not how you like develop and make a video game, but how like you turn all that
07:56SPEAKERwork into a functional running program.
07:59SPEAKERYeah.
08:00SPEAKERSo, so for, so for example, when I make a change to the sound stuff in the game, I submit
08:05SPEAKERit to our, our source control software.
08:07SPEAKERWe use per force.
08:08SPEAKERIt's popular in the games.
08:09SPEAKERWe'll talk about that in a minute, the, the, the, then that goes up onto the source control
08:17SPEAKERserver.
08:18SPEAKERAnd at some point in the future, somebody will say, okay, Hey, well, or realistically
08:21SPEAKERevery night at 3 AM, our build servers look at the, at the source control stuff.
08:26SPEAKERAnd they say, they're like, Hey, isn't anybody submit anything today?
08:29SPEAKERIf they did, then we'll make a new build.
08:30SPEAKERAnd if not, then we'll just use yesterday's build stays.
08:35SPEAKERAnd then those get built, they get uploaded to steam and the next morning, our, our internal
08:39SPEAKERbranches are updated with like the latest, uh, daily play test build, which is, which
08:44SPEAKERis, uh, which is our, like the current it's, it's equivalent of a nightly in like open
08:50SPEAKERsource.
08:51SPEAKERAbsolutely.
08:52SPEAKERI started this, I'm sure every studio is different.
08:55SPEAKERI'm like shocked to hear it's that automated, not the build process.
08:58SPEAKERCause of course you, why not automate nightly builds the part where you deploy it to steam,
09:03SPEAKEReven though I'm sure that's like filtered to a few testers, like not, that's certainly
09:06SPEAKERnot for public consumption.
09:07SPEAKERBut the idea of it, the idea that a build is going out the door, even to testers without
09:11SPEAKERanybody touching it is kind of crazy to me.
09:13SPEAKERWell, so that internal build is just internal.
09:15SPEAKERLike that's not even for, that's not even for external testers.
09:18SPEAKERWe don't generally, okay.
09:19SPEAKERWe have in the past made those builds available to internal testers, but so, so there's more,
09:25SPEAKERokay, let's take it, let's take a step, take a step back.
09:28SPEAKERMost studios probably have at least two and probably three to five private builds, or
09:37SPEAKERplus the release, the release build that everybody, that everybody that's bought the game has
09:41SPEAKERaccess to.
09:42SPEAKERSo there's like, there's going to be a debug slash development build that has all the hooks
09:46SPEAKERthat people can hook their, their engine tools into to see like what's going on inside the
09:51SPEAKERgame.
09:52SPEAKERRight.
09:53SPEAKERAnd that you don't want to put out.
09:54SPEAKERThere's usually like, if it's a multiplayer game, there's like a staging environment or
09:57SPEAKERa development environment one where you can hook into the server and see the same stuff.
10:01SPEAKERSometimes those are the same build in our, in our game they are.
10:04SPEAKERThere's a private testing pre-release, like a release candidate build that's essentially
10:11SPEAKERa ready to ship to machines build.
10:13SPEAKERSo, so there's development builds, there's testing builds, and then there's builds that
10:17SPEAKERyou give to, to customers, to players.
10:19SPEAKERAnd there are different divisions inside of us.
10:22SPEAKERUsually the testing one has some debug hooks and we'll dump extra debug stuff.
10:26SPEAKERIt may have cheats enabled so that the QA folks can, can bypass like, you know, can
10:32SPEAKERspawn specific weapons.
10:33SPEAKERLike if there's a change to a weapon, you want to spawn like 50 of those and try a bunch
10:36SPEAKERof different circumstances, stuff like that.
10:39SPEAKERIn multiplayer game, you obviously don't want to leave that stuff in for the shipping build.
10:42SPEAKERAnd then, and then there's the shipping builds and the shipping candidate builds and all
10:45SPEAKERthat stuff.
10:46SPEAKERYeah.
10:47SPEAKERI've, this is just kind of tangential, like having covered console games for a long time.
10:50SPEAKERI've always assumed that's why, or that's a big part of the, the debug stuff is a big
10:54SPEAKERpart of the reason that console dev kits are so much beefier than the actual console hardware
10:57SPEAKERbecause you need a ton, ton of extra memory and maybe extra compute to handle all the
11:02SPEAKERcrazy extra debug stuff you have to run on top of the actual game.
11:05SPEAKERThat's my understanding of that.
11:07SPEAKERIt's that's a double edged sword because you can use debug kits to test, but you, we like,
11:12SPEAKERwe had a bug with one of the X-Boxes, I think that was only, it didn't happen on the debug
11:20SPEAKERkits because they had that extra memory.
11:22SPEAKERThat's I was going to say, it seems like you'd have to do some very meticulous like resource
11:25SPEAKERand performance profiling to make sure you're not exceeding your eventual hardware target
11:30SPEAKERbudget.
11:31SPEAKERCause we did not realize that that was the problem.
11:33SPEAKERSo we didn't realize it was a problem when we were testing.
11:36SPEAKERCause we assumed that when you flip those to retail mode, then they work like retail
11:40SPEAKERkits, but they don't.
11:41SPEAKERThat sounds like in an X-Box context, cause that's the only one that's got modes of that
11:46SPEAKERnature.
11:47SPEAKERI believe so.
11:48SPEAKERYeah.
11:49SPEAKERThat's we only ship on X-Box.
11:50SPEAKERSo okay.
11:51SPEAKERSo you have all these different things.
11:54SPEAKEREach of those is a pipeline in your, in your build server.
11:57SPEAKERRight.
11:58SPEAKERAnd the build server has basically all of the stuff on it that you need to make the
12:06SPEAKERgame, either the debug versions, the testing versions, the shipping versions, whatever.
12:11SPEAKERAnd on all the platforms that you release.
12:13SPEAKERSo like it'll have all of the libraries and all of the compiler stuff and what it has
12:18SPEAKERon it varies widely based on your game, the platforms you ship on, whether it's a console
12:23SPEAKERgame or PC game, whatever, but the build server knows like, Hey, grab these commits, do, hook
12:36SPEAKERinto the different API versions and then you need to make a version for steam and you make
12:39SPEAKERa version for epic.
12:40SPEAKERYou need to make a version for the X-Box one for X-Box series for windows store, all the
12:45SPEAKERplaces you might ship the game.
12:46SPEAKERIt'll it'll build those.
12:47SPEAKERAnd each of those takes time.
12:48SPEAKERLike it's a, it's a fairly involved process.
12:49SPEAKERI've got, I could probably spend the rest of this podcast talking to you about your
12:53SPEAKERbuild server and the concept in general, from, from everything from like the general specs
12:58SPEAKERand how they're built at a low level to like how you manage different build environments
13:02SPEAKERbased on the pile targets and platforms and stuff like that.
13:05SPEAKERLike are most, I'm sure it varies, like, you know, I'm trying to think like, what's the
13:11SPEAKERbiggest Ubisoft team out there?
13:13SPEAKERI mean, your studio is quite small.
13:14SPEAKERA lot of the indie developers are just a single digit number of people, but then there are
13:17SPEAKERlike a hundred strong studios out there, like 500 people or something these days or bungee
13:23SPEAKERis like gigantic.
13:24SPEAKERRight?
13:25SPEAKERSo like, I'm sure, I'm sure what constitutes like a build server or build farm is very
13:30SPEAKERdifferent as, as size scales.
13:33SPEAKERYeah.
13:34SPEAKERWe have a baby ass baby build farm.
13:35SPEAKERIt's only like three machines.
13:36SPEAKEROh, even though I'm actually surprised, even at your size, that it's multiple machines
13:40SPEAKERI was going to ask, like, is it, is it all just kind of, I assume off the shelf commodity
13:44SPEAKERhardware these days?
13:45SPEAKERI mean, CPU's are very fast in general, right?
13:47SPEAKERYeah.
13:48SPEAKERI think I feel like more cores and more memory, it's a real right turn intensive thing.
13:54SPEAKERMy guess is like, we've been looking at upgrading our build servers at some point in the future.
13:59SPEAKERJust because we're on the time we built these machines or bought these machines, I don't
14:05SPEAKERthink that there were a lot of really large NVME SSDs available at a scale that we could
14:11SPEAKERokay.
14:13SPEAKERSo you have to have a lot like the, the there's a lot of space, it requires a lot of space.
14:17SPEAKERSo if you like, our game is like 16 gigs on steam, which decompresses down to like 30 gigs
14:22SPEAKERwhen you, when you, when you unpack it.
14:26SPEAKERAnd to do the builds and have, you have to have all the source code files.
14:29SPEAKERYou have to have the compiled binary assets that make up the game and the unreal project,
14:34SPEAKERall that stuff.
14:35SPEAKERIt's, it's big, it's like 500 gigs of space for the project for the current version of
14:41SPEAKERthe project.
14:42SPEAKERThat's pretty big.
14:43SPEAKERYeah.
14:44SPEAKERI mean, you know, like a relatively small scale game.
14:47SPEAKERI'm not saying like your game is small or whatever, but like, but like it's like a million
14:50SPEAKERdollars.
14:51SPEAKERIt's not a big game.
14:52SPEAKERRight.
14:53SPEAKERLike it's a focused game.
14:54SPEAKERIt's a multiplayer only game more or less, you know?
14:55SPEAKERYes.
14:56SPEAKERYeah.
14:57SPEAKERAnd that, and that's, that's like you said, that's half a terabyte for a, an indie multiplayer
15:00SPEAKERgame.
15:01SPEAKERLike imagine, imagine what the total kind of asset or package size is for like your
15:07SPEAKERaverage sprawling console RPG.
15:09SPEAKERYeah, to give you some context, I think Chet likes to say that Far Cry 6 had more studios
15:15SPEAKERthat worked on it than we had people that worked on the end of Cruces.
15:19SPEAKERSure.
15:20SPEAKERAnd, and some of those studios, like the Ubisoft Shanghai is like a thousand people.
15:24SPEAKERIt's they're enormous.
15:25SPEAKERRight.
15:26SPEAKERYeah.
15:27SPEAKERSo, so they have a whole, like where we had a person who did this for a while and then
15:31SPEAKERit shifted out to like, to be part of somebody else's job or a couple other people's jobs.
15:36SPEAKERAnd this, this at someplace like Ubisoft, I assume that there's entire teams that do
15:40SPEAKERnothing but this kind of DevOps work, right where they're, where they're, you know, everybody
15:45SPEAKERhas their own per force and then stuff filters up and their Knightleys are real weird.
15:49SPEAKERI'm sure.
15:50SPEAKERI don't even like, I can't even, the amount of art that goes into something like an Assassin's
15:53SPEAKERCreed.
15:54SPEAKERI don't, I can't imagine that you have a daily build cycle.
15:58SPEAKERAnd then, and then you hear about things like Destiny 1 where Bungie didn't, had problems
16:02SPEAKERwith their backend and like they had their, just to be clear, the build server doesn't
16:08SPEAKERdo the computationally expensive stuff like build light maps and stuff like that.
16:11SPEAKERThat happens either an artist does that, or we have a build a light map machine that,
16:15SPEAKERthat does that.
16:16SPEAKERInteresting.
16:17SPEAKERSo what are the, what are, when you say an artist does that, do they, would they have
16:20SPEAKERthat hard, surely they wouldn't have that hardware at their desk, right?
16:23SPEAKERI assume that's like heavier iron than like would be in an end user context.
16:27SPEAKEROkay, so for our game, again, this is a place where things differ based on the scale of
16:33SPEAKERyour game, right?
16:34SPEAKERYeah.
16:35SPEAKEROn our game, I can run the light maps in like 40 minutes on my 40/90.
16:39SPEAKEROh, okay.
16:40SPEAKERCause there, there are GPU, there's a GPU accelerated light map maker now.
16:43SPEAKEROh, of course, of course.
16:44SPEAKERSee, my, my brain is very stuck in the nineties back when like every, every like Quake licensee
16:48SPEAKERhad a, had a, like a light map.
16:50SPEAKERYou don't have an SGI machine in any place running this stuff.
16:52SPEAKERLike running radiosity calculations.
16:54SPEAKERYeah, no.
16:55SPEAKERI mean, like the ray tracing stuff has accelerated that pretty dramatically.
16:59SPEAKERIt's not, it's not real time, but it's like, it's gone from a, Hey, you start this before
17:04SPEAKERyou leave work at the end of the day and then you come back in the morning and it's maybe
17:06SPEAKERdone to Hey, you can hit the button and yeah, like on a 30, 70, it'll be done in like an
17:12SPEAKERhour.
17:13SPEAKEROkay.
17:14SPEAKERYou can go eat lunch and come back.
17:15SPEAKERMaybe take a nap.
17:16SPEAKERYeah, I guess, I guess this is a good time to point out like the stuff we're talking
17:19SPEAKERabout in terms of how fast you can, you know, crunch calculations and produce a build like
17:23SPEAKERthat has very real implications for workflow, right?
17:26SPEAKERLike how fast you can check changes in and get a build out and test them like just dictates
17:31SPEAKERhow fast you can move on to the next thing.
17:33SPEAKERRight?
17:34SPEAKERYeah.
17:35SPEAKERAnd, and so we have, so, so the two things to take away from this are even a simple game
17:38SPEAKERthat's only shipping on two platforms, we ship on steam and Xbox right now, when we
17:43SPEAKERrelease something, it makes five builds, right?
17:48SPEAKERSo each of those builds takes an hour to two hours.
17:51SPEAKERA lot of that is just disc churn.
17:53SPEAKERThere's not a lot of, there's not a lot of complicated stuff.
17:55SPEAKERIt's not like we're baking light maps or something on all of those builds.
17:58SPEAKERAnd like that takes, that takes a long time.
18:01SPEAKERWe also have a, Hey, here's, we want to do something quick for testing that just jams
18:04SPEAKERit up on, on a special steam branch.
18:06SPEAKERAnd that's like an hour, hour and a half, but it's, it still takes time.
18:09SPEAKERIt's a, it's upload bandwidth and it's it's, it's the compiling time and all that kind
18:15SPEAKERof stuff.
18:16SPEAKERYeah, I assume it's, it's probably safe to say that the common commercial engines like
18:19SPEAKERunreal and unity have, I assume, have focused on iteration time as a, as a single point.
18:24SPEAKERI don't think they give a flying fuck about iteration time.
18:28SPEAKERNevermind then.
18:29SPEAKERWell, you, you mentioned the bungee example, which like, from my understanding, that was
18:32SPEAKERjust like a legacy tech issue.
18:34SPEAKERYou know, they had just been working on the same core architecture in their engine for
18:38SPEAKERso long that there were just a lot of legacy things that made it cause that, that was very
18:42SPEAKERwidely reported about the first game that like, you know, like kind of like you said,
18:46SPEAKERit took 12 hours to bake them to see a map change or something nuts, right?
18:50SPEAKERLike replace one pickup in the map.
18:52SPEAKERThis may not be a hundred percent accurate, but it will make a very minor change to the
18:55SPEAKERmap.
18:56SPEAKERAnd then you'd have to wait like all night and come back the next morning before you could
18:58SPEAKERtest the map again.
18:59SPEAKERYeah.
19:00SPEAKERI think so.
19:01SPEAKERI don't know about bungees example in particular, but, but the other thing is, so I say that
19:07SPEAKERI don't think the, the engine makers give a fuck about that, but like there is unreal
19:12SPEAKERhas play an editor, right?
19:13SPEAKERSo you can hit a button in the game and it'll load up the game with whatever kind of starting
19:17SPEAKERstate you say.
19:18SPEAKERSo you can say, okay, I want to play in versus mode.
19:20SPEAKERI want to play in survival mode.
19:21SPEAKERI want to do this, this, this, and then you can play through the game there.
19:24SPEAKERIt's not exactly one to one.
19:26SPEAKERLike there's some places where it's a little bit weird and you can get, you can put yourself
19:31SPEAKERin trouble there.
19:32SPEAKERBut if you're making like, if you're making like, for example, when I'm making audio changes,
19:36SPEAKERI can make my change, hit the play an editor button and, and get an idea, a pretty good
19:41SPEAKERidea of what the end result is going to be.
19:43SPEAKERYou mean when you're getting your wise on what I'm getting my wise on if other stuff
19:49SPEAKERis less like light map changes, you still have to make the light maps.
19:51SPEAKERIf you want to see what, what happens and there's always a, did this, did this run right?
19:57SPEAKEROr is there some weird math glitch to the cosmic Ray hit the GPU at the wrong moment
20:00SPEAKERand jack up the whole thing.
20:02SPEAKERBut, but that usually results in like weird artifacting, not, not a broken bill, not a
20:08SPEAKERbroken build.
20:09SPEAKERSo the, the, the thing about the thing about like light maps are a weird one cause it's
20:13SPEAKERalways a balance between, Hey, how long do I want to bake the light map and Hey, what
20:18SPEAKERlevel of janky artifacting my okay with and a shadow in a corner of the map that nobody
20:22SPEAKERever sees.
20:24SPEAKERAnd it turns out the answer is you're probably okay with a little bit of artifacting in a
20:26SPEAKERweird corner that nobody ever sees.
20:29SPEAKERAnd if that's a continual problem that you can put something in that corner that will
20:32SPEAKERkind of fix that problem and tell it to spend a little more time there.
20:34SPEAKERI don't know if this is getting to like 3d graphics engineering for you or if you can
20:38SPEAKERspeak to this, but like, can you elaborate on like what constitutes a light map in 2023?
20:43SPEAKERLike I hear a lot about light probes.
20:45SPEAKERLike are those synonymous at this point?
20:46SPEAKERLike I don't know what the and, and also, are we talking about fully static lighting
20:50SPEAKERat this point?
20:51SPEAKERLike are, are those types of quote unquote light maps, like things that you place and
20:55SPEAKERthen don't move?
20:56SPEAKERLike, is that still, how does that, how does that work these days?
20:58SPEAKERYeah.
20:59SPEAKERSo depending on the platforms you're shipping on, for example, like if you're making an
21:04SPEAKEROculus quest game, you can have a relatively meager number of dynamic lights, dynamic lights
21:09SPEAKERor lights that can move that can be on or, or, well, I mean, light map lights can be
21:13SPEAKERon or off too, but the option is intensity, you move around and stuff like that.
21:17SPEAKERYeah.
21:18SPEAKERThey're there.
21:19SPEAKERYou can, you can programmatically during the gameplay and they are on traditional hardware,
21:25SPEAKERvery expensive, like they call it, especially on mobile stuff.
21:28SPEAKERThey're really expensive to run, which is why I like lighting has been like the, the
21:32SPEAKERit's hard to say the Holy grail, but that's more like the white whale, I feel, I feel
21:36SPEAKERlike of, of 3d graphics since the beginning of time, right?
21:39SPEAKERLike that's always the thing that takes the most work and that people throw the most different
21:42SPEAKERlike techniques at the wall to try to solve.
21:45SPEAKERWell, and it's, and it's a thing that our brains, our brains know what lights do, what
21:49SPEAKERlight does in the real world, just instinctively.
21:51SPEAKERWe understand it.
21:52SPEAKERSo it's a place where when the light doesn't look right, you know, it nearly, this doesn't
21:55SPEAKERlook good or, or, or it's, alternately you can go with a really stylized approach and
21:59SPEAKERthen you, it doesn't matter if it looks good.
22:01SPEAKERDo you just have to be internally consistent?
22:04SPEAKERThe, the, the, the, there's a bunch of different ways to cheat having a bunch of dynamic lights
22:11SPEAKERin the game.
22:12SPEAKERThat's, that's the kind of TLDR and the different engines use a bunch of different ways inside
22:17SPEAKERan engine.
22:18SPEAKERYou can choose to use a bazillion different ways.
22:21SPEAKERThe exciting, this is one of the reasons people are excited about UE5 is that because of the
22:26SPEAKERthings that they're doing, you can have dynamic, like dynamic lights because of the way they're
22:31SPEAKERusing the ray tracing hardware and, and, and some stuff that they've done on the back end
22:35SPEAKERof the engine, dynamic lights as a solution for your game are much more viable than they
22:41SPEAKERhave been in the past.
22:42SPEAKERSo you can have a game that is entirely dynamic lights, like the sky, the sun is a dynamic
22:47SPEAKERlight.
22:48SPEAKERAnd, and like, so for example, in PUBG, the sun moves across the sky, right?
22:52SPEAKERBut they move the sun across the sky by generating light maps at all the different positions
22:56SPEAKERthe sun can be, and then blending between them as the time changes in the game.
23:00SPEAKERYeah.
23:01SPEAKERIs that why in some games you'll, if you look at kind of the shadows on the ground, you'll
23:03SPEAKERsee them moving in very rigid increments.
23:06SPEAKERYeah.
23:07SPEAKERYeah.
23:08SPEAKERAnd, and so they don't blend between them when they're moving in rigid increments.
23:11SPEAKERBut at the end of the day, all the light map is, is a texture that says how bright a part
23:14SPEAKERof the map is.
23:15SPEAKERRight.
23:16SPEAKERRight.
23:17SPEAKERAnd then the light probes and stuff like that are just, just further refined, give you more
23:21SPEAKERcontrol over specific subsets of the game and stuff like that, or dynamic control over
23:25SPEAKERa different area, stuff like that.
23:26SPEAKERRight.
23:27SPEAKERSo like fully, fully not to turn this into lighting chat, but like fully generalized
23:31SPEAKERdynamic lighting has all that's, that's the Holy grail that I was talking about.
23:34SPEAKERLike ever since like Doom 3, it's always been like different techniques of chasing.
23:38SPEAKERLike we don't want to have to bake any, any rigid, unchangeable lights into this game.
23:42SPEAKERIf we can make everything work in the same way dynamically, then or even in different
23:46SPEAKERways dynamically.
23:47SPEAKERAnd typically game lighting doesn't work the same way that light works in the real life
23:51SPEAKERin real, in real life, you know, a light source exudes photons, the photons bounce off of
23:55SPEAKERthings.
23:56SPEAKERAnd then the photons that smash into your eyeballs are, are represent colors and, and,
24:01SPEAKERand information like their wavelength has changed based on what they've been absorbed
24:05SPEAKERand re-emitted into in a video game.
24:09SPEAKERTypically they, you blast rays out from the eyeballs and you see what you can see, and
24:15SPEAKERthen you kind of back it, you reverse engineer into what color those should be.
24:18SPEAKERYes, I was going to bring that up as, as, as I've read a little bit about the current
24:21SPEAKERray tracing techniques.
24:22SPEAKERI was surprised to find out that they do it in reverse of the way that, that it works
24:25SPEAKERin the real world.
24:26SPEAKERTurns out it's a lot cheaper cause you have to do a bunch of pixels, a bunch of rays that
24:29SPEAKERnobody ever sees.
24:30SPEAKERYes.
24:31SPEAKERSo like that stuff is super fascinating and I could barely speak to it.
24:33SPEAKERAlthough I was actually just looking at the Reeser white paper yesterday, are you familiar
24:36SPEAKERwith that?
24:37SPEAKERI haven't read it yet.
24:39SPEAKERIt's what Cyberpunk uses for their path tracing mode.
24:41SPEAKERIt's a, it's a relative, I think it's white paper was published in like 2019, 2020.
24:46SPEAKERIt's just a new, it's a new algorithm for ray casting that is like wildly more efficient.
24:51SPEAKERIt's probably, it's probably like less accurate if you're running an offline render farm and
24:55SPEAKERyou want pure, pure image quality, but it's not going to use that, but it's great for
24:59SPEAKERvideo games.
25:00SPEAKERIt sounds like R E S T I R like it's the math is super over my head.
25:04SPEAKERI didn't even bother trying to understand it.
25:06SPEAKERBut if you, if you Google that, you'll get straight to the Nvidia like research page
25:09SPEAKERfor it.
25:10SPEAKERAnd you can see a picture of like, here's the, here's the, here's the, here's the kind
25:14SPEAKERof like, it's not a point cloud, but what is the, you know, the, the undi noise, like
25:18SPEAKERjust the, the speckled image that you see of like, here's, here's where every ray hit
25:21SPEAKERa surface with the traditional algorithm and then right next to what research does.
25:25SPEAKERAnd it's like orders of magnitude, more samples, more dense.
25:28SPEAKERYeah.
25:29SPEAKERSo it's cool.
25:30SPEAKERIt's really cool seeing all this research stuff going on.
25:31SPEAKERThat's like massively accelerating how that stuff works.
25:33SPEAKERWe're let's, we should probably get back to building games.
25:36SPEAKERI'm sorry to turn this into 3D graphics time.
25:38SPEAKERWell, so the, I mean, that's why we don't focus too much on stuff like light maps.
25:42SPEAKERLike we, we set up a big Threadripper machine in the office and let that be the light maps
25:46SPEAKERserver.
25:47SPEAKERAnd then, and then the artists can say, okay, I need to bake level, you know, episode three
25:51SPEAKERlevel two.
25:53SPEAKERAnd that sends a command to that server and it runs the light maps and like an hour later
25:57SPEAKERit uploads it, it makes it commit to the source control.
26:01SPEAKERThat's like, Hey, new light maps for episode three level two.
26:04SPEAKERThat's okay.
26:05SPEAKERThat's interesting.
26:06SPEAKERSo that's, that's what I was wondering about when I said like, are, are these types of
26:09SPEAKERservers just commodity hardware these days?
26:11SPEAKERLike Threadripper is not exactly end user hardware, but it's also not like, it's also
26:15SPEAKERnot a hundred thousand dollar supercomputer type stuff.
26:19SPEAKERLike you, like you said, that's like, what is like a decent Threadripper build is like
26:23SPEAKERfive grand or something.
26:24SPEAKERI don't think we've been spent that much, but, but yeah, basically the build servers
26:31SPEAKERwill benefit from more cores, but you don't need like a workstation, like a multi-core
26:36SPEAKERworkstation machine.
26:37SPEAKERYou just need something with a shitload, like a Threadripper, a big Intel machine will totally
26:42SPEAKERdo it.
26:43SPEAKERAnd you're probably better off given that while one build is building, like we would
26:48SPEAKERbe better off having four small machines than two big machines.
26:51SPEAKERI think we have two big machines now.
26:53SPEAKERAnd because we have five builds to make when we hit the button then, and they can't run
26:59SPEAKERin parallel on the same machine, we would be better off having five NUCs with a lot
27:04SPEAKERof storage attached to them than two big 64 core machines, which is what we have now.
27:09SPEAKERSo why are those two machines not in some kind of cluster situation where they're both
27:12SPEAKERworking on the same, or is each one doing a discrete build?
27:14SPEAKEREach one is doing its own build because of the way the build process, you can't come.
27:18SPEAKERYeah.
27:19SPEAKERLike I guess theoretically we could put VMs on them, but also you pay for TeamCity based
27:23SPEAKERon the number of machines that you have.
27:25SPEAKERSo like it's, as with everything, you can put a bunch of numbers in a spreadsheet and
27:31SPEAKERthen you get a curve where you're like, okay, here's the sweet spot.
27:33SPEAKERWe're willing to pay this much money a year to get a build in an hour versus a third of
27:38SPEAKERthat to get a build in two hours.
27:40SPEAKERRight.
27:41SPEAKERAnd it turns out the two hour number is usually the right number.
27:44SPEAKERYes.
27:45SPEAKEROkay.
27:46SPEAKERSo then once the builds are made on the build server, theoretically, if you've done, if
27:51SPEAKERyou set yourself up right, it should just go to the platforms.
27:54SPEAKERLike it should just be uploaded into the back ends of the platforms.
27:58SPEAKERDisclaimer.
28:00SPEAKERSo some are more reliable than others.
28:03SPEAKERThere's a button in TeamCity that you can press that's like, "Hey man, re-run the upload
28:08SPEAKERprocess."
28:09SPEAKERSo TeamCity sets this up as like, I've used Jenkins before in the past, which is a similar
28:14SPEAKERpiece of software that's open source, but TeamCity sets this up as a series of tasks.
28:18SPEAKERSo it's like build for Steam, build for Xbox, build for Xbox One, build for Windows.
28:25SPEAKERAnd then the next step in that chain is deploy to Microsoft, deploy to Steam.
28:32SPEAKERSo if the build completes correctly and it doesn't deploy, you just hit the deploy button
28:37SPEAKERand it uploads it again.
28:38SPEAKERIt's fine.
28:39SPEAKEROkay.
28:40SPEAKERI didn't realize TeamCity also handled the uploads and hooked into those back ends directly.
28:43SPEAKERIs that built in or is that something that you have to script, like here's the upload
28:48SPEAKERtarget for Steam, here's the API keys, do you have to build that part yourself?
28:54SPEAKERSo yeah, I didn't set this up on our end.
28:56SPEAKERI did it for our Jenkins server as a FU, but basically, Steam just has a bunch of command
29:04SPEAKERline utilities that you can script into.
29:06SPEAKERSo it's a lot of PowerShell scripting.
29:09SPEAKERFrom TeamCity's perspective, this is all just PowerShell scripting and it has hooks into
29:14SPEAKERsome of the compilers and stuff like that, so you can see how it's going.
29:17SPEAKERBut mostly it's just like, "Hey, I'm going to run some scripts in a sequence of orders
29:21SPEAKERand then I'm also going to archive a bunch of files for you.
29:23SPEAKERI know how to handle the file archiving bits."
29:25SPEAKERBut also that's probably PowerShell scripting too.
29:27SPEAKERI haven't actually looked at those to see how they work, but basically when you're setting
29:32SPEAKERup a small game studio, it's good to have one person who's really excited about doing
29:35SPEAKERPowerShell scripting.
29:36SPEAKERWell, because all this stuff has to run on Windows, right?
29:41SPEAKERYou run this on Windows because that's where your environment is and if you're shipping
29:45SPEAKERto Xbox, I guess they'd probably let you build on Linux now.
29:49SPEAKERBut realistically, you're going to run on Windows for most of this if you're a small
29:52SPEAKERstudio.
29:53SPEAKERThat makes sense.
29:54SPEAKERIf anybody, any game devs listening to this, I want to write in, I wonder if the WSL2 changes
30:00SPEAKERthat at all.
30:01SPEAKERLike if Linux build environment on Windows is more viable now that that's a factor.
30:05SPEAKERSo we set that stuff up.
30:07SPEAKERWell, you could definitely build a Linux build on Windows.
30:09SPEAKEROh, a Linux build, yes.
30:10SPEAKERI'm talking about-
30:11SPEAKERWe could build a Linux build no problem if we wanted.
30:13SPEAKERSupporting a Linux build is a whole other issue.
30:14SPEAKERNo, no.
30:15SPEAKERI meant like managing your whole build and environment.
30:17SPEAKERThat probably doesn't make sense.
30:18SPEAKERLike you probably, you want to build for the OS on the OS, right?
30:22SPEAKERYeah, I don't know what the Unreal Linux situation is.
30:25SPEAKERI know that they have a Linux pipeline, but I bet it's from Windows.
30:28SPEAKERI don't know.
30:29SPEAKERI haven't looked at that.
30:31SPEAKERSupporting Linux is fraught with peril.
30:36SPEAKERI love Linux as much as the next person, I think everybody knows that, but the beautiful
30:40SPEAKERthing this team has done is let us support Linux without having to actually test more
30:43SPEAKERstuff.
30:44SPEAKERBecause the other thing is, each of these builds adds testing overhead, right?
30:48SPEAKEREvery time you make a new version of the game that you have to release to people, you have
30:51SPEAKERto test stuff.
30:55SPEAKERWith a small team testing five versions of the game, even just to test the stuff that
31:01SPEAKERwe change from build to build -- we've gone a little bit out of order here -- even just
31:09SPEAKERtesting the stuff you change is a huge task.
31:12SPEAKERIf you have a patch that has 20 changes and you have five versions, that means you have
31:16SPEAKERto test those 20 changes on five different builds of the game and see what works and
31:22SPEAKERwhat doesn't.
31:23SPEAKERIf there's any weird regressions, it's complicated.
31:25SPEAKERDo you as a studio have a standard suite of tests outside of, like you said, if you change
31:31SPEAKERfive things, of course you have to test those five things.
31:34SPEAKERIs there also just a short checklist of these are the basics we have to reconfirm every
31:39SPEAKERsingle time there's a new build that these work?
31:41SPEAKERThat would be a good idea.
31:43SPEAKERI mean, obviously, that's constrained by resources and human power.
31:48SPEAKERYeah.
31:49SPEAKERI mean, we do have a list of things that we test, but also the game's early access.
31:54SPEAKERIf we were releasing -- we're a relatively small game, we're in early access, and also
32:01SPEAKERwe can move quickly, so we can release a follow-up patch within a day, basically.
32:06SPEAKERWe typically ask our wonderful community of players to help with that kind of stuff and
32:11SPEAKERlet us know if we've done something really bad.
32:15SPEAKERBecause the challenge of, hey, if we change something about the way matchmaking works,
32:21SPEAKERwe can test matchmaking a little bit internally, but not really.
32:24SPEAKERWe can't test across regions and stuff like that, because we only have people in the United
32:27SPEAKERStates.
32:30SPEAKERIt's difficult for us to find out if we've actually messed something up with matchmaking
32:36SPEAKERuntil it's actually on the live game, is the unfortunate thing.
32:39SPEAKERMy assumption is, people might be listening and thinking, well, if you only change five
32:44SPEAKERthings, why do you have to test anything other than five things?
32:46SPEAKERBut my assumption is that it's very hard to predict second- and third-order consequences
32:52SPEAKERof seemingly -- I mean, it's straight up a butterfly flaps its wings on map two and type
32:59SPEAKERstuff.
33:00SPEAKERI don't know.
33:01SPEAKERSo yeah, a lot of stuff's interconnected.
33:02SPEAKERA lot of stuff's -- for example, the attenuation profiles for environment sounds in the game
33:07SPEAKERare often shared across levels, which is a choice somebody made a long time ago that
33:13SPEAKERseems crazy now.
33:14SPEAKER[laughs]
33:15SPEAKERIt's on you to this day.
33:17SPEAKERYeah.
33:18SPEAKERIt means that if I change how far away you can hear a fan on level one of episode one,
33:23SPEAKERthen all of a sudden, a big, huge blower on episode five or four is going to sound like
33:27SPEAKERa jet engine, and I won't know it until I play episode four three days later.
33:34SPEAKERThe better example of that probably is something like the player's collision capsule had to
33:40SPEAKERchange a few months ago.
33:47SPEAKERWhoever wrote the AI code for the bots, when a player gets gooed and our game gets incapacitated,
33:52SPEAKERyou shoot them to knock the goo off of them and free them.
33:55SPEAKERThe person who wrote the bots was like, "Oh, the player collision capsule's a little bit
33:58SPEAKERoff, so I'll just aim at the 20 units right of the collision capsule in Cartesian space
34:06SPEAKERto free them when they get gooed, and it'll just work itself out."
34:09SPEAKERThat was great, until we fixed the collision capsule, and now all the bots are shooting
34:13SPEAKER20 units to the right of where the player actually is, so the bots just don't ever free
34:20SPEAKERplayers anymore.
34:23SPEAKERWe find that out because we play every day.
34:25SPEAKERThat's a bad example of that.
34:30SPEAKERIf we change something about the way you invite friends to the game on Steam, and that had
34:34SPEAKERknock-on effects on Xbox, we might not think to check that, because who knows?
34:40SPEAKERThat's what build servers do.
34:42SPEAKERThey make the game, they turn the code into game, and then they push the game to the server,
34:46SPEAKERif you've said that, to the platforms.
34:48SPEAKERFrom there, you can go to a UI thing and push the build live, where you open up a web interface
34:53SPEAKERsomeplace and you're like, "Okay, I want this upload to be the live game on servers right
34:57SPEAKERnow."
34:58SPEAKERI want to ask more about the build process.
35:00SPEAKERI don't want to get too much into the weeds, and I also don't know how much of it you set
35:02SPEAKERup yourself.
35:03SPEAKERI didn't set up any of the build process.
35:06SPEAKERI didn't know how to go in and press the button to make a build.
35:09SPEAKERMaybe that answers the question, because I was going to ask if there is just a button
35:12SPEAKERto press, at least with Unreal these days, like you're not necessarily running.
35:16SPEAKERI'm curious about which compiler is being used, and what's the build environment and
35:20SPEAKERthe tool chain look alike and stuff, but I'm guessing that at least with a commercial engine,
35:24SPEAKERthat's probably all pretty much dictated and set up by the engine.
35:28SPEAKERYeah, so that's all Unreal stuff.
35:30SPEAKERWe use some of the stuff we use, like the different APIs that we use from Microsoft
35:36SPEAKERfor the Xbox stuff, and all that have different requirements for Visual C and XNA, or X and
35:41SPEAKERwhatever the current Xbox backend stuff is called, whatever their API package is called.
35:51SPEAKERThat's all set up in the scripts for the compiling scripts with the building scripts that somebody
35:55SPEAKERset up for TeamCity two years ago, at this point probably three years ago.
35:58SPEAKERInteresting.
35:59SPEAKERThere are starting to be some open source game engines out there now, and I'm sure those
36:05SPEAKERtypes of questions are maybe more relevant there.
36:06SPEAKERI'm sure they automate some of that stuff, too.
36:09SPEAKERJust to be clear, this isn't Unreal automation.
36:11SPEAKERThis is stuff that we built internally.
36:14SPEAKERThis is why TeamCity is a product we pay for, that takes the stuff that I could build -- just
36:22SPEAKERto be clear, I could load up my computer in Unreal right now and hit a button that says,
36:26SPEAKER"Hey, make an Xbox version of this game."
36:28SPEAKERI guess I don't have Visual C installed, but if I did, Visual Studio installed it.
36:31SPEAKERIf I did, then I could make a version of the game that I could upload to the Microsoft
36:35SPEAKERbackend or to Steam, and people would be able to play.
36:38SPEAKERBut, in order to do that, it's going to tie up my computer for an hour and a half, and
36:43SPEAKERthen I'm going to have to manually upload it.
36:45SPEAKERActually, Steam doesn't let you upload stuff bigger than a certain size, so I'd have to
36:48SPEAKERget the Steam command line tool to upload it and all the credentials and all that stuff.
36:53SPEAKERThis is a way to manage -- what a build server does is give you a way to make those builds
36:59SPEAKERwithout having to give everybody in the organization the Steam keys and the Xbox keys and all the
37:04SPEAKERcrypto stuff that you need to sign the Xbox games and all of that stuff.
37:08SPEAKERSo, it lets you keep your information secure that's important to the thing, but also more
37:15SPEAKERwidely distribute the ability to actually make the game than just one person.
37:20SPEAKERYeah, that's fascinating to think about Steam as a development tool in that context and
37:23SPEAKERnot as an end-user distribution platform.
37:25SPEAKERLike, four of four, like you said, build management and dispersal to people who don't necessarily
37:33SPEAKERneed to worry about that sort of thing on the team.
37:36SPEAKERAnd it's also worth mentioning that Steam is really neat, because they give you the
37:39SPEAKERability to put infinite -- A, they let developers hand out keys without having to pay for them
37:45SPEAKERor something stupid like that.
37:47SPEAKERB, they also let you do branches, so you can do betas and stuff like that that are either
37:52SPEAKERpublic or private.
37:53SPEAKERPeople use them for all sorts of interesting stuff.
37:55SPEAKERThe other day, I was poking around dead cells, and they have in their beta branches public
38:02SPEAKERversions of every past version of dead cells, I think, is in there.
38:06SPEAKERSo, if you want to go back and play launch dead cells, you can.
38:09SPEAKERThat's cool.
38:10SPEAKERYeah.
38:11SPEAKERLike, if you have a server backend infrastructure, obviously that's not going to work.
38:15SPEAKERBut you can do whatever you want.
38:17SPEAKERWhen we wanted to test versus publicly, like semi-publicly, we could just generate a branch
38:25SPEAKERwith a passcode on it and share that with the community, then people could load it up
38:29SPEAKERand play the game.
38:30SPEAKERYeah.
38:31SPEAKERLike, all the Steam developer docs largely are public.
38:35SPEAKERYou can go read about a lot of that stuff.
38:36SPEAKERLike, I've looked into some of how the backend branching stuff works, and they give you an
38:40SPEAKERamazing amount of granular control over who has access to a given branch and stuff like
38:45SPEAKERthat.
38:46SPEAKERYeah.
38:47SPEAKERAnd it's up to us, then, to make sure that we don't share, for example, the branch with
38:52SPEAKERthe developer credentials or whatever that kind of stuff is in there.
38:57SPEAKERYeah.
38:58SPEAKERWell, real quick before we move on, just to touch on a couple of the open source game
39:03SPEAKERengines that I mentioned.
39:04SPEAKERGodot is the big one that I hear about for the most part.
39:07SPEAKERG-O-D-O-T.
39:08SPEAKERYeah.
39:09SPEAKERWe're waiting for it still.
39:10SPEAKERYes.
39:11SPEAKERI bet they get that a lot.
39:13SPEAKERLook, they named the game engine.
39:15SPEAKERIt's their own fault.
39:16SPEAKERI guess you may have a point there.
39:19SPEAKERIt's not like me, where I was named Will Smith, and then 20 years later, somebody got famous.
39:23SPEAKERThey knew what Godot was when they named it that.
39:25SPEAKERYes.
39:26SPEAKERI mean, I don't know if this is true, or I can't say for sure this is true, but I kind
39:30SPEAKERof see Godot spoken about a lot in the context of Unity.
39:33SPEAKERI see a lot of, like every time Unity does something unpopular, I'll see some game developers
39:37SPEAKERon Twitter say, "Well, maybe it's time to finally start learning Godot."
39:40SPEAKERSo I wonder if it has similar application or similar amount of scope.
39:44SPEAKERI would say disaffection with Unity seems to be pretty high in my communities right
39:48SPEAKERnow.
39:51SPEAKERI haven't looked at Godot yet.
39:52SPEAKERI don't know where Godot's at.
39:53SPEAKERThere's also Bevy.
39:54SPEAKERI don't know about that one.
39:55SPEAKERI don't know about that one.
39:56SPEAKERI think it is pretty, I shouldn't say limited, I get the sense it's fairly early.
40:01SPEAKERIt's written in Rust.
40:02SPEAKEROkay.
40:03SPEAKERAnd there's some people on our Discord have spoken highly of it or at least have, I think,
40:08SPEAKERenjoyed tinkering with it.
40:11SPEAKERIt's hard to not do a commercial product if you're spending, like if you think about your
40:18SPEAKERinvestment in a commercial game engine versus the payroll of your small studio, even like
40:27SPEAKERwith five or six people, it's scary to not have, like, I don't know that Godot has a
40:33SPEAKERsupport infrastructure around it at this point that's adequate.
40:37SPEAKERYeah, you absolutely want like proper support, commercial support for something like that
40:41SPEAKERin an actual business.
40:42SPEAKERI was throwing that stuff out there more as a, like, if you're curious about this stuff,
40:47SPEAKERhere are a couple of free engines you can go to and download and mess with yourself
40:51SPEAKERif you just want to like kind of see how some of these pieces are put together.
40:53SPEAKERI was going to say also, I think both Unity and, so one of the things that surprised me
40:57SPEAKERabout all of this stuff is that at small scale, most of the software is relatively inexpensive
41:01SPEAKERor free.
41:02SPEAKERSo, for example, our source control per force is free until you have more than five users.
41:07SPEAKEROh.
41:08SPEAKERYeah.
41:09SPEAKERI mean, you still have to pay for a server to host it and all, or you have to host it
41:12SPEAKERyourself.
41:13SPEAKERI guess that's kind of, that's like been the model though, right?
41:16SPEAKERDid Unreal kick that off like 15 ish years, maybe 20?
41:19SPEAKERI feel like the first time they did the build something Unreal thing after that, they announced
41:24SPEAKERthat you could download it for free and previously it had been like you have to pay a million
41:27SPEAKERdollars to get access to Unreal.
41:29SPEAKERRight.
41:30SPEAKERWhich was obviously smart.
41:31SPEAKERLike you want to put the tools in people's hands with barrier to entry and then once
41:35SPEAKERthey start making money, you start taking some of that money.
41:38SPEAKERI was just saying that one thing I forgot about on the build server front is that if
41:42SPEAKERyou have internal tooling, the build server probably also manages and updates and distributes
41:46SPEAKERthat to people.
41:47SPEAKEROkay.
41:48SPEAKERSo, for example, we have a custom version of Unreal editor because we modified our Unreal
41:52SPEAKEReditor to do things that were specific to the game.
41:56SPEAKERAnd when we update that, the build server does that and then you run a batch file on
41:59SPEAKERyour computer that downloads the latest version of it from a server someplace.
42:04SPEAKERAnyway, yeah, the different game engines, like everybody always wants to talk about
42:11SPEAKERgame engines.
42:12SPEAKERAnd it's funny because as I get further into this, it's more and more clear that really
42:18SPEAKERthe game engine matters way less than people think it does.
42:23SPEAKERI'm sure on the developer side, it's a means to an end, right?
42:26SPEAKERLike it's just a tool.
42:27SPEAKERYeah.
42:28SPEAKERI mean, well, it's a toolbox, right?
42:31SPEAKERWell, yes.
42:32SPEAKERYes.
42:33SPEAKERBecause when you buy Unreal, you're getting all of their...
42:37SPEAKERIf you want to make a big ass open world, it turns out Unreal is pretty good at that
42:40SPEAKERbecause what Epic uses Unreal to make is a big ass open world game, right?
42:45SPEAKERIf you want to do destructible stuff, they're good at that.
42:48SPEAKERIf you want to ship stuff on mobile, maybe Unreal is not your choice, right?
42:54SPEAKERThere are probably better choices than that.
42:56SPEAKERIf you want to make a text adventure, Unreal, probably massive overkill, right?
43:02SPEAKERThere's much better choices for that.
43:03SPEAKERBut there's just a lot of options out there now, which is I think good for everyone.
43:10SPEAKERYeah.
43:11SPEAKERYeah.
43:12SPEAKERI mean, a lot of that seems to have come from YouTube and I don't know how to describe like
43:19SPEAKERforum chatter.
43:20SPEAKERI guess it's like a kind of discussion among consumers and users about which game looks
43:25SPEAKERbetter and runs better and what engine is it on, oh, everybody should use that engine.
43:29SPEAKERLike it's just kind of a very reductive style of discourse.
43:32SPEAKERI'm as guilty of this as the next person.
43:34SPEAKERI did a what you need to know about game engines article in like 2005 for Maximum PC.
43:41SPEAKERAnd I was like, look, Gabriel, really?
43:42SPEAKERThey're doing some crazy stuff up there.
43:44SPEAKERLook at what the people in Norway are doing, the dice, they got cool things going.
43:48SPEAKERAnd then we all know how to Frostbite turned out.
43:50SPEAKERSo my bad.
43:53SPEAKERBut yeah, it's more like all those YouTube videos about, oh, I could tell this is a Unity
43:59SPEAKERgame cause it has the Unity look what they're, what they're identifying is, Hey, these people
44:03SPEAKERuse Unity's default lighting model and which is a fine choice.
44:09SPEAKERI don't know.
44:10SPEAKERIt's just YouTube.
44:12SPEAKERI don't like YouTube discourse.
44:13SPEAKERThe older I get bread, there should we, should we talk about a source control or QA next
44:21SPEAKERsource control was the first topic, but we kind of blew past it a little bit.
44:24SPEAKERIt's fine.
44:25SPEAKERWell, we talked about per force.
44:26SPEAKERI mean, the interesting thing here with games to me is that like, when I think about source
44:31SPEAKERcontrol, it's a source, it's a method of versioning, right?
44:33SPEAKERLike just to be clear, it is a method of like keeping your code base in sync between multiple
44:39SPEAKERdevelopers, managing changes and versions that you can roll back to and et cetera, et
44:43SPEAKERcetera.
44:44SPEAKERBut when I've always thought about source control, it's purely in the context of text
44:48SPEAKERfiles, which are code, but then, but then like you mentioned, you've got like, you've
44:52SPEAKERgot mass amounts of assets, sounds, textures, sounds like 3D models, Maya files.
45:01SPEAKERAnd I, and I'm sure this varies between studios, but I like, I wonder how much, I don't know
45:07SPEAKERwhat the trend is about.
45:08SPEAKERLike do, does a single artist generally kind of have ownership of a model, for example,
45:11SPEAKERlike a specific character model, or are there like a bunch of different artists working
45:14SPEAKERon the same model at different times?
45:16SPEAKERSo I'm going to say, well, generally only one person can check something out at a time
45:21SPEAKERthat's like,
45:22SPEAKERI mean, I mean, successively, like, like, oh, like I'm sure, I'm sure it happens that
45:27SPEAKERmaybe somebody else will go back and touch something up if somebody else is busy or whatever.
45:30SPEAKERI'm just wondering how much of those types of assets need to also be versioned and controlled.
45:34SPEAKERAll of them.
45:35SPEAKERCause if you're spending money making them, you should version them.
45:37SPEAKEROf course.
45:38SPEAKERLike, like just realistically, if you have, if you have cost in something, it should be
45:43SPEAKERon your Perforce.
45:44SPEAKERLike we even keep marketing images on the Perforce.
45:47SPEAKERI phrased that poorly.
45:48SPEAKERI meant more just, I wonder how many different people are touching things that aren't because
45:51SPEAKERlike code you're constantly collaborating on with different people.
45:54SPEAKERArt is the same.
45:55SPEAKERSo there are, there are, I mean, it depends on the scale of your studio.
45:58SPEAKERIf you're a one programmer, one artist studio, then one person's doing all the art.
46:02SPEAKERRight.
46:03SPEAKERBut if you have 10 people working on the game, then, and eight of them are artists, then
46:07SPEAKERprobably eight people will touch a lot of the, six or eight people will touch a lot
46:10SPEAKERof the stuff in the game.
46:11SPEAKERFor example, if you're talking about a character, a character is a good example.
46:18SPEAKERSo if you want to make a new character for the game, you have a concept artist probably
46:22SPEAKERthat draws, draws pictures of what it looks, what the character looks like, shows them
46:25SPEAKERin some different positions, shows some like potential animation ideas, stuff like that.
46:30SPEAKERThen that person hands that, that concept off to a character artist, the character,
46:34SPEAKERand often, oftentimes, especially at a small studio, the character artist, the concept
46:39SPEAKERartist will be somebody who also does other things, they do texture work, or they do like
46:43SPEAKERpunching up of environments or something like that.
46:45SPEAKERThey're just, they're just good at drawing, too.
46:46SPEAKERYou get a twofer with that person if you're lucky.
46:50SPEAKERSometimes it's the character artist.
46:52SPEAKERThe character artist will model that character out if it's a 3D game or draw sprites if it's
46:55SPEAKERa 2D game.
46:57SPEAKERAnd then the character artist hands the model off to the, to the technical artist.
47:01SPEAKERThe technical artist rigs it so that it can move so that it has, you know, so you can
47:05SPEAKERhook it into the animation system of the game and the technical artist hands it to the animator
47:08SPEAKERand the animator puts the animations on it.
47:11SPEAKERThat's a, that's a never ending job because, you know, there's thousands of animators,
47:15SPEAKERlike, like we have dozens and dozens and dozens of stumble animations in our game for like
47:19SPEAKERaliens when they get shot so that when you shoot them in the legs, they stumble appropriately.
47:25SPEAKERStumble artist has to take over at that point.
47:27SPEAKERSo many stumbles.
47:28SPEAKERBut yeah, I think that's where I was being reductive or overly simplifying the situation.
47:32SPEAKERI didn't, I guess I didn't realize all those disciplines had like abstracted out into different
47:36SPEAKERroles as much as they apparently have, like I kind of was thinking of it from probably
47:40SPEAKERa very old fashioned, like full stack, like, well, if you model it and then you animate
47:44SPEAKERit and then it's done.
47:45SPEAKERRight.
47:46SPEAKERBut it sounds like that those, I'm sure just like everything else, things get so complicated
47:50SPEAKERthat they have to spin out into different roles.
47:52SPEAKERRight.
47:53SPEAKERYeah.
47:54SPEAKERAnd each of those has different strengths, right?
47:55SPEAKERLike the person who's good at sculpting a soft body model is maybe not going to be great
47:59SPEAKERat figuring out how the movement should work for them.
48:02SPEAKERRight.
48:03SPEAKERBecause doing animating is really hard, doing the rig is a technically challenging job that
48:09SPEAKERalso requires people to like, like technical artist is one of those technical artists is
48:16SPEAKERlike associate editor in the publishing business, right?
48:18SPEAKERAn associate editor, depending on which publisher, which, which, which, whether you're at a magazine
48:22SPEAKERor website or what magazine or website you're at an associate editor could be the lowest
48:26SPEAKERperson on the totem pole, or it could be the person who does the vast majority of work
48:30SPEAKERon the publication, like the frontline work, and then all of their work gets filtered up
48:35SPEAKERthrough other people.
48:36SPEAKERRight.
48:37SPEAKERA technical artist is they can be doing rigs for character models or for anything that
48:42SPEAKERmoves in the game.
48:43SPEAKERThey can be writing shaders.
48:44SPEAKERThey can be helping the VFX artists with, with like, with like writing scripts to, to,
48:50SPEAKERto make the VFX work.
48:52SPEAKERThey can be hooking stuff like virtual visual effects into the game.
48:55SPEAKERRight.
48:56SPEAKERThey can be building particle systems or lighting models or all sorts of like crazy, crazy,
49:01SPEAKERbroad spectrum.
49:03SPEAKERBut a lot of that work is stuff like rigging skeletons on, on and rigging, rigging character
49:09SPEAKERmeshes.
49:10SPEAKERAnyway.
49:11SPEAKERSo yeah, everybody touches everything.
49:14SPEAKERAnd like when, when you get all that done, then an engineer has to hook it into the,
49:18SPEAKERinto the game, like has to actually put it in and say, okay, this can be it, but in the
49:21SPEAKERspawn tables, here's the AI for it.
49:24SPEAKERHere's the, here's the, here's the, the game, the gameplay designer will say, here's what
49:28SPEAKERit, what?
49:29SPEAKEROkay.
49:30SPEAKERIt has an attack.
49:31SPEAKERIt has a ranged attack.
49:32SPEAKERIt's gonna be a fireball.
49:33SPEAKERIt's gonna be an ice ball.
49:34SPEAKERIs it going to be a laser?
49:35SPEAKERIs it going to be him picking up rocks from the ground and throwing them at you?
49:38SPEAKERAnd, and like, so 20 people will touch that even if it's a 10 person studio, somehow,
49:42SPEAKERI don't know how that works, but
49:43SPEAKERThat'll make sense.
49:46SPEAKERSo, so you, you fall to use Perforce, is that like super common, would you say in game development?
49:51SPEAKERSo in, in like tech world, get is what people get, get is what people seem to use.
49:59SPEAKEREngineers like get, cause it's really good at doing code.
50:01SPEAKERThere's a lot of, a lot of, a lot of good command line tools, so you can just jam out
50:05SPEAKERcommits really fast.
50:06SPEAKERRight.
50:07SPEAKERYeah.
50:08SPEAKERI'm, I'm, I'm thinking more once, once a mass quantity of binary assets are also involved,
50:11SPEAKERlike, like I, like I said, traditionally, I think of, of get as, as a text only thing,
50:16SPEAKERbut, but you've, you've mentioned here, GitHub LFS, is that a, what is, what is that?
50:20SPEAKERYeah.
50:21SPEAKERIt's their binary, is their binary tool.
50:24SPEAKERWe use LFS at, at foo with kind of mixed results.
50:28SPEAKERIt was fine.
50:29SPEAKERIt was also brand new when we were using it at foo.
50:31SPEAKERSo I'm sure, I'm sure it's come along pretty nicely.
50:34SPEAKERI think it stands for large file system, but I'm not a large file storage.
50:38SPEAKERAnd it basically gives you versioning on, on binary assets.
50:42SPEAKERIt's a little like Perforce has been around for a long time.
50:46SPEAKERYou probably, if you're a small team and you've had somebody who works in dev ops or on production
50:49SPEAKERor whatever at a studio, they probably know how to use Perforce.
50:53SPEAKERIt's not particularly complicated to set.
50:55SPEAKERIt's it requires a fair amount of touching, but it isn't, it isn't complicated touching,
51:00SPEAKERif that makes sense.
51:02SPEAKERAnd, and also as with many things, the fact that you're, that most of the people you'll
51:09SPEAKERhire in know how to probably use Perforce is important.
51:16SPEAKERThe big thing you get from source control is the ability to roll stuff back.
51:18SPEAKERIf you make a change, you don't like it.
51:19SPEAKERYou can say, "OK, I only want to do this part of this change," or whatever.
51:22SPEAKERPerforce does that.
51:23SPEAKERThe knock against Perforce is it's expensive.
51:26SPEAKERLike, GitHub, both from a licensing perspective per user and from a hosting perspective, it
51:34SPEAKERcan be pretty unwieldy.
51:36SPEAKERPart of that is because game assets get large.
51:39SPEAKERI think at one point, our Perforce repo was two terabytes because we had versions of all
51:45SPEAKERsorts of old versions of different stuff and weird offshoot projects and all sorts of other stuff.
51:52SPEAKERBut you can't self-host it.
51:53SPEAKERYou can put it on a machine in your premises, then you're responsible for figuring out how
51:55SPEAKERto back it up and all that stuff.
51:58SPEAKERBut it is a thing you can do.
52:02SPEAKERYou can also put it on a $300 a month colo machine someplace, and it'd probably be fine.
52:10SPEAKERIt's free, it's one of those things that's free for the first five people, and then once
52:14SPEAKERyou go beyond that, you have to talk to somebody and they're like, "OK, here's what we're going
52:17SPEAKERto charge you."
52:19SPEAKERCall for a quote.
52:20SPEAKERIf you have to ask, you're in too deep.
52:25SPEAKERBut it's reliable and good, and it's what most game studios use, is my understanding,
52:30SPEAKERif they can afford it.
52:33SPEAKERThey actually built an add-on for the engineers so that you can just interface with your Perforce
52:38SPEAKERrepo through Git now, which I thought was pretty funny.
52:41SPEAKERThat's pretty good.
52:44SPEAKERYour Git commits into Perforce now.
52:46SPEAKERThat's the build environment equivalent of using the Vim bindings in VS code.
52:51SPEAKERIt really and truly is.
52:58SPEAKERI mean, that's kind of it.
52:59SPEAKERIt's source control, man.
53:00SPEAKERIt's important.
53:01SPEAKERHere's my question.
53:03SPEAKERWho manages all of this stuff?
53:05SPEAKERBecause between the source control and the build server, it seems like you would need
53:08SPEAKERa full-time person just to focus on doing all of this.
53:13SPEAKERIt's one of those things that like when we set up the company and when we first started
53:16SPEAKERdoing daily builds, one of our engineers set all this stuff up, right?
53:20SPEAKEROK.
53:21SPEAKERMaybe it is automated enough that you'd actually...
53:23SPEAKERI was asking that somewhat facetiously because I was expecting an answer of like, "Yeah,
53:27SPEAKERit sure would be nice to have somebody to focus on just this, but maybe it actually
53:32SPEAKERis more sustainable than it used to be, or than it sounds."
53:36SPEAKERWell, so for a long time, we were doing a cloud host for the Perforce, which was expensive
53:40SPEAKERbut nice because we didn't have to do any server maintenance stuff.
53:45SPEAKERThe server maintenance, it's not turnkey, but it's like, "Hey, you can spend a couple
53:49SPEAKERhours a month working on something and it's probably fine."
53:54SPEAKERIf something breaks, then everything stops while you fix that.
53:58SPEAKERThat's kind of what I was thinking about.
53:59SPEAKERThis is a weird tangential example, but it made me think back to which Pixar movie?
54:03SPEAKERWas it "Toy Story 2," I think?
54:04SPEAKER"Toy Story 2," yeah.
54:05SPEAKER"Toy Story 2," you can read about this.
54:08SPEAKERThere are articles from the time describing this.
54:10SPEAKERThey basically lost the movie.
54:12SPEAKERThere's a bonus feature.
54:13SPEAKERThere's a short on the DVD.
54:15SPEAKERIt's on YouTube.
54:16SPEAKERI think you guys did some stuff around that, didn't you?
54:21SPEAKERI thought I saw some tested work cited as reading about that.
54:24SPEAKERWhen they posted it, Wes did a short post about it, I believe.
54:28SPEAKERThat's what I was thinking of.
54:30SPEAKERAnyway, the whole studio ground to a halt because it was all hands on deck trying to
54:33SPEAKERrecover stuff, work that had been lost.
54:36SPEAKERI was just thinking along those same lines, if something bad happens where backups need
54:41SPEAKERto be restored or X, Y, Z, it seems like having somebody who was permanently on top of that
54:46SPEAKERsituation would be better than having to pull people off of their actual work to fix things.
54:51SPEAKERWell, yeah, but also you have to pay that person the time when there's nothing bad happening.
54:55SPEAKERSo pros and cons.
54:57SPEAKERAgain, this is one of those places where being a Swiss army knife is sometimes useful because
55:03SPEAKERif you're looking for an indie studio job and somebody's like, "Hey, what else do you
55:06SPEAKERknow besides the thing that you do 20 days a month?"
55:11SPEAKERAnd they're like, "Oh, well, I ran the Perforce server at my last studio."
55:15SPEAKERI was like, "Oh, OK."
55:18SPEAKERSomebody says I love PowerShell scripting in a job interview, you're like, "Fuck, yes.
55:22SPEAKERYou're my boy."
55:23SPEAKERAnd then back away slowly.
55:24SPEAKERYeah, no, and then you never talk to that person again.
55:30SPEAKERSo I was gonna say the last thing that people don't think about, I think often, is that
55:34SPEAKERyou have to have some core source of truth.
55:37SPEAKERSo you need a place that everyone has access to that you put your documentation for how
55:43SPEAKERto do stuff in the game.
55:45SPEAKERSo for example, if you want to put a new sound into Wwise, here's the process for doing that
55:50SPEAKERwith our game because it's different for every single game on the face of the earth because
55:56SPEAKERwhen you get Wwise, you get access to source code and people make source code changes and
56:00SPEAKERthey build tools that work specifically with your type of game.
56:03SPEAKEROur game has a bazillion external lines of dialogue that are triggered by things that
56:07SPEAKERhappen in gameplay.
56:11SPEAKERWwise isn't set up for that.
56:12SPEAKERWe have a system that triggers that in the game and then hands off the audio file to
56:18SPEAKERtrigger to Wwise and then Wwise just plays the audio file.
56:21SPEAKERThat's a completely non-standard way to do that.
56:23SPEAKERNormally, you would just put the dialogue into Wwise and then build a bunch of independent
56:26SPEAKERtriggers and it would have been an unbelievable amount of work on our end to hook all that
56:31SPEAKERup.
56:32SPEAKERI can tell you from having fiddled with a zillion open source projects at this point
56:35SPEAKERand also having recently done a couple of very small utilities myself, it seems very
56:39SPEAKEReasy to build bespoke functionality and then not document how to use it.
56:43SPEAKERLook, I don't want to pick on the person who did this because they started on the right
56:47SPEAKERpath and then got distracted and did something else, which happens a lot at small studios.
56:52SPEAKERBut I was looking for how to add this the other day, and I went to the Wiki page for
56:59SPEAKERit.
57:00SPEAKERThe thing I saw was, this page is a stub.
57:03SPEAKERSo, you know...
57:06SPEAKERBarker, everyone loves a stub.
57:07SPEAKERHill: Yeah, everybody loves a stub.
57:10SPEAKERSo, you want core source of truth for both documentation.
57:13SPEAKERYou want to also have work that needs to be done, a place to put tasks that's shared across
57:18SPEAKERthe team.
57:19SPEAKERThere's a lot of stuff that anybody can do, or any subset of the team can do.
57:24SPEAKERYou just want to have a bin of things to work on.
57:29SPEAKERThe way we often would work is, if we had a four-month loop on a patch, on a big patch,
57:34SPEAKERwe would spend a four-week loop on a patch.
57:39SPEAKERWe would spend two weeks working on features, and then the third week would be all the bugs
57:45SPEAKERthat we found in that time period, in that two weeks, would go into a bug bin.
57:50SPEAKERThen, the engineers and the artists and everybody who didn't have other stuff to be doing would
57:54SPEAKERjust sit and mow down those lists of bugs.
57:58SPEAKERThey'd grab the first one out of the pool that they looked like they knew how to fix
58:01SPEAKERand tackle it.
58:02SPEAKERBarker, I assume that's where terms like "agile" and "scrum" come in.
58:06SPEAKERThose are just different philosophies for tackling project management and flow and scheduling
58:12SPEAKERand stuff.
58:13SPEAKERKline>> Yeah, they're a higher concept than that.
58:19SPEAKERIf you think about personal organization and scheduling, like the 40 boxes or whatever,
58:25SPEAKERthat thing, they're frameworks in which you can manage a team's time across disciplines.
58:38SPEAKERI think the core -- I'm bad at this, because this isn't the side of the stuff I ever do.
58:42SPEAKERI think the idea of "agile" is that you ship every X period of time, and you devote a certain
58:48SPEAKERpercentage of that time to new stuff, and a certain percentage of that time to fixing
58:52SPEAKERbugs, and a certain percentage of that time to doing -- "scrum" is a little bit of a different
58:56SPEAKERapproach.
58:57SPEAKERBarker, I'm throwing out, clearly, jargon that I cannot well define myself.
59:03SPEAKERThe point is, there are entire modes of thinking built around making this stuff more efficient.
59:09SPEAKERKline>> Yeah, this is all stuff that came from somebody reading the Mythical Man month
59:12SPEAKER20 years ago.
59:13SPEAKERBarker, yes.
59:14SPEAKERI remember hearing about that book in the 90s from game developers, or seeing them talk
59:18SPEAKERabout it.
59:19SPEAKERKline>> Yeah, well, the idea of the Mythical Man month is that if you have a project that's
59:25SPEAKERgoing to take one person two months, and you add two people, it's going to take three months
59:29SPEAKERinstead of two months, instead of one month.
59:34SPEAKERSo yeah, the idea is that you have to build in some way to communicate and some way to
59:39SPEAKERshare information.
59:40SPEAKERThat's what a lot of these tools are for.
59:41SPEAKERA lot of people use Wikis.
59:44SPEAKERConfluence is just a hosted wiki platform that's private, and you can build docs in
59:49SPEAKERrelatively easily.
59:50SPEAKERPeople use Notion and stuff like that for that, as well.
59:52SPEAKERBarker, I feel like Trello picked up some steam, because when Epic launched the Epic
59:57SPEAKERstore, they published their Trello board publicly for people to look at to see what they were
1:00:01SPEAKERworking on.
1:00:02SPEAKERKline>> So, publishing public Trello boards was very popular five years ago.
1:00:07SPEAKERThat's just become a thing that people do, especially if you're an early access title
1:00:10SPEAKERor something like that.
1:00:11SPEAKERIt's good to have a public Trello, because people understand it.
1:00:13SPEAKERTrello is more in the task management side.
1:00:17SPEAKERThe idea Trello, Jira, stuff like that are what often do that.
1:00:25SPEAKERWith Trello, you break down individual tasks and subtasks.
1:00:30SPEAKERFor example, in our building a new character model aspect, you'd have a Trello or a Jira
1:00:35SPEAKERtask that was, "Hey, add character model X to the game."
1:00:40SPEAKERThen there'd be subtasks for concept, creating the character model, rigging the model, doing
1:00:46SPEAKERthe base animations you need to get in the game, adding the gameplay design, adding it
1:00:50SPEAKERto the game, hooking it up to the spawn tables, and stuff like that.
1:00:56SPEAKERDepending on how big the team is, that can be anywhere from a thing that somebody does
1:01:02SPEAKERas part of their job, or if you have a conscientious small team, everybody can just do it.
1:01:08SPEAKEREverybody manages their own stuff, and then you go through and work down the list together.
1:01:13SPEAKERIf you have a Ubisoft-sized Assassin's Creed team, I assume they have 20 people at each
1:01:21SPEAKERstudio that do that work.
1:01:23SPEAKERThey have people on the individual teams.
1:01:25SPEAKERThey have some sort of core.
1:01:28SPEAKERIf anybody at Ubisoft wants to share how you do this, I'm fascinated by it, because it
1:01:32SPEAKERseems like a nightmarish job.
1:01:34SPEAKERI'm sure they hand out tasks like, "Make Rome to Ubisoft Shanghai."
1:01:42SPEAKERThen there are notes in there of places that have to exist.
1:01:45SPEAKERIt has to have the Coliseum, it has to have the blah, blah, blah.
1:01:52SPEAKERFrom there, the individual teams at the individual studios parse out what they're working on.
1:02:00SPEAKERBecause of the scale of that art and the size of that art, I'm sure that Ubisoft Shanghai
1:02:05SPEAKERworks on one part of the map, and somebody works on a different part of the map.
1:02:08SPEAKERSame thing for Fortnite.
1:02:09SPEAKERFortnite has a bazillion contributors for it now.
1:02:16SPEAKERPeople call that a producer, is what it often ends up being.
1:02:21SPEAKERThose are folks that manage the Trello, manage the Jira, manage the Confluence, and help
1:02:28SPEAKERkeep the whole team on time.
1:02:30SPEAKERHonestly, having a good, well-populated Jira with the actual tasks helps you know how much
1:02:35SPEAKERtime stuff's going to take more than anything else.
1:02:38SPEAKERThey're always outliers in a bug that'll take one person four days to figure out, or a week
1:02:42SPEAKERto figure out.
1:02:43SPEAKEROften, if it's building a new map for the game, you know that's going to be a month
1:02:51SPEAKERfor an environment artist, and that's all they're going to do that month.
1:02:53SPEAKERYou can just put that down and know that's what they're going to do.
1:02:57SPEAKERThen it'll be another three months to pretty that up after they've done the first pass.
1:03:01SPEAKERThat's so crazy to think about.
1:03:03SPEAKERAgain, I know I keep referencing this, but coming off that Romero book, reading about
1:03:06SPEAKERsome of his stories about staying up all night and making an entire Doom map, it's just like,
1:03:12SPEAKER"I just decided to stay up all night, and by 6 a.m., I had a finished map that ended
1:03:15SPEAKERup in Doom II."
1:03:18SPEAKERJust the complexity, the increase, the orders of magnitude increase in complexity in building
1:03:22SPEAKERthings like environments in the last 20, 30 years.
1:03:26SPEAKERBuilding from 2D to 3D was a big one.
1:03:28SPEAKERBut then, think about the fidelity.
1:03:30SPEAKERI think what Quake 1 characters were, like, 300 polys or something.
1:03:37SPEAKERYou think about that versus a normal mapped, modern, first-person shooter game character
1:03:41SPEAKERthat's a million polys, effectively.
1:03:49SPEAKERThis is why, if you want to make a game with a small team, you don't make something that
1:03:52SPEAKERlooks like Metal Gear Solid 5 or Call of Duty or whatever.
1:03:56SPEAKERIt's because it's too many people.
1:03:57SPEAKERThere are more people making guns in Call of Duty than working on most indie teams,
1:04:02SPEAKERI think, at this point.
1:04:04SPEAKERThat checks out.
1:04:05SPEAKERYeah.
1:04:06SPEAKERThat's why the guns are all animated and cool.
1:04:09SPEAKERWhen they do those cool animations where the guy is running and he looks at his gun as
1:04:12SPEAKERhe's running, he holds up and does something you've never seen him do before, somebody
1:04:16SPEAKERspent a month working on that.
1:04:18SPEAKERI'm guessing that's a month that a lot of smaller studios don't have.
1:04:22SPEAKERSeems like a bad way to spend your money.
1:04:26SPEAKERTrello's good.
1:04:27SPEAKERIt gives you public visibility.
1:04:31SPEAKERIt's important to have that shared source of truth, so that if somebody gets hit by
1:04:35SPEAKERa bus or somebody doesn't have time to ... Honestly, it's just nice, so you don't have to go ask
1:04:41SPEAKERthe person who knows how to do everything how to do everything to have that documentation.
1:04:47SPEAKERYou're buying future time for yourself when you write good docs.
1:04:52SPEAKERSo, that's it.
1:04:53SPEAKERThere's a ton of stuff we didn't talk about here that other people ... Also, the important
1:04:58SPEAKERthing to think about with games, every single game is different from every other game, including
1:05:05SPEAKERsequels inside the same studios.
1:05:07SPEAKEREverybody has different processes.
1:05:10SPEAKERWhat is a producer at one job?
1:05:11SPEAKERIs a publisher at a different one?
1:05:14SPEAKERThere's all sorts of DevOps stuff.
1:05:18SPEAKERThis all lives under the shell of DevOps, I think.
1:05:22SPEAKERIt's kind of crazy.
1:05:23SPEAKERIt's a weird thing.
1:05:24SPEAKERI don't think I've heard a whole lot of people talk about it publicly, so maybe our nerdy
1:05:30SPEAKERaudience will enjoy it.
1:05:31SPEAKERI hope you like this.
1:05:32SPEAKERYeah.
1:05:33SPEAKERHopefully this gives some insight into the nuts and bolts part of getting a video game
1:05:37SPEAKERout the door.
1:05:38SPEAKERYeah.
1:05:39SPEAKERThe big thing for me was realizing that realistically, most of the time when you make a change to
1:05:45SPEAKERthe game, you're not going to get to test it in the game until the next day at the earliest.
1:05:51SPEAKERIt's not like you press compile, and then it's on the test version of your iPhone in
1:05:55SPEAKER10 minutes, like a phone app.
1:05:58SPEAKERYes.
1:05:59SPEAKERSo, anyway.
1:06:00SPEAKERMaking games.
1:06:01SPEAKERGame dev.
1:06:02SPEAKERMaking games.
1:06:03SPEAKERNobody should do it.
1:06:04SPEAKEREvery game is a miracle.
1:06:05SPEAKERWhoa.
1:06:06SPEAKERAll right.
1:06:07SPEAKERMaking games is...
1:06:08SPEAKERI enjoy making games, but it is...
1:06:11SPEAKERWe're pushing a patch this morning, actually.
1:06:13SPEAKERLook, you're not the first nor the 10th developer I've heard say something like that.
1:06:18SPEAKERYeah.
1:06:19SPEAKERThat's a pretty widely held opinion.
1:06:21SPEAKERBasically the fact that any game ever ships is a surprise to me.
1:06:24SPEAKERWhen somebody ships a game, they're like, "How did they ship this when it had this many
1:06:27SPEAKERbugs?"
1:06:28SPEAKERI'm like, "Man, I know exactly how that happened."
1:06:29SPEAKERAt some point, you've got to get it out the door.
1:06:32SPEAKERYes.
1:06:33SPEAKERYeah.
1:06:34SPEAKERIf you have questions, please send them in.
1:06:36SPEAKERWe'll answer them in the next Q&A at techpod@content.town or the Questions Seeking Answers channel on
1:06:41SPEAKERthe Discord.
1:06:42SPEAKERIf you're not in the Discord and would like to be in the Discord, you can do so by becoming
1:06:47SPEAKERa fabulous techpod patron.
1:06:49SPEAKERYou can?
1:06:50SPEAKERYeah.
1:06:51SPEAKERThanks.
1:06:52SPEAKERBrad, did you know this is a listener-supported show?
1:06:53SPEAKERI've heard.
1:06:54SPEAKERI saw a friend of the pod talking about advertising, Jeff Gerstmann talking about advertising on
1:06:59SPEAKERhis Blue Sky yesterday.
1:07:01SPEAKERI was like, "Man, I don't want to have to deal with advertising."
1:07:04SPEAKERLook, all it's going to do is annoy people into signing up for the Patreon, so they get
1:07:09SPEAKERthe ad-free version.
1:07:10SPEAKERI'd rather just ask them to support the podcast nicely and say, "Hey, we'd rather not have
1:07:15SPEAKERto annoy anyone.
1:07:16SPEAKERWe'd rather just have no ads," is where I land.
1:07:19SPEAKERYou can support the podcast by going to patreon.com/techpod.
1:07:24SPEAKERAgain, it's patreon.com/techpod.
1:07:28SPEAKERBecause you have to say everything three times, I'm going to say it again.
1:07:30SPEAKERWait.
1:07:31SPEAKERI can say it.
1:07:32SPEAKEROh, yeah.
1:07:33SPEAKERWhat is it, Brad?
1:07:34SPEAKERIs it patreon.com/techpod?
1:07:35SPEAKERIt is patreon.com/techpod.
1:07:36SPEAKERWait.
1:07:37SPEAKERYou said it four times.
1:07:38SPEAKERDoes that now cancel out?
1:07:39SPEAKEROne of the three?
1:07:40SPEAKERI think more people are going to sign up, because we said the fourth.
1:07:44SPEAKERYou can go to the Patreon.
1:07:46SPEAKERYou can sign up for five bucks a month to get access to the Patreon-exclusive episode.
1:07:51SPEAKERLast month, we talked about a bunch of stuff.
1:07:54SPEAKERPeople seemed to really like it.
1:07:56SPEAKERIt generated a lot of conversation in the new episode discussion subforum.
1:08:00SPEAKEROh, and it did exactly what you said it might do, which is that people started posting their
1:08:04SPEAKERown tips.
1:08:05SPEAKERI got a tip out of there yesterday that made me very happy.
1:08:08SPEAKERYep.
1:08:09SPEAKERYep.
1:08:10SPEAKERFor five bucks, you can join our ranks of executive producer tier patrons who we thank
1:08:17SPEAKERevery month, including Nick Johnston, Paddle Creek Games' makers of Fractured Veil -- that's
1:08:21SPEAKERcoming out in October, by the way.
1:08:25SPEAKERAndrew Slosky, Jordan Lippett, Bunnyfiend, Just Wedge 3.0, Joel Krauska, Twinkle Twinkie,
1:08:32SPEAKERDavid Allen, James Kamek, and Pantheon, makers of the HS3 high-speed 3D printer.
1:08:37SPEAKERThank you all so much.
1:08:38SPEAKERYes, thank you, one and all.
1:08:42SPEAKERAnd that'll do it for us this week.
1:08:43SPEAKERWe'll be back next week.
1:08:44SPEAKERYeah.
1:08:45SPEAKERI think next week we might talk about iOS stuff, huh?
1:08:46SPEAKERBecause it's time.
1:08:47SPEAKERNo, wait a minute.
1:08:48SPEAKERI've got another idea.
1:08:49SPEAKEROoh.
1:08:50SPEAKERWe'll talk about it.
1:08:51SPEAKERForeshadow?
1:08:52SPEAKEROkay, we'll talk about offline.
1:08:53SPEAKERIt's a milestone, let's say.
1:08:54SPEAKEROh.
1:08:55SPEAKEROh, my.
1:08:56SPEAKERWe'll discuss, and we'll see what we come up with.
1:08:59SPEAKERYeah.
1:09:00SPEAKERNew iOS is imminent.
1:09:01SPEAKERMaybe we should talk about that.
1:09:02SPEAKERYeah, I'm toying with installing the beta.
1:09:03SPEAKERI don't know if it's a good idea or not.
1:09:04SPEAKERUgh.
1:09:05SPEAKERUgh.
1:09:06SPEAKERThey're not usually beta anymore by this point.
1:09:09SPEAKERI don't know.
1:09:10SPEAKERFor this exploit that just happened.
1:09:11SPEAKERWell, fair.
1:09:12SPEAKERFair.
1:09:13SPEAKEROkay, we'll see you all next week.
1:09:14SPEAKERThanks for listening, everybody.
1:09:15SPEAKERHave a good one.
1:09:16SPEAKERBye.