Manager In A Strange Land: Fat Build Processes Considered Harmful

Working with the XNA GS content manager is a mixed blessing.  The content manager subscribes to this philosophy:

In a perfect world, we wouldn't check in our game-ready assets.  We'd check in our raw, source assets only and our build systems will build them locally using the parameters we've defined.  If you want to change the compression, resolution - or even maybe the gamma ramp or color saturation! - on a texture or mesh, you just tweak a number and then everyone, on their local machines, the next time they build, get the new parameter.  Also, a change in an art asset would percolate through to all the levels on which that asset is used automatically.  Most importantly it insures that the source assets are actually checked in.

Sounds pretty good, right?  I've even heard it said that doing it the other way, the way where you manually build some of your game-ready assets and check those in, are an "antipattern in game development."

But if that's the case, how come most of the succesful studios I know do it the "antipattern" way (anyone using Unreal with its wad files, for example) and the teams I've heard of with the big fat build systems are catastrophic?

What I've found is that in this non-perfect world we live in, computers are just too slow for the "perfect" build system.  Because for every file that you need to take down the chain from raw asset to finished shippable product, you have to do a dependency check.  "Did this file or its parameters change?"  And the dependency check takes finite time.  And on your typical big-budget console or PC title, you have tens or maybe even hundreds of thousands of files.  That's if nobody changed anything.  It turns out it's way too much time to do every time you hit the "build" button (which is the way it currently is with XNAGS, which would probably have to changed or worked-around if you wanted to do a big-budget game with it). 

So now you've got a choice - you can provide your artists two ways to get their assets into the game (the slow way, build everything; or a back door to get an individual texture in) at which point you no longer have the perfect build system...or you can make your artists suck it and wait a few minutes every time they actually want to see one of their changes in the game.

And that's for a build where nothing has changed.  Typically, you come in the morning, you do an update/get/sync (whatever your asset control system calls it) and hundreds of files have changed because you're working on a team.  YMMV, but this can take up to an hour depending on how much has changed and how slow your build processes are.  So you've got your artists sitting idle in the morning, waiting.  If they're working together on something, and need to pass assets back and forth, then after Person A checks in his change and Person B gets it, Person B has that up-to-an-hour wait to start working with Person A's stuff.

This isn't likely to change in the near future!  The size of our games seems to be increasing just as fast as the speed of our hard-drives.  Maybe even faster. 

Which is why on most of my last projects we didn't subscribe to the "perfect build system" philosophy, as much as we would have liked to.  Artists shepherded each asset from raw to console-ready and checked everything in.

The biggest problem with this sytem - and the reason the "perfect build" is so appealing - is an artist could forget to check in the source asset.  Another artist might then work on the same asset by accident, or the first artists' hard drive might crash and the source asset might get lost forever!  These problems don't happen often enough to be worth the cost of the "perfect build" - and they can be solved.  The first problem can be solved by setting up the asset control system intelligently - the second problem can be solved by local backups.

XNA's content manager is tolerable for us, because we only have 2000 files.  On my machine a nothing-changed build takes 15 seconds, which is fine (though I'd prefer 0 seconds), and less than our load times.  If we had 20,000 files it would be a problem.

So, next time you're bitching about Unreal's wad files - just remember the alternative is worse!

Now, I expect a lot of comments after this post, because this has been good holy war material in sweng-gamedev, so let's have them!

Comments

[alexjc]

What's wrong with having a simple script that runs on your machine at about 6 A.M. before you arrive, and builds all the different versions of the game you'll need during the day?

That alone would solve a majority of the problems you mention.

As for the other issues, it seems that too much emphasis is placed on the "build" process. Sure, you need it for self contained deployments, but for incremental development why not build the resources on the fly like id's Tech5 and other engines have done for years?

~ alexjc
AiGameDev.com

[Max Battcher]

It very obviously depends on the system you are building, as you point out. For small projects the XNA content pipeline more than suffices and the time for an incremental build isn't a large expense. For larger projects you probably want to start looking into more team-oriented build systems such as using a parallel/background build system such as a continuous integration engine like CruiseControl.NET. Getting a decent CruiseControl.NET set up for XNA projects shouldn't be too tough (thanks to MSBuild) and could quite help in larger projects. (My own so far are "small", so I haven't yet resorted to something like CruiseControl myself...)

But no approach is necessarily perfect. As with anything else there are drawbacks to any approach.

[Miguel]

OT: I haven't seen much activity on sweng-gamedev. Did something change?

[Jeremy Statz]

I'd argue that the "perfect world" build system you're describing is actually still pretty terrible. A perfect world content pipeline wouldn't have a pipeline at all, it'd be loading the content directly so there's no separation between source and game-ready asset.

Not going to work well in practice, unfortunately. Maybe it could be faked by having the game automatically compile/load from the source asset on startup if its time/date is newer than the existing compiled version?

All the pipelines I've worked with lately are in the "spend a half hour compiling in the morning" category, though at least we've gotten away from the nightmare hell that is expecting everybody to build their own EXEs. One of the biggest problems I've noticed with this sort of pipeline, is that the compiling step takes long enough that a lot of the artists simply won't run the game. They'll stick to their little test map or something and that's the end of it, because recompiling and running the production maps takes too long. I'd really like to see this go away, and I suspect next project with this sort of pipeline I'll put some substantial time into getting a cache server going, if we can't do anything better about it.

[Josh Szepietowski]

Variety is the spice of life.

One good option is to check both the source and processed assets into version control. Setup a build machine to regularly (constantly) build the incremental changes of the source assets. This machine should hold a lock on the final assets in the repository, so that the only way a final asset filters to the rest of the team is if the source asset was put into the repository.

On individual boxes allow to building of only the assets they are interested in (for fast artist iteration), and on every box allow a 'full incremental build' to take place (at an option).

Often programmers will be happy to plug away with just the final assets sent down from the build machine, and thats fine. Slowing their build process down by having them build the art assets is a mistake, IMHO, but disconnecting the 'source' from the 'final' asset is also a mistake.

This has been a theorypost, unfortunately, as I have not seen a build system setup this way just yet! :)

[Mat Noguchi]

If there were no bottlenecks the Perfect Build System would be the closest to ideal, since everything is described at the highest level of abstraction and you can generate the final build instantly.

Unfortunately, we have two main bottlenecks: I/O and runtime processing.

Any system that is designed ignoring those two bottlenecks is doomed to scaling hell.

MSN

[greggman]

Wow, I almost violently disagree. First off,

"But if that's the case, how come most of the succesful studios I know do it the "antipattern" way (anyone using Unreal with its wad files, for example) and the teams I've heard of with the big fat build systems are catastrophic?"

Hmmm, nearly every team I've worked on and every product I've shipped has done the big fat build. Our processes were closer to the one Josh mentions but you could type "build all" and get all the assets rebuilt from source. Having artist manually shepard stuff is the ultimate nono in my experience.

That said, I think we might be mixing ideas. The ideas above can be broken down

1) It should be possible to automatically rebuild all assets with one command from source materials

This is so you can make huge architechural changes in safety.

2) It should be possible to quickly build any single asset

Otherwise you have the long waits

3) When building a single asset, it should be possible for the system to automatically build everything else that asset needs (ie, if the Island level needs the palm tree asset then building the Island Level should also (optionally) build the palm tree if it needs to. If the palm tree needs the palm-leaf texture that should get converted.

Otherwise you have to hunt down the missing unbuilt assets

4) Assets should be built from source

Otherwise they get impossible to edit because all the stuff that made them easy to edit in the first place can get lost.(contraints, layers, IK rigs, expressions)

On top of that building from source means the source gets checked in. Which solves the "I lost the source" problem that here about on other teams.

5) updating a source assets should build any assets that depend on the fact that that asset changed.

Otherwise you either manually remember to rebuild those other assets which requires hunting things down.

None of this is that hard to setup and there are lots of easy ways to optimize, safe guard, etc.

Examples:

-Speed-

*) Building some assets (a level) can be slow so in our current system when any user builds an asset the command lines and source files (and tools) are MD5ed (MD5s are cached so they don't have to be re-MD5ed) and the MD5s compared. If they match then the baked file on the server is going to match what the user is trying to build and so it's copied off the server instead of built locally

-starting from source-

*) Exporting is easy to automate. Maya has mayabatch or you can write your exporter to run using OpenMaya as an executable. So there's no need to have artists manually export. Max has Maxscript which can be called on launch through the commandline so it can be automated as well.

-being flexible-

*) Version FOLDERS can be useful so that if a major format update happens you change the path the tools write to by version. So for example instead of storing the result in data/models/tree.bin it's in data//models/tree.bin. Then the code is also updated to load that version. The result is people who have not updated the code or tools still build to and run from the old folders. Programmers working on tools or artists working with them on new features can safely move ahead

-starting from source-

*) If Excel is easier for artists/designers to use for data entry it's trival to parse its XML. (see my website for examples). Its XML stores everything so there's no reason to use XLS files ever which means no manual steps for that data either. Exporting to CSV or running some macro will both end up which chances for error and other issues and should be avoided.

That's just off the top of my head but anyway, the point is that yes, if you have the system you described that you type "build all" every morning and then wait 1~2 hours then yea, that sucks. But only alternative is not "hand build everything and have no build all ability". There are plenty of other and *better* options.

[greggman]

I noticed that the browser or the blog software eat my version example. The second path should be

data/(version)/models/tree.bin

where (version) changes with each file format change

Oh, and as for Unreal people using the non big build. Every unreal project appears to be 6 to 12 months late using the non-big build. The projects I worked on shipped on time. (yea, I know that's an anecdotal point but then so is yours :-p)

[Jamie Fristrom]

That does sound really good...
I'm jealous.

[Jonny]

The main problem with from source asset build systems I have worked on is that there usually isn't any dependency management. Which means incremental builds are unsafe and thus full builds become the norm. I think that if you look at source compilation of code you can see that dependency management, when implemented well, is a real time saver. Just as with must code in games, the build system is just hacked on as an after thought.