Manager In A Strange Land: Fat Build Processes Redux

Gregg's comment was so good I'm going to escalate it.  He describes what is truly the perfect build system, an ideal we never reached on the Spider-Man team while I was at Treyarch.  And I'm jealous:

Wow, I almost violently disagree. First off,

"But if that's the case, how come most of the succesful studios I know do it the "antipattern" way (anyone using Unreal with its wad files, for example) and the teams I've heard of with the big fat build systems are catastrophic?"

Hmmm, nearly every team I've worked on and every product I've shipped has done the big fat build. Our processes were closer to the one Josh mentions but you could type "build all" and get all the assets rebuilt from source. Having artist manually shepard stuff is the ultimate nono in my experience.

That said, I think we might be mixing ideas. The ideas above can be broken down

1) It should be possible to automatically rebuild all assets with one command from source materials

This is so you can make huge architechural changes in safety.

(We could rebuild a lot, but not from the MAX files, from intermediate text files that were exported by the artist from MAX.  So we could make huge architectural changes to the renderer, but not to some of the other systems.-JF)

2) It should be possible to quickly build any single asset

Otherwise you have the long waits

(We had that. -JF)

3) When building a single asset, it should be possible for the system to automatically build everything else that asset needs (ie, if the Island level needs the palm tree asset then building the Island Level should also (optionally) build the palm tree if it needs to. If the palm tree needs the palm-leaf texture that should get converted.

Otherwise you have to hunt down the missing unbuilt assets

(We didn't have that but there weren't many dependencies - mostly just mesh -> texture, and it would load a placeholder texture when it couldn't find it.)

4) Assets should be built from source

Otherwise they get impossible to edit because all the stuff that made them easy to edit in the first place can get lost.(contraints, layers, IK rigs, expressions)

On top of that building from source means the source gets checked in. Which solves the "I lost the source" problem that here about on other teams.

(Yeah, artists could forget to check in MAX files.  Everyone had automatic backups.  What about PSDs?  Those can be expensive to check in. -JF)

5) updating a source assets should build any assets that depend on the fact that that asset changed.

Otherwise you either manually remember to rebuild those other assets which requires hunting things down.

None of this is that hard to setup and there are lots of easy ways to optimize, safe guard, etc.

Examples:

-Speed-

*) Building some assets (a level) can be slow so in our current system when any user builds an asset the command lines and source files (and tools) are MD5ed (MD5s are cached so they don't have to be re-MD5ed) and the MD5s compared. If they match then the baked file on the server is going to match what the user is trying to build and so it's copied off the server instead of built locally

-starting from source-

*) Exporting is easy to automate. Maya has mayabatch or you can write your exporter to run using OpenMaya as an executable. So there's no need to have artists manually export. Max has Maxscript which can be called on launch through the commandline so it can be automated as well.

(One of the reasons we didn't do this was because of the funky way we exported animations.  -JF)

-being flexible-

*) Version FOLDERS can be useful so that if a major format update happens you change the path the tools write to by version. So for example instead of storing the result in data/models/tree.bin it's in data/(version)/models/tree.bin where (version) changes with each file format change. Then the code is also updated to load that version. The result is people who have not updated the code or tools still build to and run from the old folders. Programmers working on tools or artists working with them on new features can safely move ahead

-starting from source-

*) If Excel is easier for artists/designers to use for data entry it's trival to parse its XML. (see my website for examples). Its XML stores everything so there's no reason to use XLS files ever which means no manual steps for that data either. Exporting to CSV or running some macro will both end up which chances for error and other issues and should be avoided.

That's just off the top of my head but anyway, the point is that yes, if you have the system you described that you type "build all" every morning and then wait 1~2 hours then yea, that sucks. But only alternative is not "hand build everything and have no build all ability". There are plenty of other and *better* options.

Oh, and as for Unreal people using the non big build. Every unreal project appears to be 6 to 12 months late using the non-big build. The projects I worked on shipped on time. (yea, I know that's an anecdotal point but then so is yours :-p)


Gregg's system is the best of both worlds.  You don't have to check every single dependency whenever you make any trivial change to the project (the way XNAGS is currently setup) -- I think we can all agree this is horrible and won't scale -- but you've got the big build which you can use to make burns and to validate source assets.

So here's a question:  do you build this elaborate system up front, or can you evolve up to it?  The Spider-Man team's build system was continually improving, and may have eventually arrived to Gregg's ideal after I left, I don't know.  (FWIW, Spider-Man 2 shipped on time with only a partially implemented "big build" as Gregg describes.)  Should we have halted production for a while to make it perfect?

 

Comments

[Michael]

Interesting topic, but this post is pretty hard for me to read, even though I did read the original article. (Not the comment, however...)

Could you add some color or italics to make it more clear which of the three sources a given sentence is from? (Original article, Gregg's comment, this article...)

(Love the blog!)

[Max Battcher]

Have you put much effort into attempting to the bend the Content Pipeline to your needs in this area? It seems to me that the Content Pipeline is extremely powerful and relatively easy to work with (define new importers/processors in the .NET language of your choice).

One solution I have seen is that a "level format" is used as indirection. The level format references individual art assets and art assets are checked into source control but not built directly by the Content Pipeline. Instead, when a level format is rebuilt by the Content Pipeline (in the case where a level is changed or a full rebuild done) the level format's Content Pipeline Importer/Processor then do the work to import the referenced art assets.

Plus if I remember correctly there are some "easy" ways to make smarter dependency graphs directly in MSBuild's XML if you don't mind hand-editing the .csproj generated by Visual Studio...

[Paul Du Bois]

Hey Jamie, gotta get home but thought I'd drop some notes on the asset build that we evolved into.

Artists check in DCC source (.ma files and such), and platform-neutral intermediate files (collada, uncompressed dds, and so on)

At first, the game read directly from the intermediate files. After a while this got kind of slow.

Then we added a way to serialize out the loaded assets into something easier/faster to load, but platform-specific. We'd cache these faster-loading files and use them if the timestamps looked OK.

Then we wrote a program (basically a command line version of the game) whose job was to create these cached platform-specific binary files, and wrote a primitive wrapper that loaded up intermediate assets to generate "cached" binary files, and checked them in.

Then we added dependency checking; whenever an intermediate asset is loaded, we keep track of what other assets are requested. The code that loads an animation collada file tries to load a bindpose, so we note that as a dependency. These auto-generated dependency files are saved and checked in. (Note that no programmer needs to do anything to specify a dependency besides use some data during asset processing; it works pretty well)

Recently we haven't had to touch it very much; mostly error reporting, figuring out who checked in an asset that fails to process and sending them an email, stuff like that. The next big push will probably be asset aggregation into wad files or whatever.

This is a pretty nice-to-use system. Artists don't need to shepherd anything through a content build, and they can see their changes without doing any building (since the content build code is all linked into the game executable). If we need to change some binary format, we build and check in a new version of the command-line asset builder (automated via ugly but functional web page) and kick off a data build (via same page).

Our dependency scanning (~18K asset files) ranges from 5 or so seconds to 2-3 minutes, depending on the state of the FS cache.

Again, this was all built slowly over time, so we both had stuff up and running quite quickly, and a lot of time to ponder our next moves (which was good, because some of the problems we ran into were subtle and required clever rather than simple solutions) I think the biggest PITA was our use of Collada (ick) but that's not really on topic.