GameDevBlog

Manager In A Strange Land: MS Project, How I Hate It

I used to love the idea of MS Project.

While working on Spider-Man 2, we'd often hear the same story:

"I was going to work on this, but I couldn't because such-and-such wasn't ready, so I tried to work on this other thing instead, but such-and-such wasn't ready, so I ended up working on item C."

I kept hearing those stories and thinking - we really should be using MS Project.  It tracks dependencies!  It will help prevent these sorts of problems!  It's the piece of the puzzle we're missing!  Once we start tracking dependencies, we'll be unstoppable!

So I looked into using Project, and some other scheduling software, and experimented with some dummy schedules and leveling tools and decided that not only did they not work very well, but we weren't ready to switch to a whole new system, not with Spidey 2 already more than halfway done.

Still, when Spidey 3 came along, I spent over a month near the beginning making a detailed schedule with MS Project.  We did a bunch of things right.  We asked the people doing the work how long the work would take (or their leads if those people weren't actually available yet).  We extracted our staffing plan from the schedule.  And we deputized some producers to run around and check to see if people were working on what the schedule said they'd be working on and to adjust it if necessary.

And after a month or two (it's been a while, I don't exactly remember) we scrapped it, because the schedule seemed completely unrelated to what people were actually doing in the offices.  They still weren't working on "what they were supposed to be working on", despite the wonderful dependency tracking that was supposed to take care of that.  Game development is still far too fluid and unpredictable, even when dependencies were being noted, we had problems with staffing and prototyping and changing plans that made dependencies pale in significance. 

Now, some people swear by MS Project.  I think they break down into a few categories:

*  People who use it but not to track dependencies.  For example, they might make a list of tasks, select them all and make one giant dependency chain, and then have a pretty (and completely linear) gannt chart they can show off, but under the hood this is no better than using excel and doing a Sum.  They might as well use Excel.

*  People who have modded the crap out of MS Project.  Mike McShaffry's tools are prety cool but they actually don't seem to use Project's ability to automatically track and level dependencies - as he points out at the bottom of the page, his tools are for tracking high-level dependencies, and even then you have to eyeball it:  oh look, Joe is starting a level before the World Editor is done. 

*  People who "do it right" but it still doesn't help them - Erik Bethke's book showed how they used Project and its leveling features for Black 9...a game that never shipped.  We can't blame Project for that but it does help to show that we have bigger things to worry about than tracking dependencies.

*  People who "do it right" and it does help them.  I did an informal poll of the room before my Theory of Drag talk last year and it looked like maybe 1 in 200 people used Project's leveling features and believed it helped.  Still they're spending an awful lot of resources on it - resources that could possibly be spent more directly on the game. 

Now, let's go back to this original "problem" we had that I thought MS Project would solve.  "I couldn't work on Item A or Item B, I had to work on Item C."

Is this really a problem?  It's only a problem if Item C is some low-priority item that we might want to cut from the game - chances are, all of these things need to be done and although it might be nice to work on Item A first it's just fine to hit Item C instead.

And if Item C really is optional, or someone really is blocked, there are lots of things you can do that will suck down less resources:

http://www.gamasutra.com/features/20041013/fristrom_01.shtml

http://www.gamasutra.com/view/feature/2170/manager_in_a_strange_land_.php

When I wrote those articles I was still trying to make Project work.  I have since given up - now I believe Excel beats Project, hands down.  (Microsoft still wins, of course...)

FWIW, dependencies weren't an issue on Schizoid at all - the biggest scheduling issue on Schizoid was that I was an overbooked resource, and this was a blindingly obvious problem that we didn't need any fancy scheduling software to see.

Manager In A Strange Land: Theory Of Drag 4

Hey, I'm back.  I'll have some news about Schizoid soon, and I have the energy to blog again.

A thing about drag.  I forgot one of the most important ways to mitigate drag, which is:  Don't Start From Scratch.  Almost nobody does, anymore - one of the first questions you ask yourself when beginning a game project is "What engine am I going to use?"  The main reason to do this is because it saves you a lot of work.  But another reason it's helpful is because you schedule more accurately - to keep using the snowball metaphor, the snowball you're rolling uphill is already nice and big:  progress starts slow and doesn't get that much slower.  In fact, a question I was asked at my IGDA talk went along these lines:  "We just do content drops into an existing engine - does this really apply to us?"  I fumbled the question at the talk, but now I'd agree and say, yes, although there will always be at least a little increase in drag as a project evolves, it may be so little as to be immaterial.

And I said I was going to talk about measuring velocity.  But it's nontrivial, and I don't really have a good way to do it.  I've been looking at this old progress graph:

And, if you're at any given point on that graph, what do you use for velocity?  The ideal is to find a velocity that will accurately predict when you'll be at at "zero bugs."  Here are some options:

- Use when you started the project as a starting point, and your current point as the end point:  this always gives you an overly optimistic estimate.

- Use your instantaneous velocity.  This gives you a wildly fluctuating estimate.  One day, you're going to finish tomorrow.  The next day (after a slip), you're never going to finish.

- Use some amount of history.  But how much?  Something that would have been fairly good for us, though still too optimistic, would be to use the last half of the project.

I tried a bunch of stuff - moving averages, fitting curves, nothing hit.  But my math skills in this area are weak.  If someone out there can take a graph like the one above and find a good approximating/predicting curve, please let me know how.  I did ask my brother-in-law, a financial analyst, if he could use stock market momentum measuring tricks to do it, and he wasn't able to help.

In the end, my best guide was actually my eyeballs.  When we were in December, and hoping we'd be finished in January, I was able to say - "Just look at the graph.  We're looking at the end of March."

That turned out to be on the money - the first time we hit the zero bug line.  Of course, we still weren't finished yet;  a little more testing and we were right back in the red again. 

One must imagine Sisyphus happy.

Manager In A Strange Land: Theory of Drag Part 3

So, yeah, hey, credit where credit's due, Clinton Keith's post was what first got me thinking about this - you can even see my comment there in his post, when Schizoid was young but we were already seeing diminishing returns.

So, what do we do about drag and diminished returns?

1) Mitigate

One reaction is - "Let's get rid of it!"  Particularly if we use the tractor-pull metaphor;  let's get out of the tractor, clean everything up, and get ready for another pull.  That idea doesn't fit so well with the snowball-up-a-hill metaphor, though.  You've been working hard to collect that snow, you're not going to get rid of it just so you can push faster. 

But still, yes, good idea - let's periodically refactor and pay down our technical debt.

That'll help mitigate the drag somewhat, but it's not going to solve it.

(Another way to mitigate is to bring back waterfall - bigger designs up-front, more rigorous planning up-front.  You'll have a less werewolfy curve, because you're A) going slow at first while you do all that planning and B) probably reduce the discovered work.  The disadvantages in waste on speculative generality and building the product-you-thought-you-wanted-but-didn't-really-want probably aren't worth it, though, so I'm not recommending this.)

2) Bring Back The Contingency Buffer

Nothing new here.  Goldratt in *Critical Chain* spends hundreds of easy-to-read pages explaining why this is a good thing, and he's echoed much more tersely by Tom DeMarco in *Slack.* 

Their message:  "Don't adjust your estimates by a fudge factor.  Instead, use that fudge factor to create a contingency buffer or pad at the end of the project."  Because when you set pessimistic goals, people take their time.  I'm not a fan of Parkinsonian scheduling (setting unrealistically tight dates) but I'm also not a fan of unrealistically loose dates either.  My favorite thing about the contingency buffer is it helps keep scope in check.  "Sure, you think you can get it done before our ship date, but can you get it done before we hit alpha?"

Speaking of alpha, game development typically already has a kind of contingency buffer built in, which we call "alpha" or "beta" or "content complete" depending on where you work.  The point where you stop "adding features" and just fix bugs or maybe polish.  (Of course, the area between bug and feature is a gray one.  (The dial is analog, was how I heard Mike McShaffry put it once.))

I have problems with alpha.  I have problems with it being defined differently everywhere you go.  I have problems with the definition frequently being murky even within a team: ask different guys on your team what alpha means and you might get different answers. 

Another problem I have with alpha is it has become too small in this day and age.  Back when I did six month projects, we tried to stop adding features a month before we shipped.  Now that we're doing three year projects, if we're being proportionate, we'd save six months - but I don't know anyone who does.

But my biggest problem with alpha is it encourages sloppiness - an "add features now, we'll fix the bugs later, in alpha" attitude.  Steve Maguire, in *Debugging The Development Process* goes to great lengths to say why that's Bad.

But one thing he doesn't say is, when alpha is treated that way, it is no longer a contingency buffer.  For it to be a contingency buffer, your goal has to be to be done by then.  Not done-except-for-bugs done.  Zero-bugs-done. Done-done-put-it-in-the-box-done. 

Schwaber doesn't really do a contingency buffer, either.  On the release burndown chart, each estimate is multiplied by a fudge factor.  Just what Goldratt says not to do.  So what happens?  At the beginning of the project, when velocity is high, those fudge factors are canceled out by your high velocity.  It looks like you're going to eat through that burndown chart in no time, and Schwaber's method predicts an optimistic release date.  Then as velocity decreases the date continually slips back.

So here's my modification to Schwaber.  Track your backlog just like Schwaber suggests.  (And include all bugs that you intend to fix in your backlog - at Torpex, we use the rule of thumb that 3 bugs equals one story point.  YMMV.)  But rather than multiplying all the estimates by a fudge factor, add a contingency buffer at the end.

At the beginning of the project, when you have no idea what your progress curve is going to look like, you could use Schwaber's "complexity assessment" to guess how big the contingency buffer should be.  I could see it going anywhere from 20% to a well-understood-requirements simple-technology project all the way to 300% for the chaos of poorly-understood-requirements and crazy tech.  (Schizoid needed at least a 150% contingency buffer, we know now.)

A problem with the contingency buffer at the end is - if you make your scheduling process visible to your publisher, which most say is a Good Thing - and your publisher sees your large contingency buffer, they're going to ask, "What the hell?  Looks like you're billing us for a game much larger than the one you've scheduled!"

So, for the schedule you show your publisher, keep the fudge factors in the original estimates rather than in a large pad at the end.  I said this at my talk and Ben Hoyt called me out - "Isn't that dishonest?" he said.  At the time I said, "Yes, I suppose," because I tend to cave during Q&A.  But in hindsight I don't think it is, really, dishonest - the fudged estimates are your "because this is a project of significant complexity" estimates, and I'd be perfectly willing to share that with a publisher.  I'd also be perfectly willing to say that, internally, we have these aggressive goals we're striving to hit, so for internal purposes we don't actually multiply in the fudge factors, but rather leave them as part of a large contingency buffer at the end, but those aggressive goals are not ones we want to be contractually bound to.  Trent Oster, in his talk, said similar things -  he also said to double your estimates and apparently at Bioware they try to work well ahead of the milestones they've promised their publisher as well.

Now, suppose you're partway into the project, and you're experiencing drag, and you realize even you underestimated your contingency buffer - you're going to sail right past that final deadline with an unfinished game.  At that point, a couple things need to happen:

- You have to have that hard conversation with your boss / publisher.  The "Please don't sue us for breach, but..." conversation.

- Add a bigger contingency buffer to your next estimate.  It's tempting to assume that everything that can go wrong already has and it's going to be smooth sailing for the rest of the way, but then you'll just have to have that hard conversation again with the next slip.

We're trying to approximate this curve:

With these:

On Schizoid, at first, our trajectory had us finishing in April, but we told Microsoft June.  Call that our 50% contingency buffer.  After our first slip, it looked like we could still get it in under the wire.  Then we started optimizing and we slipped again.  Our contingency buffer is used up.  But it was just a blip, right?  We'll start making progress again any day now!  No.  The blip dragged out and became a plateau. 

If we had been honest with ourselves at that point, say, by plotting out where our new trajectory was going to hit and adding a larger buffer to that, we would have ended up with a much more realistic estimate about when we'd be done.  An incredibly disheartening estimate, but realistic. 

And still wrong - here we are, almost February of next year, and still not done yet!  But it would have been much less wrong than what we were going with back then.

Sadly, "Less Wrong" is all I can offer. 

Next time (notice I didn't say next week - we just did more focus testing and bug triage and we can't bring ourselves to mark as much stuff "Will Not Fix" as maybe we should and now blogging is starting to feel like a luxury):  Just how do you calculate "velocity" anyway?

Manager In A Strange Land: Theory of Drag 2.5

Before I go on to Part 3 and what we can do about drag let me deal with some of the discovered work unearthed by the comments on Part 2.

* Graph Quality

Mark Nau calls me out and says he doesn't see the curves.  If we still worked in the same office I probably would ask him to help me do an actual rigorous statistical analysis of the data.  The curves are there but they look shallow and I admit I was pretty disappointed after generating the graphs that they weren't more obvious.

As for not seeing much of a bump when we brought the new programmer on, I can explain that.  He's only part-time - I'd estimate at him increasing our manpower by 25% or so.  Factor in Fristrom's Law and it's less.  He needed a ramp-up period.  And his first task was huge but greatly underestimated.  It could have shown on the graph as a simultaneous increase in scope and work, but instead it shows on the graph as an increase in neither.

But that goes to prove Mark's point - the graph isn't accurate.  I'm not able to truly tease apart work and scope, and probably won't even bother on future projects.  (Though it was nice to see that we actually have been working all these months, something that isn't obvious from the first graph.)

Still, it's not two straight lines.  Keep in mind that first graph:

This graph is simply the delta between the two lines in the other graph - though they don't plot out to the same point in time;  I made the graphs at different times.  If the lines in the other graph were straight, this graph's lines would be straight as well, but this is not a straight line.

* Other curves?

Jake Simpson and Simon Cooke say they see different shaped curves on their projects - they both see sine waves or something approximating sine waves.  It is possible on this project we're just looking at a portion of a big sine wave with a really large period.  Which would be a good thing, because that means at some point we'll take off like a rocket and ship!  I wouldn't be surprised, actually, if we do get a burst in the end, as we put our foot down about feature creep and the remaining bugs are all highly detailed spot fixes.  If only there was some way to predict when that change would come.

Cooke sees the opposite from us, in a way:  he sees little visual progress in the beginning, during ramp-up, later followed by a lot of visual progress.  (Much like our programmer with his big, underestimated task.)  He may be more waterfallish than we are - we tend to get something up-and-running *now* and refactor later.  For us, the "up and running now" shows up as a lot of progress, and then the refactoring shows up as drag.  (Maybe this agile stuff isn't all it's cracked up to be.)

* Feature Creep vs. Discovered Work?

How much scope increase was stuff we wanted vs. stuff we needed?

FWIW.  Red & most of green are needed.  Blue, yellow, and pink are wanted.  Call it 60-40.

Though I hesitate to call that 40% literal 'feature creep' - most of our unnecessary fixes were not "add X" but "X would be better if you do Y".  The feature was already there and we were tweaking it.

More interesting would be to see how this changes over time, but I didn't know until late in the project that you could set up Bugzilla to generate snapshot data that you can later graph.  It would be fraught with inaccuracy anyway - at the beginning of the project, p3 meant must-fix and later came to mean nice-to-fix.

I'm guessing Busse's point:  if you tighten the reins on feature creep, you can mitigate the effects of drag.

I'll try to get back on track next week, unless there are more interesting comments.

Manager In A Strange Land: Theory Of Drag, Part 2

Part 1

So where does this long tail come from?  Why doesn't velocity average out so that after you've been working on your game for a couple of months you've got something you can dead-reckon with?

Some would say it's because of lazy developers.  We set a goal - we realize we're going to miss the goal - so we set an easier goal - and then we slack off since we've set that easier goal - and then, before we know it, we're going to miss that goal, too - etcetera.  Parkinson's Postulate.  Freshman Syndrome.  I'm not a big fan of this theory.

Another cause is plain old underestimation.  Discovered work, for one thing. "We just forgot we'd have to do that", etcetera.  But - shouldn't Discovered Work average out after a while?  Maybe not in two months, but after several months shouldn't you have a pretty good idea of the ratio of discovered work to originally planned work?  No.  The further in the future the work is, the murkier it is, so you're going to tend to see more and more discovered work as you get towards the end of the project.  In a way, your post-alpha bug list is *all* discovered work.

Underestimation can show up on the schedule graph as peaks ("We forgot we had to this!  Better add it to the schedule!") and plateaus ("You're *still* working on that?").  I can look at the peaks on the schedule graph for Schizoid and say, "That was when we realized we'd have to multithread";   "That was when we started working on our certification requirements and discovered just how hellish they are this time around".  [Side note:  Oh, Mr. Allard, you promised!  You promised it would be easier this time!]  [Side side note:  for those of you doing 360 development, don't think of that 130 line item TCR list as the real list.  Go straight to the test-cases list.] and "That was when we entered QA."

A pernicious kind of underestimation is technical debt - when you think you've finished a feature but there are still lingering issues in it that get caught later.  This shows up on the schedule as good velocity now for a creeping increase in bug count later.

Underestimation is most of the story.  A lot of people think it's the whole story - that's part of the waterfall "gather *all* requirements and outlaw feature creep" thing.  "If only we knew what the client really wanted / hadn't changed our minds / hadn't forgotten x,y, and z, we'd be done on time!"

To get the rest of the story, we need to split the graph into two:  a graph of scope increasing and a graph of work done.  Schwaber's burndown charts only deal with 'work remaining', and that's what we used for Schizoid, so it took me a while to massage our data to show both:

The green line is scope - as we discover things we forget (we need to multithread;  the designers want the macro game to work differently;  we should have known we'd have to do *that* for TCR) and as bugs get discovered the green line goes up. 

The red line is work done:  features completed and bugs fixed.

Visually it looks like the green line is curving one way (the rate of discovering work increases as we go) and the red line is curving the other (the rate of work done is decreasing.) 

Something the graph doesn't show is we added a programmer halfway through.  You'd think we'd see a bump in the red line there but there isn't much of a sign.  So even with the extra manpower we're getting less work done.  (You might think this means he's actually slowing us down but trust me, he's not; I don't know where we'd be without him.)

And that bowing of the red line, despite having that extra programmer, is the evidence of drag.  Patrick Hughes predicted where I was going with this in the last post - as the system gets larger, your efficiency working on it decreases. 

It's like a tractor pull - or perhaps a better metaphor would be pushing a snowball up a hill. 

We see this in a variety of ways when working on videogames:

* The components of the system have to play nice together.  As you add components, each future component becomes that much harder to create, because it has to work with all the previous components.  Boehm's cost of change curve is supposedly for unplanned features - I believe that it applies to all features whether they're planned or not!  I'm too lazy to graph our lines-of-code against time but you know what we'd see - the same bowed curve - which doesn't necessarily mean we're doing less work but is a good sign. 

* As the game gets bigger turnaround times increase.  It takes longer to build, and to run, and to load a level.  It takes longer to play through a level to get to that point where the bug is.  (You can implement a cheat that lets you zip straight to that point;  or write a custom test that lets you execute the code without running the whole project;  but doing that takes time, too.)

* QA gets tougher on the larger system.  Finding and reproducing bugs becomes more and more difficult as you get fewer needles in a larger haystack.

So that's Drag.  Common sense, you're probably thinking.  But if it's common sense, why do we schedule as if it doesn't exist?

Why does Mike Cohn, in *Agile Estimation and Planning*, say we can use our velocity for long-term estimation?

Why does Mark Cerny say we can use what we've learned prototyping to accurately estimate how many levels we'll be able to make in production?  (Not to dis on Cerny, his Method is awesome and I wish it hadn't been seemingly overshadowed by Scrum.)

Why does Joel Spolsky say you can simply compare estimates to actuals to adjust future estimates?  (BTW - *something* must have been wrong with Spolsky's old system, otherwise why would he have felt the need to create a new one?  But the new one doesn't solve the problem.)

We're all using linear math to deal with this nonlinear problem.

Credit to Ken Schwaber - he never said his methods could be used to make an accurate long-term estimate...he just cops out and says you can't predict anything more than a couple months away.  Not too useful for a studio that might like to, say, sign a contract promising a certain delivery date.  We want to be able to hit goals and keep promises!

Coming up:  we're basically screwed, but here's some things we can do to mitigate the damage.

Manager In A Strange Land: Theory of Drag, Part 1

Finally getting started on this, the text version of my talk at the IGDA leadership forum.  Just got my talk reviews back and there's a Love It Or Hate It thing going on.  Some of the Haters were particularly brutal, making the trolls who every now and then appear in the comments on the blog look like gentle friendly neighbors, and making me wonder if it really is "better to be ridiculed than ignored."  But at least they've gotten me motivated to get this stuff written down where I can debate it with text rather than voice.  Much more comfortable with text.

Schizoid is the first project where I've used the velocity-tracking scheduling method from Scrum / XP / Cohn's *Agile Estimating And Planning* - and, on one hand, I've got to say I have never so dramatically underestimated a project.  On the other hand, seeing our progress, from the beginning of the project, as a graph of work-remaining-to-be-done over time, and using that to estimate our velocity, has crystallized something for me.

The graph looks like this. 

This graph looks familiar - it looks a lot like the "bugs remamining" graph of a project that's well into QA.  And if you stare at it until your eyes glaze over, which I have done many times, you see a trendline that looks something like this:

The curve is also familiar.  It looks a lot like that long-tailed power law curve.  (We're not actually on a power law, thank God, otherwise we'd never cross zero, but it's a lot like that.)

And suddenly I realize - if I had used this method to track my many previous projects, I would almost always have seen this curve.  (Tony Hawk 2 for the Dreamcast the one exception, the only project I've scheduled that was a slam dunk, hit the scope we promised, hit the date we promised, hit the budget we promised, 90+ on gamerankings miracle.)

Because on all of these projects, we'd slip, and we'd push our ship date back proportional to the slip.  (We aren't committing that classic rookie mistake of sliding the ship date back one week for a one week slip - the ship date moves back proportionally.)  And yet - we slip again. And again.  How can this happen?  If we've factored in our velocity from those first slips, we should be golden.  Unless...

Velocity ain't constant.

This should have been obvious.  Every project I've worked on seemed like it was going great at first - and then:

"This isn't going great anymore.  We should stop adding features."

"This is going even worse.  We should cut features."

"This is going still worse.  We should transfer people from other teams."

And, finally - "We better crunch."

It's Fred Brooks' werewolf - project seems fine at first, turns into a monster at the end - the one against which there is no silver bullet.  But unlike a werewolf, the project doesn't turn on you overnight - it gets hairier and hairier as you go along.  Like Brooks says - "How does a project end up a year late?  One day at a time."

Once I discovered this I talked to a few others about it, and not just in our industry - this "diminishing returns" curve seems to be everywhere - it's not just because I shouldn't be scheduling.

So, how about your projects?  Is it only after you're well into production that you realize you're in trouble?  Do plateaus and long tails sound familiar?

Next post I'll talk about some of the causes of these "diminishing returns" projects, one of which is drag.

How Much Planning - High Level Concepts?

Hmm, not many comments on that last post.  Guess it was just Too Damn Long.

Have a long build right now, so I'll respond to a comment.  Pag asks:

Don't you need planning for the higher level ideas though? Iterations work great for low-level features that can be implemented and tweaked relatively easily: controls, enemy AI, etc. But you can't really iterate on high-level concepts. Sid Meier's dinosaurs game died after a while and as I understand it, they still weren't sure if the game would be turn-based or real-time -- that can't have helped production.

Basically, how do you balance the need for a clear overarching vision and the need for flexibility during development?

Don't know anything about this dinosaurs game, but I'd think turn-based vs. real-time would be something you'd nail down in prototyping early on, while there's only a handful of guys on the team and your "dinosaurs" are still flat-shaded boxes - or maybe it's even in 2D and the dinosaurs are just X's and O's - or maybe you even make a board game and move toy dinosaurs around a hex map, faking the "realtime" somehow.  At this stage it's totally possible to iterate on various kinds of turn-based and realtime mechanics.  And at some point you make an informed decision about which you like best and prototype the next thing and eventually go into production.

The "more planning" alternative would be to have meetings, before any code is written (or boardgame mock-ups are created) to discuss why the game should be realtime or turn-based.  You can involve all the stakeholders.  Maybe publishers don't want the game to be turn-based for marketing reasons.  Maybe some of the designers are worried that griefers will ruin turn-based network play, or that it will be boring waiting for your turn.  And you write a document that says the game will be real-time, damn it, no matter what.  And you'll make that game.  Nevermind if you had a new take on turn-based that could have been really cool.  On the bright side, this process probably takes less time than iteration on that turn-based mockup.

This was a hypothetical example, but my real-life experience is that planning is the enemy of creativity.  None of my cool ideas would have gotten done if I'd had meetings discussing whether they were good ideas or not first.  (But maybe I'm just a bad salesman.) 

Oops, build finished a while ago, and I didn't really get to "vision."  Ok - started another build.

What is this Vision thing, anyway?  (An album by Sisters of Mercy, ha-ha.) Real-time vs. turn-based isn't necessarily under the umbrella of Vision.  It's quite possible whoever "owns" this game (whether it's the producer, director, designer, or--my favorite--a hive mind) couldn't care less whether it's real-time or turn-based, as long as whatever they end up with is fun.  Maybe all they care about are the dinosaurs.

On the other hand, maybe from the start, they had a cool idea for a turn-based game with dinosaurs.  "This game is all about turn-based dinosaurs!"  (For whatever reason.)  Well, that's just the box within which the team works and iterates, right?  And yeah, "turn-based" should be in the doc, with all the reasons why, and what that means to you - do I move all my units, then you move all of yours?  We switch-off?  You move while I fire?  Time limits?  Chess clock?  Some of this you care about, it's essential to your vision -  ("Absolutely you only move one dinosaur at a time.  That's final.")  And everything else you iterate on.

Where you get into problems is when the team doesn't share the vision.  Does more planning help here?  Maybe - you can hear them raise objections and maybe sell them on the vision or at least say, 'duly noted' and pull rank.  Does a doc help?  Maybe, if you can get them to read it.  I'll say I've definitely been there, and seen parts of games that I had strong feelings about go in directions I thought were simply wrong.  But the problem wasn't lack of planning or documentation - the problem was simply that I caved.

Note to self:  my builds really aren't long enough to work on blog posts.

And I don't know what happened with the formatting there...too lazy to edit the html.

Manager In A Strange Land: How Much Planning?

(For anyone awaiting my article on "Drag" and wondering why I took the time to write this instead...this is actually an article I wrote a while ago and never got around to posting.)

"A good plan, violently executed now, is better than a perfect plan next week." - Patton

“Plans are worthless, planning is indispensable.” — Eisenhower

Are these two quotes contradictory?  Eisenhower seems to want to continue planning (even though he knows he'll throw that plan away later) whereas Patton wants to quit planning today and do something.  When you think about the quotes long enough they become tautological.  "We should plan until our plan is good enough."  Duh.  But how long does it take to get to a good enough plan?  How do we recognize a good enough plan?  To be more precise, where's the crossover?  How do you know if that extra hour spent planning is going to save you an hour down the road?

Back in the waterfall days, we'd spend the first milestone on "the plan" - and the outcome of that would be the rest of our milestones.  (And we thought we had it good because it meant we didn't have to actually do anything real for the first milestone.)  It was actually pretty easy to justify this to the publisher.  (And it didn't seem to matter if it was a nine month project or an eighteen month one--never tried it for a twenty-four month or longer project--you'd get that first month to plan.)  Some of the least succesful projects I've worked on resulted from this:  Draconus and a Nascar racing game that never saw the light of day, to name two.  So, from my anecdotal evidence, a month of planning without coding is the wrong amount. 

Did we need more or less? 

The successful projects (these projects were only qualified successes, to be sure - I've only worked on one project that was an unqualified success - hit time, budget, scope, and 90%+ on gamerankings) I've worked on had a lot less.  Typically we couldn't wait to get started coding, and documents were often an afterthought to either explain what we'd already done or talk about where we'd go from there.

In *Agile Software Development With Scrum* they talk about how Scrum can be seen as a Kuhnian phenomenon.  I haven't the foggiest what they're talking about, but I do have a very vague, layman's knowledge of the Hegelian dialectic.  (Recently refreshed from watching the movie *Half Nelson*.)

People typically think of agile and Scrum as being the antithesis of Waterfall.  I disagree.  Rather, Waterfall is the antithesis of "dive in and code".  The thesis:  we should dive in there and start coding!  The antithesis:  we should plan every detail - we need to gather requirements and model in UML and flow charts and write all of our header files before we write the actual .c files and blah blah blah.  Synthesis:  Scrum - we should spend *two days* (one day on the project, one day on the first iteration) planning the project.  From then on, we should spend one or two days planning for each eighteen - nineteen coding.  (And on a project with more than ten people, we should spend one day planning the macroproject and then each subteam should spend a day on their dish.)  And we may want to stop and do a little side-planning here or there along the way - we'll see.  A plan, as far as Scrum is concerned, is a list of requirements with somewhere from a sentence to a paragraph describing each one - they fit on file cards.  Accompanied by wild-ass-guesses for how long each of these things are going to take.  And we'll probably forget a bunch, and that's okay.

Compared to some projects that consider themselves agile, that's actually a lot of planning.  To be honest, that is more planning than has gone into some of my "successful" projects.  Maybe those successful projects could have been even more successful.

It looks like Clinton Keith wants more planning these days.  What he proposes isn't a Gantt chart per se but it's damn close. I'm not crazy about Gantt charts. (Check it out; I spelled it right this time.) It looks to me like he's feeling the pull to Plan More.

And speaking of arguing for More Planning, we have Joel Spolsky. 

But I'm going to put my foot down and say, "No.  Enough planning."  I think Clinton's hearing the siren song of the Gantt chart because game development projects always suck, and you look for causes, and one cause, if you're not doing much planning, might be "we're not planning enough." Post hoc ergo propter hoc. But like Ken Schwaber says, you quickly hit very diminishing returns when you spend more time scheduling/planning - and we should just accept that we're going to be wildly inaccurate and get on with it.

As for Joel Spolsky, maybe spending two months designing makes sense for products like FogBugz and CityDesk...because iterating with on-paper design with those projects actually might make the designs better.  But, since games are more complex systems, and systems are notoriously unpredictable even to the seasoned designer, if you spend two months designing your game while I spend one day designing out of every ten coding, my final design is going to be six iterations better than my first design but your final design - well, it's practically a coin toss whether it's going to be better or worse than your first design.

Let's take some of the games I've worked on...

Die By The Sword - very little planning, though what first got me onboard with the project was seeing a map they'd made on paper for one of the combat arenas; back then I was sick of the cowboy mentality and wanted to see more BDUF;  of course, we rebooted the project partway through and turned it from a fighting game to an explore-a-dungeon-beating-things-up-along-the-way game, and that arena got thrown out.  In a way, though, we stumbled upon Cerny's method - taking almost a year to make that awesome first level, our "prototype / vertical-slice" - and doing lots of kleenex testing on the way there.  It slipped twice but was a great game.  (Sure, only 75 on gamerankings, but I'm convinced it would be higher if gamerankings existed when we shipped.  5 stars in Next Gen, for example!)

Draconus - Draconus was the closest I've come in my career to doing full waterfall.  We had a giant whiteboard in Pete's room with our UML diagrams.  We had big design documents that we argued about in long meetings.  We did research on state-of-the-art approaches before writing our highly abstract code.  Also one of the most troubled games of my career - the original publisher dropped it and another publisher picked it up but only wanted to give us nine months to finish - cancel the network play and the this and the that and the other thing, but do it for the Dreamcast instead of the PC.  All that BDUF thrown out. We were six months late, and only got 70 on gamerankings.

So, for Spider-Man 1, we backflipped.  Just get in there and make some levels, damn it!  More planning would have helped.

Spider-Man 2 had almost the right amount.  Could have used just a tad more.  In the beginning, we'd spend mornings designing while we spent afternoons coding - but it wasn't long before we were sketching a level on a napkin and then spending a month or two of implementation to make it as good as we could.  20 page design doc;  continually evolving table of what all the controls on the gamepad did;  map of the city...

And Spider-Man 3 I think had too much, though that's supposition because I wasn't around for most of the project.  Although it wasn't even close to full waterfall, in the early days we did a sort of waterfall with the individual levels - first described in text by the designer and then isometric concept art made by the concept artist and then a meeting in front of a lot of people where the designer presents and defends the design - and we'd typically find that the level they drew on paper was the level that got made and it would get kinda stuck that way.  One of the reasons we were doing more planning was because a cross-studio Activision consultant came by with the level design docs from Infinity Ward and said, "Look, here's how our best studio makes their game.  Why don't you do it like this?"  So we did.  Later I mentioned that we borrowed the "Infinity Ward method" to one of the Infinity Ward guys and he said, yeah, they don't do it that way anymore, because the junior designers would would keep making crappy levels and then the senior designers would ask why they weren't fun and the junior designers would say they just implemented what was in the doc.  (This was a couple years ago, so I don't know what Infinity Ward does now or did for CoD4.)

That's a key thing with game levels - you don't know how fun they are until you build them.  And if you make your sketch too detailed, if you make it more of a blueprint than a sketch, people are going to model that detail right into the level and there's going to be resistance to changing it later.

So - how much planning?  Here's what I think:

Small team, downloadable game, no need to hit a specific ship date:  almost none.

Console action game:  20-page design doc;  story outline first - script and game evolve simultaneously;  levels sketched (but not blueprinted!) right before they're built.  In the early days, split your time between planning and implementing.  In the later days, fall back to one day of planning for every ten - twenty implementing.

RPG:  more, but I don't know how much more, because I haven't made one since 1993.

MMORPG:  still more, but again I don't know.  Damion Schubert's the guy to go to for this.

Another Writeup; Heisenbug

If you can't wait for me to write it up, Tim Longo did a writeup of my talk for the forum that covers some bits the Gama writeup missed.  And all the other talks are written up as well.

In other news, I was crowing about Schizoid's mean-time-to-failure a while back.  But for the past few weeks it's been horrible!  We had a nasty heisenbug off the main thread, and it only showed up every five-to-twenty hours of soaking.  Every night I was adding more instrumentation (thank you for that cool-sounding term for printfs, Billy Zelsnack, and thanks to Brett Douville for telling me to man up and pull out the printfs instead of killing myself - I tell you, I've gotten so used to our modern tools that I forget how we used to have to debug in the old days), soaking, and slowly narrowing it down.  Finally got it, right before the show - (the collection of circumstances that had to combine to make it happen were staggering though at the bottom of it all there was some bad code by me) - and I'm now happy to say there are no known failures in the soak.  It is possibly the worst bug I've ever tracked down.

IGDA forum fallout

Chris Avellone put the cartoons he did for Bill's talk up on his blog.  The audience was in stitches.

Good write-up on my talk at gamasutra - I've seen a talk write-up or two where I wondered if I was even at the same talk as the person writing it up, so quite pleased at this one's accuracy.  I'd e-mail the author and give him some props but can't seem to dig up his address.  (Granted, I didn't try *that* hard.)

I will be far away from my dev kits over Thanksgiving, so that should give me time to write up a more formal paper about the whole thing.  Call me arrogant, but I really think I'm on to something with this "Theory Of Drag" I've concocted...maybe some obscure project management literature covers it but I haven't seen it in any of the stuff I've read.