Manager In A Strange Land: Theory of Drag Part 3

So, yeah, hey, credit where credit's due, Clinton Keith's post was what first got me thinking about this - you can even see my comment there in his post, when Schizoid was young but we were already seeing diminishing returns.

So, what do we do about drag and diminished returns?

1) Mitigate

One reaction is - "Let's get rid of it!"  Particularly if we use the tractor-pull metaphor;  let's get out of the tractor, clean everything up, and get ready for another pull.  That idea doesn't fit so well with the snowball-up-a-hill metaphor, though.  You've been working hard to collect that snow, you're not going to get rid of it just so you can push faster. 

But still, yes, good idea - let's periodically refactor and pay down our technical debt.

That'll help mitigate the drag somewhat, but it's not going to solve it.

(Another way to mitigate is to bring back waterfall - bigger designs up-front, more rigorous planning up-front.  You'll have a less werewolfy curve, because you're A) going slow at first while you do all that planning and B) probably reduce the discovered work.  The disadvantages in waste on speculative generality and building the product-you-thought-you-wanted-but-didn't-really-want probably aren't worth it, though, so I'm not recommending this.)

2) Bring Back The Contingency Buffer

Nothing new here.  Goldratt in *Critical Chain* spends hundreds of easy-to-read pages explaining why this is a good thing, and he's echoed much more tersely by Tom DeMarco in *Slack.* 

Their message:  "Don't adjust your estimates by a fudge factor.  Instead, use that fudge factor to create a contingency buffer or pad at the end of the project."  Because when you set pessimistic goals, people take their time.  I'm not a fan of Parkinsonian scheduling (setting unrealistically tight dates) but I'm also not a fan of unrealistically loose dates either.  My favorite thing about the contingency buffer is it helps keep scope in check.  "Sure, you think you can get it done before our ship date, but can you get it done before we hit alpha?"

Speaking of alpha, game development typically already has a kind of contingency buffer built in, which we call "alpha" or "beta" or "content complete" depending on where you work.  The point where you stop "adding features" and just fix bugs or maybe polish.  (Of course, the area between bug and feature is a gray one.  (The dial is analog, was how I heard Mike McShaffry put it once.))

I have problems with alpha.  I have problems with it being defined differently everywhere you go.  I have problems with the definition frequently being murky even within a team: ask different guys on your team what alpha means and you might get different answers. 

Another problem I have with alpha is it has become too small in this day and age.  Back when I did six month projects, we tried to stop adding features a month before we shipped.  Now that we're doing three year projects, if we're being proportionate, we'd save six months - but I don't know anyone who does.

But my biggest problem with alpha is it encourages sloppiness - an "add features now, we'll fix the bugs later, in alpha" attitude.  Steve Maguire, in *Debugging The Development Process* goes to great lengths to say why that's Bad.

But one thing he doesn't say is, when alpha is treated that way, it is no longer a contingency buffer.  For it to be a contingency buffer, your goal has to be to be done by then.  Not done-except-for-bugs done.  Zero-bugs-done. Done-done-put-it-in-the-box-done. 

Schwaber doesn't really do a contingency buffer, either.  On the release burndown chart, each estimate is multiplied by a fudge factor.  Just what Goldratt says not to do.  So what happens?  At the beginning of the project, when velocity is high, those fudge factors are canceled out by your high velocity.  It looks like you're going to eat through that burndown chart in no time, and Schwaber's method predicts an optimistic release date.  Then as velocity decreases the date continually slips back.

So here's my modification to Schwaber.  Track your backlog just like Schwaber suggests.  (And include all bugs that you intend to fix in your backlog - at Torpex, we use the rule of thumb that 3 bugs equals one story point.  YMMV.)  But rather than multiplying all the estimates by a fudge factor, add a contingency buffer at the end.

At the beginning of the project, when you have no idea what your progress curve is going to look like, you could use Schwaber's "complexity assessment" to guess how big the contingency buffer should be.  I could see it going anywhere from 20% to a well-understood-requirements simple-technology project all the way to 300% for the chaos of poorly-understood-requirements and crazy tech.  (Schizoid needed at least a 150% contingency buffer, we know now.)

A problem with the contingency buffer at the end is - if you make your scheduling process visible to your publisher, which most say is a Good Thing - and your publisher sees your large contingency buffer, they're going to ask, "What the hell?  Looks like you're billing us for a game much larger than the one you've scheduled!"

So, for the schedule you show your publisher, keep the fudge factors in the original estimates rather than in a large pad at the end.  I said this at my talk and Ben Hoyt called me out - "Isn't that dishonest?" he said.  At the time I said, "Yes, I suppose," because I tend to cave during Q&A.  But in hindsight I don't think it is, really, dishonest - the fudged estimates are your "because this is a project of significant complexity" estimates, and I'd be perfectly willing to share that with a publisher.  I'd also be perfectly willing to say that, internally, we have these aggressive goals we're striving to hit, so for internal purposes we don't actually multiply in the fudge factors, but rather leave them as part of a large contingency buffer at the end, but those aggressive goals are not ones we want to be contractually bound to.  Trent Oster, in his talk, said similar things -  he also said to double your estimates and apparently at Bioware they try to work well ahead of the milestones they've promised their publisher as well.

Now, suppose you're partway into the project, and you're experiencing drag, and you realize even you underestimated your contingency buffer - you're going to sail right past that final deadline with an unfinished game.  At that point, a couple things need to happen:

- You have to have that hard conversation with your boss / publisher.  The "Please don't sue us for breach, but..." conversation.

- Add a bigger contingency buffer to your next estimate.  It's tempting to assume that everything that can go wrong already has and it's going to be smooth sailing for the rest of the way, but then you'll just have to have that hard conversation again with the next slip.

We're trying to approximate this curve:

With these:

On Schizoid, at first, our trajectory had us finishing in April, but we told Microsoft June.  Call that our 50% contingency buffer.  After our first slip, it looked like we could still get it in under the wire.  Then we started optimizing and we slipped again.  Our contingency buffer is used up.  But it was just a blip, right?  We'll start making progress again any day now!  No.  The blip dragged out and became a plateau. 

If we had been honest with ourselves at that point, say, by plotting out where our new trajectory was going to hit and adding a larger buffer to that, we would have ended up with a much more realistic estimate about when we'd be done.  An incredibly disheartening estimate, but realistic. 

And still wrong - here we are, almost February of next year, and still not done yet!  But it would have been much less wrong than what we were going with back then.

Sadly, "Less Wrong" is all I can offer. 

Next time (notice I didn't say next week - we just did more focus testing and bug triage and we can't bring ourselves to mark as much stuff "Will Not Fix" as maybe we should and now blogging is starting to feel like a luxury):  Just how do you calculate "velocity" anyway?

Comments

[Chris Busse]

Speaking of waterfall planning, I just watched a show on the rebuilding of the control tower for the Indy Motor Speedway race track.

They realized pretty early on that they couldn't get it done in 9 months (the time between the two big events at the track), and that it meant they were going to have to work through 2 of the largest events in the world. It was expected to be an 18 month project (and they hit their estimates).

Time spent in the planning/scheduling phase for this critically important 18 month project? 3-4 years. Granted this is apples and oranges, but I've seen said, and generally hold the belief that scheduling/planning software is harder than building physical buildings. And I'm still generally of that belief. But, I've been getting more learned over the years, and I'm beginning to come to the opinion that we might be WAY off in the amount of planning and pre-pro that we do. 1/3rd 1/3rd 1/3rd is the best I used to hope for, but they may not be enough. Maybe it needs to be more like 1/2 1/4 1/4.

[Dave Rickey]

Your quandary is why I think we will inevitably see a day when all AAA games are in some way, online subscription games. No software project of sufficient size is ever really completed, only abandoned. You reach a point of diminishing returns.

An online "game as service" business model can exist in a constant state of never being done. In an MMO, your lines *never* cross, some bugs are never fixed, some content never polished. There are bugs in EQ1 that have existed from the very first day and will probably never be fixed. The problem of "Open PK" in Ultima Online was fixed by making it irrelevant, creating an entire mirror image of the world where it couldn't happen, then tying new expansions to that ruleset. UO may be old news and small change in a post-WoW market, but it still makes more money every year than the best selling games of 1997 (it's launch year) did in that year.

There is no solution to your dilemna. As the scale of projects increases, it will simply render actual *delivery* of completely new games that can pass console certification testing effectively impossible.

5 years ago I predicted the death of the single-player PC game. That event has occurred. Once you take MMO's and Console Ports out of PC game sales, there are basically no AAA titles left for 2007. Now I'm going to make another prediction: In less than 10 years, there will be nothing like the current AAA boxed title in existence. The closest equivalent will be a digital-distribution equivalent of the GTA and sports franchises, where new packages of content for franchises are released every few months by leap-frogging development teams.

--Dave