Manager In A Strange Land: Theory Of Drag, Part 2

Part 1

So where does this long tail come from?  Why doesn't velocity average out so that after you've been working on your game for a couple of months you've got something you can dead-reckon with?

Some would say it's because of lazy developers.  We set a goal - we realize we're going to miss the goal - so we set an easier goal - and then we slack off since we've set that easier goal - and then, before we know it, we're going to miss that goal, too - etcetera.  Parkinson's Postulate.  Freshman Syndrome.  I'm not a big fan of this theory.

Another cause is plain old underestimation.  Discovered work, for one thing. "We just forgot we'd have to do that", etcetera.  But - shouldn't Discovered Work average out after a while?  Maybe not in two months, but after several months shouldn't you have a pretty good idea of the ratio of discovered work to originally planned work?  No.  The further in the future the work is, the murkier it is, so you're going to tend to see more and more discovered work as you get towards the end of the project.  In a way, your post-alpha bug list is *all* discovered work.

Underestimation can show up on the schedule graph as peaks ("We forgot we had to this!  Better add it to the schedule!") and plateaus ("You're *still* working on that?").  I can look at the peaks on the schedule graph for Schizoid and say, "That was when we realized we'd have to multithread";   "That was when we started working on our certification requirements and discovered just how hellish they are this time around".  [Side note:  Oh, Mr. Allard, you promised!  You promised it would be easier this time!]  [Side side note:  for those of you doing 360 development, don't think of that 130 line item TCR list as the real list.  Go straight to the test-cases list.] and "That was when we entered QA."

A pernicious kind of underestimation is technical debt - when you think you've finished a feature but there are still lingering issues in it that get caught later.  This shows up on the schedule as good velocity now for a creeping increase in bug count later.

Underestimation is most of the story.  A lot of people think it's the whole story - that's part of the waterfall "gather *all* requirements and outlaw feature creep" thing.  "If only we knew what the client really wanted / hadn't changed our minds / hadn't forgotten x,y, and z, we'd be done on time!"

To get the rest of the story, we need to split the graph into two:  a graph of scope increasing and a graph of work done.  Schwaber's burndown charts only deal with 'work remaining', and that's what we used for Schizoid, so it took me a while to massage our data to show both:

The green line is scope - as we discover things we forget (we need to multithread;  the designers want the macro game to work differently;  we should have known we'd have to do *that* for TCR) and as bugs get discovered the green line goes up. 

The red line is work done:  features completed and bugs fixed.

Visually it looks like the green line is curving one way (the rate of discovering work increases as we go) and the red line is curving the other (the rate of work done is decreasing.) 

Something the graph doesn't show is we added a programmer halfway through.  You'd think we'd see a bump in the red line there but there isn't much of a sign.  So even with the extra manpower we're getting less work done.  (You might think this means he's actually slowing us down but trust me, he's not; I don't know where we'd be without him.)

And that bowing of the red line, despite having that extra programmer, is the evidence of drag.  Patrick Hughes predicted where I was going with this in the last post - as the system gets larger, your efficiency working on it decreases. 

It's like a tractor pull - or perhaps a better metaphor would be pushing a snowball up a hill. 

We see this in a variety of ways when working on videogames:

* The components of the system have to play nice together.  As you add components, each future component becomes that much harder to create, because it has to work with all the previous components.  Boehm's cost of change curve is supposedly for unplanned features - I believe that it applies to all features whether they're planned or not!  I'm too lazy to graph our lines-of-code against time but you know what we'd see - the same bowed curve - which doesn't necessarily mean we're doing less work but is a good sign. 

* As the game gets bigger turnaround times increase.  It takes longer to build, and to run, and to load a level.  It takes longer to play through a level to get to that point where the bug is.  (You can implement a cheat that lets you zip straight to that point;  or write a custom test that lets you execute the code without running the whole project;  but doing that takes time, too.)

* QA gets tougher on the larger system.  Finding and reproducing bugs becomes more and more difficult as you get fewer needles in a larger haystack.

So that's Drag.  Common sense, you're probably thinking.  But if it's common sense, why do we schedule as if it doesn't exist?

Why does Mike Cohn, in *Agile Estimation and Planning*, say we can use our velocity for long-term estimation?

Why does Mark Cerny say we can use what we've learned prototyping to accurately estimate how many levels we'll be able to make in production?  (Not to dis on Cerny, his Method is awesome and I wish it hadn't been seemingly overshadowed by Scrum.)

Why does Joel Spolsky say you can simply compare estimates to actuals to adjust future estimates?  (BTW - *something* must have been wrong with Spolsky's old system, otherwise why would he have felt the need to create a new one?  But the new one doesn't solve the problem.)

We're all using linear math to deal with this nonlinear problem.

Credit to Ken Schwaber - he never said his methods could be used to make an accurate long-term estimate...he just cops out and says you can't predict anything more than a couple months away.  Not too useful for a studio that might like to, say, sign a contract promising a certain delivery date.  We want to be able to hit goals and keep promises!

Coming up:  we're basically screwed, but here's some things we can do to mitigate the damage.

Comments

[CDriK]

Nice and obvious. ;-)

You say that it's very difficult but you have a solution ?

[larsiusprime]

So, you're an *optimistic* fatalist :)

Something I'd like to know: does the degree to which a project "Werewolves" in y'alls experience increase with the scope of the project?

Is there anyway to "expect the unexpected?"

Schizoid is much smaller in scope than Spider-man, were there any difference in the way it started to drag than all the big projects you've worked on?

[Jare]

>> Is there anyway to "expect the unexpected?"

There are degrees of "unexpected." A flood destroying your offices is the "hard" type, and even for that there are ways to lessen the impact and potentially turn a disaster into "only" a major setback (offsite backups are the classic example).

We should know to tackle the "know the unknown" issue: prototype, iterate, 90/10, etc. EFFECTIVELY, but in my experience we still fail at that. Why? I feel the concept of drag is not going to help much there. Drag and progressively reduced productivity curves are important things to factor in your plans, and I love the way Jamie is digging into them, but they only deal with the management of production, not the management of creation.

[Mark Nau]

I see two linear paths that are eventually going to converge, but you ran out of time. A plot of the slope of the lines might change that estimation.

I'm far more puzzled by the failure of the red line to show a slope discontinuity at the point where you added another programmer. When something as basic as increasing your workforce by a large percentage isn't being reflected in your metrics, that makes me question the accuracy of the whole exercise unless I can find a good reason for it.

[Chris Busse]

How much of the Green line is stuff you forgot and how much is feature creep? Put another way, how much is stuff you needed and how much is stuff you wanted?

[Simon Cooke]

Here's my version of your graph, and the explanation... based on many many projects, and my experiences recently porting a game to PS3.

It *roughly* matches... but it's also targetted more at the programming work; it doesn't apply so well across the board... I think if you average out work done in all areas you get something closer to yours.

http://www.accidentalscientist.com/2007/10/visible-progress-of-broadly-scoped.html

[Zachary]

Unrelated to the above but...

http://kotaku.com/344106/bionic-commandos-swing-mechanic-explained

They're calling you out!

[Nick Doran]

I think another contributor to drag is Darwinian bugs. Bugs that get are still around late in the game because they are hard to reproduce, hard to fix, and hard to find. They've survived until late in the project because of these traits.