Open Letter To Jason Della Rocca

I made such a long comment on Jason's blog (although I may have accidentally posted it anonymously) that I thought I'd share it here as well, prompted by his article in the Escapist.  I think a lot of us get tired of hearing we're bad managers - my first ever Manager in A Strange Land article mentions that.  (It reminds me of years ago when I had back problems and a friend told me, "Dude, why don't you just do crunches every day?  That'll fix your problem."  You think I hadn't tried that?  Of course I tried that!  It didn't work!  "Well, you must not have done them right."  Fuck you!)  Anyhow, the comment:

I agree with your article -- too many studios out there, when faced with hardship, the first thing they think is: "We've got to make everyone put in more overtime."  That really should be the last thing they think.  Absolutely, let's take care of our guys and ourselves and not burn everyone out.  And there are a few studios out there that walk the talk, like Infinity Ward, and boy do they get results.

But I also find the article a little hurtful.  Your article talks about how we're in the dark with our project management, and how if we got out of the dark we wouldn't have to kill ourselves anymore.

Maybe except for Tim Moss, I think most of us bust our asses to "get out of the dark", to bring good practices, processes, and management to game development, and then the projects go into crunch anyway, and then everyone tells us, "Maybe you guys should bring good practices, processes, and management to your projects."

I felt the same way about the McConnell talk at the GDC last year (didn't see it, saw the slides online) - you guys aren't in the trenches with us, but when we screw up, you tell us "If we were running things we would have done it right, and you guys wouldn't have had to crunch, and your game would have turned out even better."

I've never worked for EA, but they are the poster child for rigorous management - they do everything they can to implement all the best practices you can name - when waterfall was big, they were all about educating their teams on how to use waterfall.  Now that agile is taking over, they're all about educating their teams on how to be agile.  (Don't get me started about the waterfall-agile thing.  If something goes wrong on your project, it's because you didn't plan enough - you should have used waterfall.  If your project flounders along for months in the design phase, you're planning too much - you should have used agile.  Usually both happen.  You should have been more agile AND planned ahead more.)  EA University, management classes, programs, you name it.  And still they end up crunching.

You guys are offering us silver bullets.  As you know, there ain't no such thing. 

Game development is one of the most difficult endeavors known to man - it combines the unpredictability of software engineering with the unpredictability of film-making and the unpredictability of creating an interactive experience that's different for every user...unpredictability cubed.

Here's a metaphor:  when you go to war, nobody says, "You know, this war'll be easy if we bring in good management practices."  Good management helps - a lot - in war but doesn't make it easy.  It doesn't mean that there won't be times when everything goes wrong and you're back in crunch mode.

Neversoft's a good example of that - kick ass studio, good practices, they kept the overtime down for three straight projects - then along came GUN, and as I understand it, they ended up behind schedule and Activision had to reassign a ton of people to their team in order to ship.  I don't know if they ended up crunching or not, but I do know things didn't go anywhere near as swimmingly as the previous few Tony Hawks.

Well, listen to me ramble.  Would I have you change your article?  No, keep preaching it, brother, but don't be surprised if even the believers are still crunching at the end of the day.

Comments

[Troy Gilbert]

I know management always gets picked on. It's the way the hierarchy works: you could pick on individuals, but it's hard to be "hand wavy" then, you gotta list specifics. You could pick on the top level, but they don't actually do any of the day-to-day. It falls on the shoulders of the middle managers (which in games would be the TDs, the DDs, the leads, producers).

I think the games industry could learn alot about better software development practices. No, strike that. I think the industry knows plenty (or enough) about the right way to make software; rather, we need to start applying these things.

You call out two examples, EA and Neversoft. First, I don't know what part of EA you've dealt with, but I've not been witness to a widespread use of agile methodology. Everything I see is still very plan-driven up front, with an MS Project schedule that proves that the product will ship with x number of resources y months from now, calculated down to the man-hour. While I'm no agile expert, from what I've learned from the folks who are best practicing it (Clint and Noel @ High Moon Studios), that ain't how it's done.

And in regards to Neversoft, their previous 3 titles before GUN were sequels in the Tony Hawk franchise! Surely a studio that's practicly defined a genre and been perfecting it over a half-dozen sequels should have the project planning, etc., down pat. Throw in a new title in a new genre and the old ways breakdown... and you get the fair-to-middlin result of GUN.

We in the games industry always like to pretend like our particular brand of software is so freak'n special that none of the rules apply. And of course it is. As you point out, game development is just about the pinnacle of collaborative product development: it's got all of the production/craft requirements of film (the previous title holder), plus the difficulties of software development and the unknowns of an infant medium. Mix in a bit of mass-market demand, platform churn, licensing restraints and a still-not-perfect-business-model-or-distribution-channel and you've got a heady problem to solve.

But just because it's hard, doesn't mean we toss the rules out the door. We break it down. We've got artists, who in every other industry they work in they have good-to-great tools (many of them paragons of software), but in the games industry are often stuck with hacky tools. At best. We've got software developers who are just now really accepting C++ as a given, who still think middleware might be a dirty word, and who are still solving problems related to databases and concurrency that were "solved" decades ago by Big Blue and others.

At the end of the day, most of our corner-cutting and reinvention is based on the premise of squeaking out that last iota of performance. I spent my first year or two at RenderWare (2001-02) trying to convince folks that middleware was a viable option for their game. "Oh no," they'd say, "the performance won't be acceptable." Answer me this: which has the greater impact on success (measured in sales, profitability or employee satisfaction), shipping on-time with no crunch at 30fps or shipping late with 2 months of crunch at 60fps? Anyone with any perspective on the business-side of things would clearly point to outcome #1.

In the end, we're victims of our own fascinations. Game development was founded by programmers, then quickly joined by visual artists. Those two together compose the majority of the production team of any game. As a result, you have a product development guided and dictated by the demands of programmers -- who given the opportunity will happily pursue all manner of problems, whether they be germane to the product or not -- and by the demands of visual artists, who want their artwork to look as good as possible, at the best framerate as possible, regardless of its applicability to the core of the game or the consequences the demands have on resources allocated elsewhere in the game.

Don't get me wrong, there's a place for everything. But I've found that by taking just a small step backwards I've found game development to be far more enjoyable and surmountable. Go grab a copy of Flash and say to yourself, "I'm going to live and work within the constraints of this tool." You know what? You'll soon find that you're not all that constrained, and you (and your artists) will appreciate how comfortable and doable the whole process is. Or, grab a copy of C# and remember the joys of programming (getting things done!). Or grab SDL and a few other "simple" libraries and enjoy working at the minimal end of the pool.

And what does all of that last paragraph have to do with process and your post? Well, it relates back to the idea of why we're crunching and why games are different. It's because we're constantly re-inventing everything. If we'd just take a step back and leverage what's already been invented and proven, I think you (we) will find huge enjoyment and satisfaction.

[Pag]

I realised recently that it's more important for a project to ship on time than to ship as fast as possible. Because projects don't exist in a vacuum, missing the deadline costs a lot: badly timed marketing, annoyed customers, broken promises to retailers, tired team, etc. As such, promising to ship a game in 18 months and doing it seems a lot better than promising to ship in 12 months but actually taking 15 months. Sure 3 months were saved, but the cost of being 3 months late more than compensates (just look at EA's stock price after they announced The Godfather was late).

As such wouldn't it make sense to purposefully plan for more time than is really necessary? I realise things aren't always that simple, but adding slack in the schedule could help with planning problems.

[Despayre]

Interesting discussion. I dont know if Jason is talking about a silver bullet, but I do know that a lot of other people are. The bullet it seems we're all trying to find are the mythical 'best practices' for our projects. Using established best practices sound great in theory, but really, I think so many of us are so enamored with finding THE best practice that we end up causing more harm than good.

A behemoth like EA can use what works for them to churn out sequels every year because that is what they do, they leave originality to other studios, which they promptly swallow..err..acquire. For those of us working on different IP's with every project, we have to be able to modify our practices based on the demands of the current project, not what worked before. Maybe this is what happened to Neversoft? I don't know anyone there, but I'm guessing that things that worked in a skatepark probably didnt translate well to a gunfight.

Also, we all know this already but we need to recognize and admit that a lot of our middle management people throughout the industry probably should not be there. We like to reward aptitude within our staff, and use that aptitude to create the best team possible, but that doesnt mean that your best coder is suddenly going to be able to write code that makes them into being a good manager.

So now we have all these best practices floating around that smart people who know they arent good managers come across and they try to apply them to their current project. Now sometimes this can work, but only if you dont start to think that this is the ONLY way to do things.

Oh and let's not forget that we're always trying to upgrade our tech to keep up with the current generation of hardware and competing products as well. Best practices work for non-entertainment software because they dont have to worry about how they're going to deliver something fun that also looks great, they just need functional. Have you ever needed directx9 shader effects for your documents? Real time deforming cells on your spreadsheets? An email program that made you want to keep writing emails because they were so fun to send?

The high burnout rate also contributes to a lot of our problems..Our best and brightest people are rightfully frustrated by the wall of crap they deal with from being put into a position they weren't suited for in the first place, and they leave the industry before they have a chance to be in a position where they can affect REAL changes to the production processes. Since we're an industry of egos, the prevalant attitude seems to be 'our way is the best way', so the chances of any real change being made in a decent sized company seem miniscule.

Whine whine, bitch moan..all that jazz..now if you'll excuse me, I have to get back to crunching...

[Liam Hislop]

It seems to me that a lot of what Jason is complaining about can be fixed by proper management training. There are "good" and "bad" practices everywhere. The fact that many people "didn't know" about some other practices tells me that they either aren't listening or aren't getting the training they need. There are dozens of production talks at GDC every year. There is no reason that someone "shouldn't know" about how things are done.

The other thing is that I think people try to pigeonhole our process into "software development" or "entertainment industry" (movies, tv, etc), and how they are both mature and why can't we be. Because we merge the interactivity and technology of "software development" with the entertainment value of the "entertainment industry", we have created something that is somewhat different.

You see the problems in movies now of changing from regular film to digital moviemaking is causing trouble and issues throughout that industry. Now imagine if they were changing those tools every 4-6 years. It would be a nightmare. That is the boat that we are in.

Jason complains about young & male creators in our industry, but does he forget, that young and male was the norm in the 1970's film industry, a time that many people call "free", "revolutionary", and a "classic age of filmamking". There were a lot of young talent coming out of film schools after the collapse of the studio system, and many young people were given a chance. They took risks, went overbudget, had giant hurdles, but came out with some of the best films ever (Godfather, Exorcist, Jaws, Closse Encounters, Star Wars).

We now have a number of game schools teaching young talent (Shameless plug for Full Sail) how to create games using good tools and management processes. It simply takes time for everything to come together, whch I think that it slowly is. We are growing by leaps and bounds, and we are intelligent enough to learn a lot from other industries (we also repeat their istakes as well), but liek everything it takes time.

You are right, and there is no silver bullet. But by having solutions and doing something about the problems instead of simply ranting about them, we are going to overcome them.

[Paul Sinnett]

I think you were right to feel annoyed by Steve McConnell's lecture. It seemed to me that Steve hadn't done his homework. He may be stuck in 1996 but the rest of us have moved on. The two biggest process improvements, that the studies he quotes make, are re-use and tools. The games industry uses these in spades. I can't think of any game conpany that doesn't have source control, automated build tools, and half a dozen pieces of middleware and libraries.

I have more thoughts on this, and how it won't affect quality of life, at http://paul.sinnett.name/archives/11

[Jare]

Software engineering practices.
Art tools and processes.
Design education, structure, language, documentation and production.
Platform and technology churn.
Marketing-driven product decisions.
Subjective "success" metrics.
Evolving audience and market.
Changing publishing and distribution channels.
Rigid and inflexible finnancing methods.
High burn rate of staff.
Questionable public image.
...

There's an awful lot of people trying to do things better, but the scope and combinatorial depth of the issues involved is just huge. We'd need a Silver Fragmentation Bomb to solve them all. In the meantime, our only options are:

Give up and move elsewhere.
Try to minimize the impact of QoL issues on yourself.
Try to improve what you can while you endure QoL issues.
Job-hop in search of better places.
Create your own studio.

Listed in what I belive to be the more-to-less frequent order.

[Timothy]

Jamie - you bring up a great balancing act that affects many kinds of projects: project rigor vs. creative urgency. The discussion generated was thought-provoking and certainly dredged up both sides of the issue. The trick is intuitively figuring out (on a project-by-project basis) how much project infrastructure is needed and how much we need to keep things fluid and dynamic without stepping on the creative types. Keep up the great work. Really enjoy the conversation.

[Jarred]

Hey Everyone!!

I'm currently a design student and practically chomping at the bit to get going with life. It's blogs, articles, etc like these that keep both my hopes alive and squelch them all in one fell blow. Good management, bad management, it's everywhere in every industry at every level. Granted I'm an outsider looking in; all that I hear are horror stories about crunch time and how this is not an industry for the faint of heart or those who want a personal and/or family life. Personally I take such articles and rantings with a grain salt while attempting to separate the body of the salt and the flavorless crystals leftover. Those of us students (looked at Full Sail, couldn't afford it so taking online classes out of a College in Cali) who are excited about the industry can't imagine the nightmares that are listed everywhere online. However, it's hearing both sides, as Jamie presented here reminiscent of Paul Harvey's "Rest of the story", that keeps our hopes alive and our excitement high!

Every student I know wants to make a difference in the industry, to leave their mark, but that's not the driving force behind us. Regardless of management people are in the industry because they love what they do, even if they hate their job. I've heard a lot about people getting burned out on it all, but most can't seem to stay away and come back in some form of fashion. Last I knew, this industry was all about creativity and expanding our horizons DESPITE management, crunch time, poor planning, new technologies, and software bugs that rival the Locusts of the great Plague. Yes, I'm believing the entire industry in EVERY position (management to artists to producers and the cleaning crew)are here to share some part of their creative side. After all, aren't they always attempting to come up with a new or better way to streamline the process or make more money? It's all in how creative we let ourselves be in our own mind. What's not possible today WILL be possible tomorrow as long as we don't let our imaginations die!

I feel that if we keep it real with ourselves, then it all works out.

[Nathan McKenzie]

On the one hand, I've seen countless commercial game projects doomed or at least maimed by bad project planning and project management, and I've been on the recieving end of a few myself. It's unquestionably a problem (and a big part of why I'm doing contract work right now, in fact). I've spent my share of days sleeping on my office floor. This is not comfortable.

On the other hand, I get exhausted watching external software consultants wandering around, providing reassuring simple big picture answers when they haven't dug in too deeply to find out the questions. Games are probably surprisingly similar to other software in all sorts of unexpected ways, and they might be viciously different in a few as well - but those difference ought to be empirically discovered and documented, not smothered by fiat, right?

I think the real problem might be how opaque the industry tends to be. What I don't need are outside experts delivering bullet points for hefty checks. Often that just ends with accusations from upper management, wondering why the team isn't implementing the consultant's three lists of generic words to the tee. What I need is a lot more information about projects that actually overcame these problems, were run sanely, and still delivered something I'd value working on or value playing. You know, real best practices, situated in real game problems.

I was talking to a former co-worker from Raven recently, and he listed every project that he knew of at Raven that went smoothly, wasn't loaded with crunch time, and didn't have a mass exodus of employees after coming to a conclusion. Every single project that filled that description was a sequel that reused an engine and lots of art assets, didn't explore new engine technologies, stuck with a genre Raven had already done without rocking the boat too much, reused tools and known (and debugged) content pipelines, and was staffed by people already accustomed to those pipelines, codebases, and processes.

Sounds pretty Tony Hawk 3, right?

So there - that's one set of honest to god best practices. That's a sane way to finish games... until, of course, hardware technology advances and you're simply not allowed to rest on your past laurels. Or until you notice that your games are all fine, just fine, but they're not pushing too far above the 86% range on gamerankings, and maybe you don't want to spend your whole career earning bronze medals.

I'd love to have more examples, and more details, of people who make stuff that keeps pushing envelopes and do so in a sustainable way. But, at least in my experience, that's pretty thin on the ground.

[Parveen Kaler]

Pseudo-trackback: http://parveenkaler.com/?p=26

[Zack Hiwiller]

This is the best post and comment thread I've seen in a while. Good stuff guys.

I've been crunching for the past X months. No, it is not desirable, but it is neccessary. If it was easy to run a business, there would be one true book out there telling you how to do it. But as it stands, there are thousands of books about management practices that come out each year and the only ones making money off of the suggestions are the publishers of the books.

The fact is that you have a dozen different groups out there pulling in different directions. You have consumers/critics that want something new, yet familiar. They want tons of content, but don't want to pay for it. You have licensors that want something fun and that leverages their IP, but they also refuse to take any risks to get there. You have developers who want to make 90+ metacritic games, use all the coolest cutting edge tools and tech and do something new and edgy, but they also want to work forty hours a week. Then you have publishers who want low risk and high returns. Take a look at the stock market and try to find a security with low risk and high returns. Only some days they really push for low risk and others they really push for high returns. In the end, you get something in the middle, if you are lucky.

People talk like this is a problem that the industry will eventually mature out of. I don't see it happening. I think the best we can do is set a clear vision for our products and run with it, despite the noise from all the external sources. We all try to do the former and all end up succumbing the latter.

I treat critics as noise, because that is the one aspect of my project that I can tune out. I have to listen to the licensor no matter how stupid their requests are. I have to listen to the publisher because they pay me. I have to listen to our focus group kids because they are who we want to buy the game. Critics, however, be they critics of product or critics of process are a dime a dozen and overvalued at that.

[Jare]

Crunching because you're trying to go the extra mile and expect that effort to be appreciated and compensated is one thing.

Crunching because someone else isn't willing or capable of investing the necessary resources usually means your effort will not yield a brilliant result (just patch things up to adequate levels), you won't get properly compensated for it (there weren't resources in the first place), and you won't have as much respect for the person and whatever appreciation he might throw your way.

Nobody wins the Olympics on a comfortable 40hr/week schedule, so putting the extra effort is always going to be a road to getting that extra level of quality and polish. But it's only worth for only those with the medals.

[Rich]

To borrow your own analogy, Jamie, most major errors in war come from strategic, not operational, errors. Or, at least, pursuing strategies that aren't fault-tolerant at the operational or maneuver levels.

You can see this all the time in games. Why do we pursue a strategy of development that depends heavily upon code implementing feature x within time frame y when we know that there's a likelihood that code will run into obstacles that will extend their work time beyond acceptable levels? Why do we constantly plan design or art operations supposedly in support of our strategy that may very well cause our strategy to fail? Sometimes this is because we have our backs to the wall and are given no choice but to pursue these strategies and these operations (by our publishers, etc etc). A lot of times, however, we pursue strategies that are not fault-tolerant because we know we can fall back on the nuclear option.

The nuclear option of which I speak, of course, is crunch. For all the myriad faults of a company owner who I've come into contact with over the last year, I had to admire his honesty about 'the big C'. He is one of the only people who I know that will unabashedly admit that crunch is openly a part of his maneuver repertoire, used in support of his operations and, therefore, his strategy. A true modern Machiavel.

The problem with most production decision-makers is that, somewhere in the back of their heads, when they're formulating their strategies, they know that they will be able to rely on that nuclear option. They say they want to avoid it, but what they really want is to not have to do one more of the following:

- Conduct the research pre-planning necessary to determine exactly what strategic goals can be supported without operations that have the barest chance of relying on crunch.

- Pre-trim the feature-set down within the covered nucleus of the above-mentioned strategy, given the manpower and technology available.

- Make sure that any extensions beyond the acceptable 'covered nucleus' are dynamically scalable in order to accomodate uncertainty in both positive and negative directions.

So, they commit before God and Publisher to a strategy that sends them and the folks they are responsible for/to on a highway to crunch hell. And, when the time comes, they blame and overexamine the tactics, maneuvers and operations that support their faulty strategy and say, 'what went wrong?'

Could you imagine Napoleon getting back from Russia and saying , 'Damn... maybe next time the guys protecting my supply lines need to be more efficient/need to be "managed" better/need better guns!' Ridiculous, right? Kind of true, though... when you think about it. Maybe if those changes happened, things would have gone differently. And maybe not. Point is, his strategy was not built to be fault-tolerant on the operational level.

The good news is that, unlike old Boney, our strategies usually *do* have fault-tolerance. The bad news that we should admit is that this fault-tolerance almost always includes crunch. Boo. Boo us.