IGDA Leadership Forum In The Box

Another year, another great conference. 

You can read all the talk summaries here - http://www.igda.org/leadership/- video and DVD's will be made available in a few months.

A highlight for me was Mark Cerny's keynote - he really is a kickass speaker, and he went to the trouble to make his slides animated and multimediaish.  The important take-home is that "culture trumps process":  a culture that values the believer and permits failure is good.  "Kill a project with love", he said, or something close to it.  (A good culture would adopt good process, it occurs to me, a la Bioware.  Even though their games almost always rock they keep trying to improve.) 

My own talk (which he sat in on, which stressed me out a bit, because I referred to his DICE talk and had this one teensy little disagreement with it) seemed to go well.  Ben Hoyt wrote it up quite well here.

"So what'd you hate about my talk this time?" Mark Cerny asked me.  Since my whole talk was about bringing back "grey box prototypes" and his talk had a bit about how "grey box prototypes don't cut it anymore" you'd think I'd take issue with that, but really we're talking about different games.  I'm talking about radical new experiments like Wii Sports or Guitar Hero or GTA or The Sims, where what you finally ship isn't really all about the graphics.  (Sure, the GTA franchise has awesome graphics *now* - but it took them a while to get there.)  He's talking about games like Uncharted and Gears where gameplay innovations, while present, take a back seat to the new challenge of capturing and rendering human motion.

And I got to meet Curt Schilling which was cool even though I'm not much of a baseball fan.  (My dad was suitably impressed, anyway.)  What I liked about Schilling, the fact that he's willing to say, "I'm not the great game designer I thought I was," is incredible to me.  Almost everyone thinks they can design.  It's so nebulous, how would you even know if you can't?  How would someone prove it to you?  (I like to think I can design, but I have no real proof, and as many failures as successes.)

The surprise talk for me was a panel on CMM, PMI, and Scrum;  three different methodologies.  CMM has a rep for being the sort of waterfally bureaucratic process that gets made fun of by Dilbert and Joel Spolsky, so it was interesting to see Tobi Launier had used one of their programs succesfully to get a DS spider-man game out the door in a mere four months.  And Rich Vogel wins my award for best culture-changing tactic:  You want people to come into work on time?  This was something we wrestled with endlessly at Treyarch, whose culture started as a mosey-into-work-whenever-and-work-late-into-the-night shop.  Here's how you do it:  serve breakfast, but not quite enough for everybody.  The strategy (more flies with honey) is well-known but it never occurred to us to apply it.

My biggest take-home:  we need to do two or three orders of magnitude more kleenex / blindtesting.  Not QA.  The first Cerny method games (Crash Bandicoot, etc) had 200 people play them start to finish, with metrics.  Then, Curtis Creamer, in his talk, said Halo 3 had even more.  When they were done, four and a half months before ship, 7 groups of 20-40 people each played through the game in its entirety - and that's not counting all the playtests they did on the way to alpha.  By way of comparison, with Spider-Man 2, we invited about six people in to play the game all the way through one weekend. 

And with Schizoid we had one person play all the way through, on his own time - we weren't even watching.  These big playtests are something Torpex isn't equipped to do (we don't have that many friends!) so we may need to consider outsourcing...

I asked myself, is that how you get 90% on gamerankings?  I talked to another friend who recently worked on a 90+ game and said they didn't do much, if any, blindtesting, and relied more on everyone playing in the office and the gut of the lead designer.  So it is possible to get the 90+ without it, but I expect most games did the big playtest.

Comments

[greggman]

Having worked on a few titles that had more testing I'm 100% for it.

The team that doesn't do it is getting lucky. The team that does do it AND actually pays attention to the results can vastily improve their game.

The tests should show what people are having problems with, where the game is too hard to play, too hard to understand, too frustrating, etc. etc. all of which can be adjusted and then tested again until players are having fun.

The team themselves know the game too well to see where a player just coming into the game is going to get stuck.