Manager In A Strange Land: Wisdom of Crowds?

I read *Wisdom of Crowds* a while ago and was taken in by its arguments...
The basic idea being that for a lot of decision-making, getting an average or composite of a group opinion is better than a vast majority of the individual opinions.  Whether it's counting jellybeans or investing in the stock market, Joe Whoever isn't going to do as good a job as the average.  This has led me to be maybe a little more willing to build consensus and compromise than I should be. 
Because here's the counter-argument:  we don't care about whether the wisdom of crowds is better than the wisdom of the random individual.  We care about whether the wisdom of crowds is better than the skilled individual.
A question I have not heard asked:  does someone trained in jellybean estimation tend to beat the jellybean estimating crowd?  Or - in a series of jellybean estimation exercises, do we tend to find that one or more people consistently beat the average?
I ran a little experiment with some friends at Treyarch a few E3's ago.  We played a game where I gave them a list of games that were being shown at that E3 and we all looked at them and guessed what their gamerankings scores would be.  Just like with jellybean estimation, the average guess tended to be better than the random individual guess.  Here's the thing - one guy (Alex Bortoluzzi) - was right on the money every time, consistently beating everyone else and the average.  Our sample size was too small to draw any really scientific conclusions (from a list of 10 games only 6 ended up ever shipping!) but when I wanted to know how Spider-Man 2 was going to do in the rankings, I didn't run a decision market.  I just asked Alex.  He turned out to be right.  (A lot easier to do than run a decision market, also.)
A more famous example is Warren Buffet.   He consistently seriously outperforms the S&P.  Although he's humble - in *The Essays of Warren Buffet* he makes it sound like anybody could do it -I'm not about to start picking individual stocks, because when it comes to the stock market I'm Joe Anyone and the average is likely to beat me.
So maybe the question we should be asking when making game design and business decisions is probably not, "What does everyone think?  Can we come to an acceptable compromise?"  But rather, "Who's our Warren Buffet?  And what does he think?"
Of course, if you're in charge, and you decide that you're the Warren Buffet...then you've just taken a step down from being the Level 5 leader to being the Level 4 one.  (Or maybe even lower...) 
So that gives me a good opportunity to do the Socratic thing.  What do you guys think?  Compromise?  Blindly Follow The Leader?  Or Is There Something Better?

Comments

[greggman]

I think there are a lot of mixed up ideas here and I don't really know how to separate them.

I'd say my dream ideal is to find someone who has the design talent and who is also able to take input. So, while he might not have all the ideas he has the skill to choose the good ones and the leadership to lead the team to create it.

Most teams have no real leadership. They have people who have roles assigned (lead designer, lead artist, co-design lead, producer) none of them come across has having either great design skill or leadership. That's not to say they suck, just that it's not clear they'd be better than your average group.

There only seem to be a few exceptions. Miyamoto has consistantly lead exceptional games. Sid Meier as well. There's probably only a hand full of other designers that have that "skill" you mention. Everyone wants to believe they personally have it but there's certainly no evidence they do. Then again, who knows how many things conspire to hinder anyone with true design skill (ie, not the lead, bad producer, lame programming team, unrealistic deadline...)

That reminds me though, I think I've mentioned this before but it's not all about skill. Sometimes it's also about process. Nintendo supposedly (A) does 1-2 year prototypes with small teams, like 5 people (B) has the Mario Club for *DAILY* kleenex testing even during protoype. Put something new in or tweek a few parameters and then have the Mario Club test it and give feedback in a day (was it less frustrating, was it more fun, was it easily understandable, etc...)

My point is, even without have the ultimate design talent, if you were at least competent and followed Nintendo's practices you should be able to make a highly ranked game.

[Andrew Cory]

Coming at this from a different perspective, and about different issues: I studied Poli-Sci in college.

The chief problem in political science is to align a leader's incentives with the best outcomes for the led. That's 100% of the constitutional design question. We have no idea how to do that. Instead, we're designing constitutions which are better and better at getting the led what they want, and keeping certain desires off the table completely...

Our Constitution, BTW, is the 1.0 design, though it's in REV 27 by now. The Brits do have an older one, but it's still in Beta-- and documentation is spotty at best! Must have been written by google...

[Jare]

You don't step off Level 5 just by deciding that you are that kind of brilliant person. What if you are? Downplaying your talent and skills is a terrible thing to do. Besides, having that kind of talent is only part of the whole:

- You need to let other people improve by letting them try and fail (or succeed); if they stop thinking for themselves because they'd rather defer to you, or they start believing that their own progress is in trouble, then you risk alienating and/or losing great people.
- You can't become too much of a choke point for decision-making; sometimes a good enough decision "now" is better than a great decision "tomorrow".
- You must be able to communicate your ideas properly; not everything is as simple as a yes/no or a score.
- You have to understand the context in which that talent is applied; someone at EA Sports may have a great idea for a FPS game, but EA Sports is not the place to do it.
- Every talent has limitations; was Alex able to predict which games were going to be cancelled?

In a nutshell: I believe in the power of individual talent, but I also believe that it rarely is enough. :)

[davidicus]

Jakob Nielsen believes polling only five decent people can get you more than 85% of your desired feedback. not much of a group, right? apparently, testing 5000 doesn't get you a much better result.

i'm an individualist, and believe group feedback should be evaluated by minds that understand the parameters at the necessary levels.

in my favorite example, when Sony first floated the Walkman balloon, the crowd outside said it was wrongheaded; they didn't need to listen to music between their home and their car. Sony disobeyed and made history, saying, "the public doesn't know what is possible. we do." (paraphrased poorly, no doubt.)

[Will McGuire]

If the team is being lead by a strong person (or collection of people), it is possible to defer strickly to the leaders with great results. However, it has been pointed out here that strong leadership is hard to find. There are few exceptional leaders to go around. These leadership positions must be filled, but are often filled by hardworking team members who are not quite ready or willing to be placed into those roles.

Also, each team (and even individuals within the team) require a different level of interaction and guidance (think Situational Leadership). There are points throughout the project where "buy-in" or collective-ownership becomes an integral motivating factor for the team and its members OR a requirement to ensure that your team (and its members) are able to increase their repective skill sets and project efficacy.

I guess the main point I am (poorly) dancing around is, there is no single solution or practice that with perfectly be implemented within every given team. Often, it boils down to the leaders experience and ability to recognize what to do and when to do it AND the teams "Developmental Level" and their willingness to accept leadership.

If team members have worked with their leader on multiple successful project, they are more likely to trust him/her and will willingly accept the guidance. The converse is also true.

[dn]

Interestingly, James Surowiecki, the author of Wisdom of Crowds, had a recent New Yorker column which discussed the faulty wisdom of focus grouped feature specs. He covered a concept that I'm sure is very simplistic to all of us. If you ask a bunch of people what they want in a product, they'll give you a laundry list of amazing features. But if you put it out into the market, users think it is too complex and often only use 15-20% of the feature set. The most successful products are generally simple and focused - not kitchen sink designed.

So what does this mean for your question? Well, I think it's important to note the extreme difference between polling people on how successful or good a product will be (Gamerankings scores) versus polling people on what features they think a product should be made up of to be good (key feature list).

Decision markets may be good for judging completed games, or even completed GDDs. But they may not be good for deciding what goes into those games.

[TimL]

I'm coming late to the party now, but I was so compelled by the post and comments (all of which are very insightful and useful for me personally) that I wanted to add just one thing. I hope this isn't too far off the target, but a factor that I had left behind quite a few years ago and then have tried to pick back up recently, when it comes to leadership decisions, comes from a different part of the brain (or heart) which is "personal passion."

There was a time when I was soo consensus driven and accommodating that it did obviously impede progress and I took the crowd mentality admittedly too far. I think I was known as a fair design leader because of it, which had it's advantages but looking back there was that potentially key element of personal passion that may have been missing.

Many of you mention the meta concept of balancing a pool of great ideas and then focusing them by clear and decisive leadership, and I buy all of that and I agree. But in a way those are quantifiable aspects of a proposed process that we can all get behind because it does make so much sense. But what about the X-Factor of passion? So in the end, yeah, we can ask the crowd, the design team, the entire dev team, and then we can make a good calculated call based on core goals and pillars, but if we don't "feel" something for it (especially the bigger features) do we lose out in a big way?

I'm not sure where I'm going with this now, but this line of questioning is something I've been grappling with lately now that I'm trying to keep an eye on something which I thought needed to be a lower priority than an effective development process.

[jmkronbauer]

Hi I would like to contact Alex Bortoluzzi.
He is a good friend of mine. I lost contact with him many years ago when we worked for the same company in Brasil, Digitel. If Alex sees this post or anyone knows his email address, please contact me.

[Alex Bortoluzzi]

Jaques! Puxa, tu me achaste da maneira mais estranha! Me manda um e-mail pro link no meu nome!

Jamie, sorry for hijacking your blog! :)

[Jamie Fristrom]

Ok, so apparently the blog doesn't put up e-mail addresses - thoughtful of it. Jaques, e-mail me directly at jdfristrom@gmail.com and I can forward Alex's e-mail to you.