Just finished Malcolm Gladwell's Blink: The Power of Thinking Without Thinking- I was leery at first because it sounds like the sort of book that's going to sell because people want to believe they can trust their gut and they're just looking for a book to back them up.  And, to be fair, that's how it's marketed – and someone even wrote a reactionary book called Think dissing on it for that.

Also, I wasn't very fond of The Tipping Point – I thought his central thesis is just wrong – there is no point – in epidemics, isn't there just a steadily growing curve?  You can't put your finger on a single point and say, “that's where it tipped.”  Although, The Tipping Point was filled with interesting factoids which made it entertaining anyhow.

But, I'm happy to report, that Blink isn't really about that at all.  Although it is filled with anecdotes about people trusting their guts and coming to correct decisions in small amounts of time, these anecdotes are all about experts who have been working in their fields for many years.  So, yes, you can trust your gut, after you've got years of experience under your belt.  This should surprise nobody.  Kasparov can find the correct move on a chessboard in a second that would take me minutes or hours to analyze.

And I was glad to hear it, because I am a very not-trusting-my-gut sort of guy, even though I have fifteen years of experience making videogames.  I'm always second guessing myself. That's going to change.  Now I'll just second guess myself a little.

Also, a lot of Blink is about the times when your gut can fail you.  In fact, it seemed like more of the book was devoted to how split-second decisions can go wrong. 

One interesting kind of split-second decision that can go wrong is the Sip Test. Coke switched to New Coke in the late eighties because Pepsi supposedly was winning the flavor wars.  They were winning on the basis of sip tests – you did a blind sip of beverage A, a blind sip of beverage B, and which did you like better?  Pepsi, because it was sweeter and had a burst of citrus.

So Coke changed their formula – earned the hatred of millions – and then changed back and gained more market share than before, managing to blunder into a happy ending. 

Al Ries claims that Pepsi tastes better and it's all about marketing, but one of the marketers at Coke says the problem is sip tests suck and to really rate a product you need to give them a home use test:  you give them two cases of the two beverages and ask which they like better.  (I suppose you better make sure the cases are just generically labeled “Product A” and “Product B”...I wonder if they do that?) Because the sweetness of Pepsi gets cloying once you've had more than a few sips, and that hint of citrus you detect with your first sip is lost on the second.

How does that apply to videogames?

It's conventional wisdom that a videogame has to grab you in the first 30 seconds or it's doomed.  I'd like to challenge that conventional wisdom – look at Tony Hawk and Spider-Man 2.  If you can fail a kleenex test—the videogame industry equivalent of the 'sip test'--these games did.  It takes 15-30 minutes of playing before you start to feel even comfortable with these games, and it takes longer before you get good.  And yet they're both multi-million sellers, outselling a couple of games that no doubt aced their kleenex tests – God of War and Ninja Gaiden – games where you feel good in the first thirty seconds, mashing buttons and explosively killing guys.

Apples and oranges, I know – multiplatform vs. one platform, franchise title, different marketing, etcetra.  But, as NASA research shows, apples and oranges are really quite similar.

Also, this conventional wisdom seems to assume that players are playing demos when considering whether to buy, and the demo that catches their interest is the one they pick up.  Yes, that happens sometimes, but I think far more often they pick games because of the box, or because of word of mouth, or because they played it a friend's house.  (It's not reviews, unfortunately...)  In which case they get more than a sip before they make their buying decision.  Or they rent it, in which case they probably play at least a level before returning it, hopefully getting in that fifteen minutes.

What I'd like to see is videogame gameplay testing move closer to the world of home-use testing.  Since we can't send devkits home with our testers, maybe the solution is to create a comfortable, living room like atmosphere for the testing – sofas and large-screen TVs – minifridges with soda and food.  The testers are encouraged to play as long as they want – say, up to 10 hours if they want to – but they can also take breaks, whenever they want, and quit and go home whenever they want.  The researchers stay out of the way – I'm wont to hover over a tester's shoulder when watching them play, I'd need to force myself to step back behind the mirror.  Although sometimes you're doing gameplay testing – where does it get frustrating? -- in which case they're only given the one game, other times you're doing market testing, and then you'd give them a choice of games to play, only one of which is the game you're really interested in. (Although a publisher with multiple titles could put them all in the hopper to see which one the players like best and therefore is most deserving of marketing dollars.)  At the end you'd ask them how they liked it, and which parts they liked best, and it wouldn't be until they'd rated it that you'd ask them to get more analytical.  Because, as Blink mentioned, getting analytical about your opinions changes them.

Comments

[zachary j. gamedesigner]

Since I've been a tester for just about ever, I'd like to mention that testers not getting treated like cattle such as they do, and I did, at microsoft would be nice. Currently I'm treated as best as I ever have been, though I suspect that is because i don't work on console or PC games right now.

[Jamie Fristrom]

I was talking about "gameplay" or "focus" or "kleenex" testing - the kind where you take preferably casual gamers and show them the game for the first time to get qualitative reactions (too frustrating, not fun, boring, etc), rather than the more-rigorous bug-finding type of testing...
But, hey, as long as I'm dreaming...sofas and big televisions for all of us.

[Amit]

Umm...the "kleenex test"? Is that as dirty as my mind is making it out to be?

[Greggman]

I don't know if there is a solution to this problem but my anecdotal evidence is that people won't give a game the time of day unless they personally want to. Maybe that's obvious but I guess, to give an example of what I mean, countless times there has been a game I loved. I loved it so much I wanted a friend to play so I bought him a copy. He never played it, EVER! This has happen many many times with many different friends. I know (or believe at a 100% level) that these friends would have loved the games if only they would have tried them but they never did because they weren't interested in the first place. Why, I have no idea. Not enough hype? Don't believe they will like it? Assume it's just not for them? I wish I knew what magic button to press to get them to spend 20 *serious* minutes *actually trying* to get into it. If they did and they still didn't like it I'd at least feel they tried.

On the opposite side, and this was arguably more true the younger I was, if they bought the game themselves they fully tried it for hours even if it was not so great. The hurdle of buying commited them to backing up their decision to buy by actually playing the game. As I got older this was probably less true, mostly because some games became more impulse buys as I got more money than when I was younger and actually committing $40~$60 for a game was a bigger deal.

[Raph]

I was actually rather dissatisfied by "Blink," and highly recommend going for Gary Klein's "Sources of Power" which gives a much more coherent picture of the same phenomenon, with better case studies, and better explanations thereof.

[BrianL]

A 'kleenex tester' is a tester who is only used once. Once you have their reaction, anything after is biased, so you start with someone new. The idea is to get peoples initial reactions to the game, which you can only do a single time.

[Carl Pinder]

I think it safe to safe that Jamie and I have frequently disagreed on nearly everything. For example, I don't find his fifteen years of videogame development to be as useful for gut-reaction thinking as he does. Over half that time was with Treyarch under the Don-and-Pete reign, suggesting most of his Blink-style thinking is geared toward that environment. Which doesn't exist anymore.

Secondly, Jamie suggests that we, as an industry, should be using testers to refine gameplay. Again I disagree. That is no more useful than screening a movie to an audience full of set-dressers or property masters. They will surely have an opinion, but it will be highly biased toward their profession and involvement in the industry.

Making games for testers is a sure fire way to make games that the mass market doesn't like and is one of the many reasons I don't play games anymore. In fact, to go a step further, it's more like screening a movie for a bunch of grips from the crew. They're involvement in the product is just enough to skew their perception of the product, but their level of skill, quite frankly, is just enough that they are only grips.

Then again, I haven't had much sleep this week. Maybe Malcolm Gladwell would suggest that affects my gut reaction.

- Carl (no TypePad account) Pinder

[Fran]

You "consume" movies. You "consume" music. You don't "consume" games. You "do" games. Gaming is a hobby. It feels kind of strange to focus-test games. Could you focus test other hobbies? An hour long focus-test of gardening? Let people collect some stamps for 15 minutes to see if they like it?