What is "Quality", anyhow?

A lot of people get bugged when I use gamerankings as an indication of the "quality" of a game.  I don't blame them:  like them, I have a list of a dozen or so games that are rated 90+ that I for the life of me can't see why.  And then there's Metal Gear Solid 2: almost nobody I know likes that game;  most hate it;  and there it is, 90+.  Clearly, there are whole groups of people not being represented by gamerankings.

And yet, I persist.

Why?

There's a world of difference between saying, "X is a great game" and "I love X."  If you say "X is a great game", you're implying that it's objectively great.  How can you back up such a claim?  Have you developed the one true formal aesthetic for that kind of game?  And your aesthetic is correct, whereas the aesthetic of Joe Assjack Game Reviewer is obviously flawed?  I know I have my own aesthetic, and it makes for interesting conversations by the water cooler as I try to explain why Galleon is in fact the Best Platformer Evar and I am shocked and alarmed that nobody else can see it.  Still, I recognize that I'm just blowing smoke.

Well, what other ways are there to back up the claim?  "X is a great game because everybody likes it."  We could do surveys.  Full-on marketing research type surveys are going to be more accurate than gamerankings and suffer less from the nonrepresentative sample problem.  Good data.  Unfortunately, most of us don't have access to a marketing research department.

How about online user surveys?  Again, you get the nonrepresentative sample problem.  You get the group of users that reads a particular site.  I actually had high hopes for Amazon.com's user ratings, figuring that Amazon.com is the closest thing we'd get to an online representative sample, but on close inspection I discovered their system is broken:  if you look carefully, on some products, you'll find that a lot of people give great reviews in the text and then give the product zero stars.  It turns out their interface is flawed, and it's very easy to give a product zero stars by accident.  So much for that. 

User reviews also suffer from the problem that well-known games get a zillion reviews and games you've never heard of get 14, so you end up with *Hitman:  Blood Money* being the #20 game of all time as far as Gamerankings users are concerned.

Okay, how about using sales as an objective measure of game quality?  Don't make me laugh.

How about the Game Designer's Current Most Popular Method of Measuring Game Quality:  "How much do I and my friends like it?"  This is an easy trap to fall into, because my friends and I kind of think alike, and when I say, "Viewtiful Joe is the Best Game Evar" they don't disagree with me.  But my friends are an even less representative sample than Gamerankings.

See where I'm going here?  As much as they suck, Gamerankings and Metacritic are the best we've got as far as trying to find some semi-objective data as to how the world feels about a game. 

Maybe we should just give up.  Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game."  And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have.  So I'm going to get back to work on my Ancient Domains of Mystery/The Sims hybrid, and I'm sure it will sell great.  Did I mention it's going to be all text?

Or we can accept Gamerankings as a useful data point.

One other thing:  JP pointed out that game reviewers can be bought.  Yes, true.  But to get that really kick-ass score on Gamerankings you have to buy all of them.  Apparently Atari got caught buying some good reviews for Driver 3.  It still didn't get Driver 3 a good score on Gamerankings.

JP also pointed out that game reviewers are as susceptible to hype as consumers.  Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers.  If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?

Back to the title of this entry.  What is "quality"?

Total Quality Management would say that Quality is an absence of defects.  That  doesn't apply to video games, because a video game that has nothing wrong with it is still not necessarily worth playing.  This attitude helps lead to the "highly polished turd" phenomenon.  Unless we start writing bug reports that say things like, "This game lacks Tim Schafer's genius.  To reproduce:  1)  play game;  2)  note absence of genius."

So I know one thing that Quality is not, but I can't actually tell you what it is.  Woah.  I just realized it took me an hour to write this entry.  Do I really spend that much time blogging?

Comments

[Paul]

You should read "The Zen and Art of Motorcycle Maintenance". Most of the book is one man's pursuit of defining and quantifying what quality IS. It actually caused him to lose his marbles for a temporary time. It's isn't that much about motorcycles. :o)

[Nathan McKenzie]

On MGS2-type situations:

I REALLY wish all reviewers were forced to go back and re-review games a year or so (or perhaps even longer) after the games are released. That's the metric I'd like to see.

I have a feeling that "technological novelty" would play a much smaller role in those reviews.

Something tells me you'd see the reviews for titles like Castlevania:SotN, Ico, and Katamari Damarcy going up and up and up over time.

[Factory]

"Maybe we should just give up. Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game." And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have."
Correct, doing something that appeals to yourself will make the best game you can possibly make. But would you really make it all text? Assume you had a typical budget for a typical studio.

"JP also pointed out that game reviewers are as susceptible to hype as consumers. Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers. If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"
Well no, but you can only derive any lessons from it if you examine the hype, not the gameplay, and I don't think that we are all interested in hype.

Personally I think if you want to talk about the design of previous games just state that x is good and y is bad, hiding behind other ppls opinions will not improve your arguments.

[Walter]

"JP also pointed out that game reviewers are as susceptible to hype as consumers. Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers. If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"

It sure as heck is. If everybody believed that Jane was creating bad luck for the town, and Jane believed it too, does that mean she was really creating bad luck for the town?

It sure as heck doesn't.

You should definitely *not* be fine with illusions, ESPECIALLY if you are trying to find solid principles for making better games. You try to extract some sort of principle from a game that didn't really deserve its acclaim, you're going to find yourself completely out of luck if you can't build the same sense of hype. But even if you can build up the hype, you can expect people to wise up fast.

The danger with people who are held accountable for the profitability of their work is that the pressure can cause them to screw up their entire reasoning process, thinking they can pseudo-scientific methods to come up with the formula for success (or quality, which these guys invariably end up confusing with success anyway).

From Raph's discussion of the 0.7 second thing in Theory of Fun, I don't get anywhere near the impression that we can take it as a "constant" for good jump duration (not to diss on Ben Cousins, but iirc, that was his suggestion). While they used empirical methods to arrive at the 0.7 second figure, from what I understand, they did absolutely no work in terms of proving that this is optimal for every game with real-time jumping in it ever (and Spider-Man is just a blatantly obvious example of where it wouldn't be). You can't skip over that stuff just because you're eager to make money, or eager to have practical results. You have to do the work.

[Patrick]

There are some fairly standard measures of quality for many of the disciplines that make up game development:

The art in a game (and the art direction) can be judged on originality, creativity, skill of craftsmanship, and its visual continuity. For example, most people can recognize that Prince of Persia: Sands of Time is a more beautiful game than Prince of Persia: Warrior Within.

The programming in a game can be judged on the lack of bugs, the game’s frame-rate, and the technical innovation. For example, most people can recognize that Doom 3 is doing new things with lights and shadows that have never been seen before. But few people are going to praise a game just because it plays at a steady frame rate and doesn’t need to be patched.

The story and characters in a game can be judged by their dramatic qualities. Are the characters sympathetic, interesting, and believable? Is the plot dramatic? Does it have a beginning, middle and end? Does it have a climax? Does the story move you?

BUT, these three things have very little to do with the quality of gameplay! Instead, the “game” itself has to be judged by a different set of standards. Are the controls intuitive and ergonomic? Is the level design convincing but still fun? Do the enemies behave in a “realistic” manner? Are the weapons balanced? Does the game have a steady learning curve? And probably most importantly… is the gameplay fun and innovative?

I think the average consumer can only judge a game on the first three types of quality. Or at least that’s all he can verbalize. If the gameplay is bad he’s more likely to just say, “this sucks”, or more likely “it’s too hard.” And until we in the industry can do one better that’s what we’ll be stuck with too…

[JP]

"So I know one thing that Quality is not, but I can't actually tell you what it is. Woah. I just realized it took me an hour to write this entry."

Excellent! That means you are thinking critically about this stuff.

"I know I have my own aesthetic, and it makes for interesting conversations by the water cooler as I try to explain why Galleon is in fact the Best Platformer Evar and I am shocked and alarmed that nobody else can see it. Still, I recognize that I'm just blowing smoke."

No, you just sound like you have absolutely zero confidence in your own, private definition of "quality". If you surrender that entirely and go with GameRankings, you are just letting other people fill your head with their own equally arbitrary garbage.

There's got to be a healthy middle ground. Developers making something they don't want to play but are convinced will sell well is just as catastrophically stupid as the converse situation.

"Maybe we should just give up. Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game." And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have. So I'm going to get back to work on my Ancient Domains of Mystery/The Sims hybrid, and I'm sure it will sell great. Did I mention it's going to be all text?"

You show that straw-man who's boss, sir.

"If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"

Define "everybody". A majority? You seem very willing to reject the notion that the rest of the world can be just plain wrong about something, when this has been the foundation of many great human achievements.

"Apparently Atari got caught buying some good reviews for Driver 3. It still didn't get Driver 3 a good score on Gamerankings."

Not a good score, but a *better* one. Better enough to tip it over the edge into "will purchase" territory for at least a few consumers. This is one of the most vile things to happen in recent years in the gaming industry and should have been the final nail in Infogrames' coffin, but almost everyone shrugged it off as "good business". Developers and consumers have internalized the mindset that Anything That Makes Money Is Good, and it is starting to show in the products.

Things don't have to be this way. Consumers need to vote with their wallets more, developers and publishers need to realize that some business practices are corrosive to the soul, and the press needs to do a better job as watchdogs and consumer advocates.

*Sigh*. I'm sorry if I always end up playing the whiny curmudgeon here, but there is just so much stinkin' thinkin' at work in the game industry, and it's only going to change if we make noise about it.

Regarding objective, popularly-defined quality, the important thing to remember is just to never take the bad with the good. GTA: San Andreas did plenty of stellar things, but it also had some ungodly annoying, stupid bits of design. Those stupid bits shouldn't be let off the hook if everything else was good. Likewise, we can recognize that a game that was a commercial failure, maybe even a design failure as well, did something new or worthwhile that's worth improving upon.

Just anything but the stupid, horrible Easy Answers that people too often settle for. "GameRankings == Quality" is an Easy Answer, and it's horribly horribly wrong more often than it's right. That much I can say with universal, objective, god-given surety :)

[Patrick]

The thing to remember about GameRankings is that it creates a superset of quality games. I would wager that all quality games have high rankings, but not all games with high rankings are quality.

For example, here are the GameRankings scores of GameSpy’s “25 Most Underrated Games”. I figured this would be a decent collection of games considered to be quality that didn’t have the hype to make them super-hits.

25. The City Building Series (Ceasar, Pharaoh, Zeus) – Approx 82% avg.
24. Wizardry 8 – 84%
23. Blood – 82.5%
22. Um Jammer Lammy – 82.%
21. Operation Flashpoint: Cold War Crisis – 85.5%
20. Codename: Eagle – 62%*
19. No One Lives Forever – 89.6%
18. Jumping Flash! 1 and 2 – 81% avg.
17. Suikoden II – 82.1%
16. Klonoa: Door to Phantomile – 86.7%
15. Persona Series – 86.2% avg.
14. Rez – 81.7%
13. Zork Zero – (unrated)
12. Valkyrie Profile – 84%
11. Clive Barker’s Undying – 84%**
10. Wheel of Time – 81%
9. Planescape: Torment – 90.7%
8. Herzog Zwei – 43%***
7. Grim Fandango – 93%
6. Rocket: Robot on Wheels – 82%
5. Sly Cooper and the Thievius Raccoonus – 85.6%
4. Alone in the Dark – 90%
3. System Shock – 92.5%
2. Virtua Fighter 4: Evolution – 93%
1. Ico – 90.7%

* GameSpy was saying the multiplayer in Codename: Eagle was underrated, not the singleplayer. Thus the discrepancy.
** This game got a 90% score on the Mac, which meant they probably fixed bugs in it.
*** Was listed as underrated because it’s the first RTS.

So, I think this shows that your average quality game is going to get a decent score. BUT, 90% or higher does seem to be reserved for games with a decent amount of hype (and probably a marketing budget to boot).

The GameSpy link is: http://archive.gamespy.com/articles/september03/25underrated/

[Darren Korman]

personally, I find gamerankings to be a great resource for figuring out what "good" game to play next. I feel very confident when considering a 90%+ rated game that at least half of my play hours won't be garbage.

as a developer, gamerankings is a good "bar" to measure my project against. Like Jamie said, at least with gamerankings you have a decent sized pool of opinions to base an idea from.

Personally, I'll be hella proud the day one of the games I work on breaks 90%.

Keep up the love for Gamerankings Jamie!

[Despayre]

Actually, I find the reviews at gamefaqs.com to be much more useful than the average paid reviewer's comments. True there are a few that just hate on a game for no other reason than they can, but there is also on objectivity there that cant be found in most reviews from other places. And the mindless ranting reviews are usually fairly easy to pick out and ignore (or laugh at)

One warning about gamefaqs reviews, anything with Final Fantasy in the title automatically becomes irrelevant to this topic, as fanbois/haters swarm to these titles instantly. For those titles, the message boards there become the primary indicator of the game's quality.

So far, I have had very few misses with picking my next half-played game from the reviews there.

[JP]

A related editorial about review scores:

http://www.ferrago.com/story/5070

[Darren Korman]

Am I a bad person for skipping to the summary paragraph of that editorial first to see if its even worth reading? 65% for useless whining is exactly correct.

[Jason Morrison]

Ah, nice to see I wasn't the only one who thought of Zen and the Art of Motorcycle Maintenance when I read this post :) Excellent book, btw, I highly recommend checking it out.