A lot of people get bugged when I use gamerankings as an indication of the "quality" of a game. I don't blame them: like them, I have a list of a dozen or so games that are rated 90+ that I for the life of me can't see why. And then there's Metal Gear Solid 2: almost nobody I know likes that game; most hate it; and there it is, 90+. Clearly, there are whole groups of people not being represented by gamerankings.
And yet, I persist.
There's a world of difference between saying, "X is a great game" and "I love X." If you say "X is a great game", you're implying that it's objectively great. How can you back up such a claim? Have you developed the one true formal aesthetic for that kind of game? And your aesthetic is correct, whereas the aesthetic of Joe Assjack Game Reviewer is obviously flawed? I know I have my own aesthetic, and it makes for interesting conversations by the water cooler as I try to explain why Galleon is in fact the Best Platformer Evar and I am shocked and alarmed that nobody else can see it. Still, I recognize that I'm just blowing smoke.
Well, what other ways are there to back up the claim? "X is a great game because everybody likes it." We could do surveys. Full-on marketing research type surveys are going to be more accurate than gamerankings and suffer less from the nonrepresentative sample problem. Good data. Unfortunately, most of us don't have access to a marketing research department.
How about online user surveys? Again, you get the nonrepresentative sample problem. You get the group of users that reads a particular site. I actually had high hopes for Amazon.com's user ratings, figuring that Amazon.com is the closest thing we'd get to an online representative sample, but on close inspection I discovered their system is broken: if you look carefully, on some products, you'll find that a lot of people give great reviews in the text and then give the product zero stars. It turns out their interface is flawed, and it's very easy to give a product zero stars by accident. So much for that.
User reviews also suffer from the problem that well-known games get a zillion reviews and games you've never heard of get 14, so you end up with *Hitman: Blood Money* being the #20 game of all time as far as Gamerankings users are concerned.
Okay, how about using sales as an objective measure of game quality? Don't make me laugh.
How about the Game Designer's Current Most Popular Method of Measuring Game Quality: "How much do I and my friends like it?" This is an easy trap to fall into, because my friends and I kind of think alike, and when I say, "Viewtiful Joe is the Best Game Evar" they don't disagree with me. But my friends are an even less representative sample than Gamerankings.
See where I'm going here? As much as they suck, Gamerankings and Metacritic are the best we've got as far as trying to find some semi-objective data as to how the world feels about a game.
Maybe we should just give up. Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game." And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have. So I'm going to get back to work on my Ancient Domains of Mystery/The Sims hybrid, and I'm sure it will sell great. Did I mention it's going to be all text?
Or we can accept Gamerankings as a useful data point.
One other thing: JP pointed out that game reviewers can be bought. Yes, true. But to get that really kick-ass score on Gamerankings you have to buy all of them. Apparently Atari got caught buying some good reviews for Driver 3. It still didn't get Driver 3 a good score on Gamerankings.
JP also pointed out that game reviewers are as susceptible to hype as consumers. Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers. If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?
Back to the title of this entry. What is "quality"?
Total Quality Management would say that Quality is an absence of defects. That doesn't apply to video games, because a video game that has nothing wrong with it is still not necessarily worth playing. This attitude helps lead to the "highly polished turd" phenomenon. Unless we start writing bug reports that say things like, "This game lacks Tim Schafer's genius. To reproduce: 1) play game; 2) note absence of genius."
So I know one thing that Quality is not, but I can't actually tell you what it is. Woah. I just realized it took me an hour to write this entry. Do I really spend that much time blogging?