« Number of Enemies | Main | Continuing my Ill-Advised Project »

February 06, 2005



You should read "The Zen and Art of Motorcycle Maintenance". Most of the book is one man's pursuit of defining and quantifying what quality IS. It actually caused him to lose his marbles for a temporary time. It's isn't that much about motorcycles. :o)

Nathan McKenzie

On MGS2-type situations:

I REALLY wish all reviewers were forced to go back and re-review games a year or so (or perhaps even longer) after the games are released. That's the metric I'd like to see.

I have a feeling that "technological novelty" would play a much smaller role in those reviews.

Something tells me you'd see the reviews for titles like Castlevania:SotN, Ico, and Katamari Damarcy going up and up and up over time.


"Maybe we should just give up. Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game." And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have."
Correct, doing something that appeals to yourself will make the best game you can possibly make. But would you really make it all text? Assume you had a typical budget for a typical studio.

"JP also pointed out that game reviewers are as susceptible to hype as consumers. Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers. If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"
Well no, but you can only derive any lessons from it if you examine the hype, not the gameplay, and I don't think that we are all interested in hype.

Personally I think if you want to talk about the design of previous games just state that x is good and y is bad, hiding behind other ppls opinions will not improve your arguments.


"JP also pointed out that game reviewers are as susceptible to hype as consumers. Which I'm fine with, as long as those consumers, in return, are as susceptible to hype as the reviewers. If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"

It sure as heck is. If everybody believed that Jane was creating bad luck for the town, and Jane believed it too, does that mean she was really creating bad luck for the town?

It sure as heck doesn't.

You should definitely *not* be fine with illusions, ESPECIALLY if you are trying to find solid principles for making better games. You try to extract some sort of principle from a game that didn't really deserve its acclaim, you're going to find yourself completely out of luck if you can't build the same sense of hype. But even if you can build up the hype, you can expect people to wise up fast.

The danger with people who are held accountable for the profitability of their work is that the pressure can cause them to screw up their entire reasoning process, thinking they can pseudo-scientific methods to come up with the formula for success (or quality, which these guys invariably end up confusing with success anyway).

From Raph's discussion of the 0.7 second thing in Theory of Fun, I don't get anywhere near the impression that we can take it as a "constant" for good jump duration (not to diss on Ben Cousins, but iirc, that was his suggestion). While they used empirical methods to arrive at the 0.7 second figure, from what I understand, they did absolutely no work in terms of proving that this is optimal for every game with real-time jumping in it ever (and Spider-Man is just a blatantly obvious example of where it wouldn't be). You can't skip over that stuff just because you're eager to make money, or eager to have practical results. You have to do the work.


There are some fairly standard measures of quality for many of the disciplines that make up game development:

The art in a game (and the art direction) can be judged on originality, creativity, skill of craftsmanship, and its visual continuity. For example, most people can recognize that Prince of Persia: Sands of Time is a more beautiful game than Prince of Persia: Warrior Within.

The programming in a game can be judged on the lack of bugs, the game’s frame-rate, and the technical innovation. For example, most people can recognize that Doom 3 is doing new things with lights and shadows that have never been seen before. But few people are going to praise a game just because it plays at a steady frame rate and doesn’t need to be patched.

The story and characters in a game can be judged by their dramatic qualities. Are the characters sympathetic, interesting, and believable? Is the plot dramatic? Does it have a beginning, middle and end? Does it have a climax? Does the story move you?

BUT, these three things have very little to do with the quality of gameplay! Instead, the “game” itself has to be judged by a different set of standards. Are the controls intuitive and ergonomic? Is the level design convincing but still fun? Do the enemies behave in a “realistic” manner? Are the weapons balanced? Does the game have a steady learning curve? And probably most importantly… is the gameplay fun and innovative?

I think the average consumer can only judge a game on the first three types of quality. Or at least that’s all he can verbalize. If the gameplay is bad he’s more likely to just say, “this sucks”, or more likely “it’s too hard.” And until we in the industry can do one better that’s what we’ll be stuck with too…

"So I know one thing that Quality is not, but I can't actually tell you what it is. Woah. I just realized it took me an hour to write this entry."

Excellent! That means you are thinking critically about this stuff.

"I know I have my own aesthetic, and it makes for interesting conversations by the water cooler as I try to explain why Galleon is in fact the Best Platformer Evar and I am shocked and alarmed that nobody else can see it. Still, I recognize that I'm just blowing smoke."

No, you just sound like you have absolutely zero confidence in your own, private definition of "quality". If you surrender that entirely and go with GameRankings, you are just letting other people fill your head with their own equally arbitrary garbage.

There's got to be a healthy middle ground. Developers making something they don't want to play but are convinced will sell well is just as catastrophically stupid as the converse situation.

"Maybe we should just give up. Throw out the idea that there can be an objective measure of game quality, and stop saying things like, "Such-and-such is a great game." And instead of trying to make games that appeal to the mass market, we make games for ourselves, because that's all the data we have. So I'm going to get back to work on my Ancient Domains of Mystery/The Sims hybrid, and I'm sure it will sell great. Did I mention it's going to be all text?"

You show that straw-man who's boss, sir.

"If everybody shares the same illusion that such-and-such is a great game...is that really an illusion?"

Define "everybody". A majority? You seem very willing to reject the notion that the rest of the world can be just plain wrong about something, when this has been the foundation of many great human achievements.

"Apparently Atari got caught buying some good reviews for Driver 3. It still didn't get Driver 3 a good score on Gamerankings."

Not a good score, but a *better* one. Better enough to tip it over the edge into "will purchase" territory for at least a few consumers. This is one of the most vile things to happen in recent years in the gaming industry and should have been the final nail in Infogrames' coffin, but almost everyone shrugged it off as "good business". Developers and consumers have internalized the mindset that Anything That Makes Money Is Good, and it is starting to show in the products.

Things don't have to be this way. Consumers need to vote with their wallets more, developers and publishers need to realize that some business practices are corrosive to the soul, and the press needs to do a better job as watchdogs and consumer advocates.

*Sigh*. I'm sorry if I always end up playing the whiny curmudgeon here, but there is just so much stinkin' thinkin' at work in the game industry, and it's only going to change if we make noise about it.

Regarding objective, popularly-defined quality, the important thing to remember is just to never take the bad with the good. GTA: San Andreas did plenty of stellar things, but it also had some ungodly annoying, stupid bits of design. Those stupid bits shouldn't be let off the hook if everything else was good. Likewise, we can recognize that a game that was a commercial failure, maybe even a design failure as well, did something new or worthwhile that's worth improving upon.

Just anything but the stupid, horrible Easy Answers that people too often settle for. "GameRankings == Quality" is an Easy Answer, and it's horribly horribly wrong more often than it's right. That much I can say with universal, objective, god-given surety :)


The thing to remember about GameRankings is that it creates a superset of quality games. I would wager that all quality games have high rankings, but not all games with high rankings are quality.

For example, here are the GameRankings scores of GameSpy’s “25 Most Underrated Games”. I figured this would be a decent collection of games considered to be quality that didn’t have the hype to make them super-hits.

25. The City Building Series (Ceasar, Pharaoh, Zeus) – Approx 82% avg.
24. Wizardry 8 – 84%
23. Blood – 82.5%
22. Um Jammer Lammy – 82.%
21. Operation Flashpoint: Cold War Crisis – 85.5%
20. Codename: Eagle – 62%*
19. No One Lives Forever – 89.6%
18. Jumping Flash! 1 and 2 – 81% avg.
17. Suikoden II – 82.1%
16. Klonoa: Door to Phantomile – 86.7%
15. Persona Series – 86.2% avg.
14. Rez – 81.7%
13. Zork Zero – (unrated)
12. Valkyrie Profile – 84%
11. Clive Barker’s Undying – 84%**
10. Wheel of Time – 81%
9. Planescape: Torment – 90.7%
8. Herzog Zwei – 43%***
7. Grim Fandango – 93%
6. Rocket: Robot on Wheels – 82%
5. Sly Cooper and the Thievius Raccoonus – 85.6%
4. Alone in the Dark – 90%
3. System Shock – 92.5%
2. Virtua Fighter 4: Evolution – 93%
1. Ico – 90.7%

* GameSpy was saying the multiplayer in Codename: Eagle was underrated, not the singleplayer. Thus the discrepancy.
** This game got a 90% score on the Mac, which meant they probably fixed bugs in it.
*** Was listed as underrated because it’s the first RTS.

So, I think this shows that your average quality game is going to get a decent score. BUT, 90% or higher does seem to be reserved for games with a decent amount of hype (and probably a marketing budget to boot).

The GameSpy link is: http://archive.gamespy.com/articles/september03/25underrated/

Darren Korman

personally, I find gamerankings to be a great resource for figuring out what "good" game to play next. I feel very confident when considering a 90%+ rated game that at least half of my play hours won't be garbage.

as a developer, gamerankings is a good "bar" to measure my project against. Like Jamie said, at least with gamerankings you have a decent sized pool of opinions to base an idea from.

Personally, I'll be hella proud the day one of the games I work on breaks 90%.

Keep up the love for Gamerankings Jamie!


Actually, I find the reviews at gamefaqs.com to be much more useful than the average paid reviewer's comments. True there are a few that just hate on a game for no other reason than they can, but there is also on objectivity there that cant be found in most reviews from other places. And the mindless ranting reviews are usually fairly easy to pick out and ignore (or laugh at)

One warning about gamefaqs reviews, anything with Final Fantasy in the title automatically becomes irrelevant to this topic, as fanbois/haters swarm to these titles instantly. For those titles, the message boards there become the primary indicator of the game's quality.

So far, I have had very few misses with picking my next half-played game from the reviews there.


A related editorial about review scores:


Darren Korman

Am I a bad person for skipping to the summary paragraph of that editorial first to see if its even worth reading? 65% for useless whining is exactly correct.

Jason Morrison

Ah, nice to see I wasn't the only one who thought of Zen and the Art of Motorcycle Maintenance when I read this post :) Excellent book, btw, I highly recommend checking it out.

The comments to this entry are closed.

The Games

  • Energy Hook
    3D grappling-and-swinging-and-running-on-walls-and-doing-tricks ... with a jetpack ... for style!

Jamie's Bragging Rights

  • Spider-Man 2
    The best superhero games of all time Game Informer
    Top five games of all time Yahtzee Croshaw
    Top five superhero games of all time MSNBC
    Top 100 PS2 games of all time Official Playstation 2 Magazine
    1001 Games You Must Play Before You Die Nomination for Excellence in Gameplay Engineering Academy of Interactive Arts & Sciences
  • Schizoid
    Penny Arcade PAX 10 Award
    Nominated for XBLA Best Original Game
    Nominated for XBLA Best Co-Op Game