« Geek | Main | This Pleases Me »

May 10, 2004

Comments

Robert 'Groby' Blum

That's interesting. We have a working hypothesis: "Skills follow a power law". We then conduct an experiment, and find the data contradicts the hypothesis. Conclusion: Since the hypothesis has been postulated by somebody with established credentials, clearly the data or measurement method must be wrong?

I don't think so.

Maybe there is a reason chess has a bell curve, and does not follow a power law? Usually, a power law is closely associated with diminishing returns. Skill improvements tend to get more costly the closer you get to perfection. So, let's try an explanation why chess is different.

Working hypothesis - if it's a Bell Curve, something must violate the law of diminishing returns.

First, there's a threshold skill to chess - you have to know the possible moves, or you cannot play. That wouldn't explain the skewed distribuiton, though. At least I can't think of a good explanation. If at all, it would move things *towards* a power curve.

But second, most of the lower level players might coast by on innate skills. When is the last time you worked on actively improving your chess skills? Most lower-level players just play games and soak up experience, without any active effort.

If you combine that with the idea that intelligence as applicable to chess might not be distributed according to power laws, it will shape the lower part of the chess distribution according to the distribution of innate skills. Assumption: Intelligence follows a Bell Curve.

The power laws would kick in at the place where players actively hone their skills. Since that part would be the trailing part of the bell curve (that is quite similar to a power curve), it might be hard to see the difference.

If the above assumption holds true, I'd predict that the chess bell curve is skewed slightly to the right side. Quick test based on your data - a Bell curve would have an equal number of players lower and higher than 1750, and the averages of that group would have the same distance to 1750. A power law would skew that.

It might seem like nitpicking, but obviously, it has implications for balancing games - difficulty levels become more important the less innate skill can be applied to a game. (Since in a purely power-law distribution, 90% of the people would be left of awful, while in the bell curve only 25% live there)

That might also explain why puzzle games have an easier time appealing to the mass market - it's harder to completely mess up the difficulty of it.

Of course, I might be totally off my rocker here - that has happened before :)

But I'd love to hear if we can see a skewed distribution in your data, or if it's a pure Bell Curve.

Robert 'Groby' Blum

Argh. First paragraph - "since the *current theory*...." , not "since the hypothesis...". What fun editing in a tiny textbox is. :)

And apologies for the length of the comment - I should probably have posted it on my blog and just pointed back here - but I only noticed after I posted.

Factory

"All I can think is that it's something to do with the ranking system itself, which doesn't really measure skill but measures your position relative to other players."
Chess ratings are non-linear, (iirc) for every 100 points you are twice as good. So if you map the linear score of a 1300 player (assuming a 100 player has the linear score of 2) you get 2**13 which is 8192, the linear score of a 2800 (which is well within grandmaster territory) is 2**28 which is 268,435,456.
Graph that and you will see a power law.
The reason why this works is that, unlike the bell curve, the lower half is effectively irrelevant in the power curve, because the upper half is so large.

Robert 'Groby' Blum

The whole topic was too interesting to leave it alone overnight :) - so, after a bunch of googling around, there's the rub:

- Detailed Information on the Elo system is not exactly easy to find. Elo's paper on the system is not available online. (I.e. I'll have to make a trip to the library now. So much for the power of the Internet...)

The best explanation I found so far comes from a Pool Club: http://www.thepoolclub.com/elorank.htm

- As far as I can tell, Elo *built* the system to follow a Bell Curve distribution. That would certainly explain why you found that distribution :)

- Splinter Cell 2 uses the Elo system to rank players

- Gary Kasparov has an ELO of 2812 - I wonder if he plays anonymously on that chess server.

- It's obviously hard to determine if skill follows a power law, since the definition of skill is completely arbitrary. Our choice of ranking system determines the final distribution, at least for comparison based ratings.

Mike Hommel

Sorry to interrupt chess talk, but on the video game related note, I've done the easiness equivalent of Nightmare mode a few times. As you say, it is cheap and easy to add in a Nightmare mode (say, double the enemy life and damage, halve the player's, maybe update the enemies twice per frame if you want to be really nasty), but it's equally as trivial to add in a Baby mode (do the exact opposite!). I did just that with the first patch to my game Dr. Lunatic Supreme With Cheese. It's a remake/new version/upgrade/whatever to Dr. Lunatic, which is a very hard game, truthfully. When Supreme came out, people complained about how hard it was (which I thought was odd, since I think it's slightly easier than Dr. L, with some additions it has like the ability to move while shooting). So, that, along with the conventionally accepted indie wisdom that the more you make a game easier, the more it sells (100% anecdotal, but lots of anecdotes, and never yet a disagreeing voice), I modified the difficulty settings. Now the Easy mode is called Normal (and is default), and it goes up from there. But much more than that, I went all out - I took what I thought was a balanced game and I made the player do double damage, enemies do half damage, and even made the enemies only update on 3 of 4 frames (so 25% slowdown to all their actions).

Well, nobody's complained that it's too easy! And it's really not, that's the interesting thing. It's still a fair challenge to get through. And even still overboard for some new players.

So I guess that's just to say that when you take what the developer feels is right and cut it in 1/4, you still have something that is a challenge to 'real' people. So when you're putting in your Nightmare mode, don't forget to also put in a Baby mode... and it wouldn't be a bad idea to call it Normal mode. The people who are beyond that type of challenge are also the type most likely to change the difficulty settings to their liking.

Ian W. Parker

So Chess could follow a power law if the number of available moves were increased on a regular basis? Is Chess not based on skill? Or is Raph examining skill on the basis of a system that continually pushes that upper limit further and further?

Robert 'Groby' Blum

I wish the link would actually go to Raph's article. It's kind of hard to comment on what he meant without reading it....

Anyways - I think that the chess skill as measured by Elo is meaningless for us in terms of power laws because the system was tuned to not follow a power law.

Actually, I'm thinking that any symmetric, comparison-based system will probably not follow a power law. (It's symmetric, after all - it will be symmetric around the average skill, if I might wager a guess.)

That might have implications on how we might want to judge/score players.

Saul Bottcher

Plotting golf scores doesn't show the distribution of skill, it shows the distribution of *results*.

Chess scores were designed to measure skill, not results, which is why adjustment is relative to your opponent's rating, rather than by some absolute scale (such as "each win is +20 points").

In any activity, there's a function that maps skill levels to results, and it's not necessarily linear. As Robert suggested, all skills below a certain "threshold" could map to a minimal result. This would compact the entire left-half of the bell curve, leaving something that looked exponential.

Another thing to consider is that some activities don't require skill to be successful, only to be *consistently* successful. (Anybody can bowl a strike once in an evening). The lower your skill, the more random your result, and this would distort the bell curve as well.

Anyway, since the distribution of many (most?) human traits follows a bell curve, I would be more inclined to assume that gaming and sporting skills do as well, and attribute any exponential data to the skill-to-results mapping.

Jamie Fristrom

Sorry about the link, I guess Raph's site uses frames or something and so it's hard to link to a particular page - the power law thing was a side point he made in Gaming -> Essays -> Small Worlds.

Saul Bottcher

Reading the actual essay on Raph's site (great read, BTW), I'm convinced "skill" is not the correct term for what's being plotted.

Gretzky's points, Tiger's scores, and Ruth's homers are all the results of skills. The fact that the result is double that of someone else doesn't require that the skill is also double:

Imagine two 100m sprinters, one who runs consistently at 38km/h and the other at 39km/h. You can think of their skill as having a ratio of 39:38, but if they race 100 times, their results will show a ratio of 100:0, or perhaps 99:1 if the fast guy has a false start on a really bad day.


(BTW, I think this is really coming down to a matter of word choice... I have no problem believing that results follow a power curve).

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Jamie's Bragging Rights

  • Spider-Man 2
    The best superhero games of all time Game Informer
    Top five games of all time Yahtzee Croshaw
    Top five superhero games of all time MSNBC
    Top 100 PS2 games of all time Official Playstation 2 Magazine
    1001 Games You Must Play Before You Die Nomination for Excellence in Gameplay Engineering Academy of Interactive Arts & Sciences
  • Schizoid
    Penny Arcade PAX 10 Award
    Nominated for XBLA Best Original Game
    Nominated for XBLA Best Co-Op Game