« Rummble | Main | Damage Control »

July 17, 2006

Comments

Paul Sinnett

Best of luck with your new company and your recruitment drive Jamie. But I have a word of caution for you. You mentioned a "ten-fold or thirty-fold difference in productivity" which is almost certainly the 28:1 Grant/Sackman legend. This paper has unfortunately been misreported over the years but what it showed was a difference in productivity between programmers working in an offline environment (writing code on paper then scheduling computer time in 2 hour cycles) and programmers working interactively (as we all do today with our own dedicated computer and development environment.) Recent studies have busted this myth showing that productivity rarely varies more than 2 to 1.

You can check out a paper on that here: http://page.mi.fu-berlin.de/~prechelt/Biblio/varianceTR.pdf

I mention this because I see the 'only hire the best' attitude as a pathology or even a pandemic within the industry. You stated in a previous post your desire for humble beginnings. I think that's a very wise course. But it's one that conflicts with the best of the best.

Jamie

Thanks Paul - the article was enlightening. I don't think it invalidates the "Only Hire The Best" strategy, though - I'll have to post my thoughts on that in a future entry.

Paul Sinnett

I look forward to reading that. I see a lot of people advocating the "only hire the best" strategy (or "only hire the best of the best" as seems to be the fad now.) But I have a feeling that this is one of those things that's so obviously true it's probably false. That is, it's an untested assumption (which, according to my defect log, is a common error in programming too.) If you can find any real evidence, that is anything other than pseudo-science, to back up the strategy in practice, I'd be very interested to read it and your interpretation of it.

The argument (as I understand it) goes that it is worth rejecting a false negative over accepting a false positive. On the face of it this seems self-evident. But we complicate matters by announcing our "best of the best" policy.

First we have reduced our pool of applicants somewhat to those who believe they are the "best of the best." That pool can now be described by its extremes. At one end we have those who are unaware of their own incompetence. Evidence of self-assessment shows a heavy bias toward this end of the pool. At the other end we have those who are not only good but have an ego to match. We probably don't want to hire from either end which means we're going to have to reject a lot of bad and some very good candidates, or maybe even all of them.

As a corollary to the self-assessment research, those who under-estimate their own competence are not only more competent on average but likely to improve faster too. We might think it would be better to hire someone in this pool but they will have excluded themselves from our search since we are looking for the best of the best and from their point of view that isn't them, even if from our point of view it is.

But it doesn't end there. Because we have stated that we're only after the best of best (and any applicant must have believed that of themselves to apply) rejection means insulting the unsuccessful applicants. They will go back to the pool poisoned. They are unlikely to reapply. They are also unlikely to paint the company in a good light to their friends who could well be the good candidates that we are looking for.

I've seen some evidence of the damage this can do to companies. For example, big tech companies (Google, Microsoft, and so on) will announce with a great fanfare that they only hire the very best of the best. This message is then slowly and tacitly removed from their advertising. It usually takes about a year or so for this to happen. After that you begin to hear stories that the company is going on a big recruitment drive and the previous system wasn't working so it was changed.

Nat

Dude, that's totally awesome. If I wasn't so freakin' happy at Toys For Bob, I would totally apply to work with you. I went home recently and found my old copy of DBTS and there was your name! Oh, I just looked at your page. It's will Bill Dugan. He's an awesome guy. I remember him from the Activision recruiting event at my first E3. He tried to hook me up with the zombie GTA game. Was it Dead Rush? best of luck to the both of you!

alienspy77

So assuming you find a 5.6 times as productive as average coder (1*x = 30/x yields roughly 5.6), do you have any plausible data to back up that the (expected?) mathematical expectation of the long-term reward will also be 5.6 times as high as industry average? :)

alienspy77

Also no matter how productive, certain things can't happen without certain insight into things. Swinging on SM2 would have never happened without the anchor finding algorithm that is arguably the core of the mechanic. There is noone at Treyarch who could conceivably have come up with something like that other than me (Andrei writing here obviously). What productivity weight do you attach to that kind of "making it possible" skill? I'm just sayin it's not always a matter of quantity sometimes you can get huge qualitative leaps, and it takes a certain insight into things for that kind of leap to happen. Albeit "productive" people like Jason Bare who can crank out a bazillion lines of code (that works) are also extremely valuable for the company (I don't have the data to back it up but believe Jason is also underpaid at Treyarch relative to his actual contribution to the company. By actual contribution i mean.. consider in retrospect replacing Jason with an "average programmer" or even lets say a 50% higher paid programmer with 50% higher than average "productivity", and see how much less money the game would've made).

Chris Busse

If you think of people like Jason as being in the X percentile of productivity and pay him in the same X percentile of compensation for someone in his position and years of experience, then you have something that makes sense.

As for white papers on productivity of programmers, fuck that. Work with lots of programmers, note who gets shit done and who doesn't. An oversimplification: people who knock out a lot of bugs tend to be the same people who can knock out the hard bugs.

Andre, you're a great programmer, but there were probably 6 or more guys at Treyarch who could have created what you speak of. You were the one who did it, for sure, but never think of yourself as irreplaceable.

Chris Busse

I do have to agree with Paul on the best of the best thing. Don't the best of the best want to work on the best of the best? Two projects in the next 18 months? And not big budget? It feels a little snobbish. Certainly hire the best guys you can find, but announcing we only allow take the creme de la creme feels offputting.

Chris Busse

PPS: feel free to delete any of my comments at anytime ever. Never fear censorship, especially where your company is concerned!

alienspy77

>If you think of people like Jason as being in
>the X percentile of productivity and pay him
>in the same X percentile of compensation for
>someone in his position and years of
>experience, then you have something that
>makes sense.

1000% agree.

In the meantime...
http://www.usatoday.com/tech/gaming/2006-07-19-activision-sued_x.htm?POE=TECISVA

alienspy77

>Andre, you're a great programmer, but there
>were probably 6 or more guys at Treyarch who
>could have created what you speak of. You
>were the one who did it, for sure, but never
>think of yourself as irreplaceable.

Btw do you actually know what you are talking about? :)

alienspy77

>Andre, you're a great programmer, but there
>were probably 6 or more guys at Treyarch who
>could have created what you speak of. You
>were the one who did it, for sure, but never
>think of yourself as irreplaceable.

I certainly understand that I am not irreplaceable. Sure in retrospect once you know the answer you can go back and find a few people who had the right mathematical/optimization background to do it.. It's like a puzzle once you know the answer it's easy to reverse engineer the solution. I can think Paul Edelstein, Dave Cook, Jivko, maybe Chuck. In reality none of those people would've been assigned to do the task. Most likely whoever else would've ended up getting assigned to it would not be qualified for the kind of work because noone really understood the underlying complexity of the problem at the time. So that person would either fail at getting the approach right or at getting it to run fast enough and ended up reverting to something simpler like swinging off thin air. Nothing wrong with that I guess, sm1 shipped like that, some people seem to have liked it more. Trying to stay as personally detached as I can I'd say this is 95% accurate assessment.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Jamie's Bragging Rights

  • Spider-Man 2
    The best superhero games of all time Game Informer
    Top five games of all time Yahtzee Croshaw
    Top five superhero games of all time MSNBC
    Top 100 PS2 games of all time Official Playstation 2 Magazine
    1001 Games You Must Play Before You Die Nomination for Excellence in Gameplay Engineering Academy of Interactive Arts & Sciences
  • Schizoid
    Penny Arcade PAX 10 Award
    Nominated for XBLA Best Original Game
    Nominated for XBLA Best Co-Op Game