Manager In A Strange Land: Time To Failure

We're all old hands at measuring quality in terms of number of live bugs.

Another metric I sometimes like to use is "Time To Failure" - how long can you play your game before you get to a stop-shipment bug?  (I almost called it "Mean Time To Failure", but when your sample sizes are typically one, saying Mean Time To Failure is...snooty.)

A combination of C#, TDD, soak testing, and having a small team and smallish scope have helped keep Schizoid's "Mean Time To Failure" very high - it's typically hours or days of soak testing before a crash bug hits us.  I haven't seen a project this continually solid since...well, ever, actually.

How can you get your TTF up?  It's fairly simple - play or soak test the game - and make the first failure your highest priority.  Lather, rinse, repeat.  You'll probably find this interesting side effect is progress on this front is exponential - your first bug may come in one minute.  Fix that, your next bug comes in at two.  Then four.  Then eight.  Just statistical likelihoods - if you have a crash bug that happens on what out of every 600 frames, that's going to bite you on average every ten minutes.

Now, it's been a while since TTF has been on our radar - our first soak test crash bug was actually marked Priority 2.  Whoops.  So it's been languishing there, even though it makes the soak test crash within 15-30 minutes.  After fixing that, the next crash bug happens around 45 minutes.  Just fixed that, and we'll see where our TTF is after that.  (Note from later - one at the two hour mark, and the latest run I shut down after four to tweak the soak.)

Why get your TTF up, you may ask?  Why does it matter what order you fix your stop-shipment bugs in?

Partly to look good to your client or publisher or boss, sure.  But also for morale!  If your game crashes a lot, people on the team are going to start to feel doomed.  And why should they fix their bugs when there are all these other obvious bugs in the product?  But if your game *doesn't* crash, then you can start feeling like you have a culture of quality on your team.

Just the other day, we were talking about crash bugs in Schizoid, and Skaff said, "Schizoid doesn't crash."  That was back when our TTF was a little higher...but that's what I'm talking about!  Good feelings all around.

So, I showed you mine.  Who will show me theirs?  Questions for the crowd:  1)  Do you know (or can you guesstimate) your own TTF on the game you have under development?  and 2)  What is it?

Comments

[acemtp]

Since I worked on an online game, I know that the TTF of the server is greater than the TTU (Time To Update), so when the server is down, it's not because of a failure, but because of a game update :)

[Jare]

In one of the projects I worked on a couple of years ago, one day we received 4 QA guys (all experienced types) in order to prepare for a coming demo and soon alpha. It took them 1.5 days to crash the game. Made me very proud of the team.

We didn't have a dedicated build machine, we didn't use TDD, and our iterations were more a result of wanting designers to always have an up to date version of the game to work on, than adherence to a process. Up until that point, we didn't have any QA staff, not even part-time.

The key was that anytime a bug showed up, it became THE focus of that area's programmer. If it was a crash, then it anyone who might possibly be related to it joined in the search. I put on my manager hat and kept buzzing around until someone (a) found the cause, (b) was able to explain it aloud, along with its origins and its plan for fix, (c) fixed it and (d) tested it.

I think all the other stuff are tools to help achieve this goal, but the only true requirement is to build and nurture that kind of culture: bugs die NOW.

Another aspect of our culture was that bugs were talked about in the open, no (or little) blame was present unless it was a clear case of laziness or carelessness. If there was someone clearly to blame, there could be general mockery and fun at the expense of the bug's originator, but never a bad word.

[Mat Noguchi]

Ours has always been never, at least since Halo 1.

MSN

[gaming mouse]

Never? What does that even mean? You've never had a bug that needs to be fixed that could stop your going gold? I don't buy it...