Test-Driven Development, A Year Later

So what do I think of TDD after I've been using it for over a year?  (And on two projects, since I abandoned that prototype to work on Schizoid.) 

Frankly, I'm disappointed.  I was hoping for a bigger, more obvious win. 

The number of "false positives" - tests that break because my assumptions changed - greatly outnumbers the number of real bugs really caught. 

Also, they say it improves your code style, design, and architecture  Frankly, my code architecture is worse, as I frequently expose things I wouldn't have exposed so I can get them under test more easily - and other abuses.

And we still end up introducing lots of bugs into the code.  Partly because we're not covering everything (not only are we not covering cosmetic features, "is this sprite/menu item/whatever in the right place?", we're not covering experiments, "Will it be more fun this way?" even if those experiments end up sticking) and partly because you just can't cover all the cases of all the functions, and partly because the tests don't catch multithread problems.  The number of bugs still introduced is an order of magnitude higher than the number of times a unit test correctly catches a real mistake.  (Although a more optimistic way of looking at it is:  unit tests are catching almost 10% of our bugs!)

Not a silver bullet. 

All that said, I'M NOT GOING BACK.

Although the code quality may be worse, my understanding of the code is very high.  Code I wrote months and months ago, code that I've totally forgotten how it was supposed to work...I write tests, and the tests fail, and I have to dig in to understand why the tests fail, and then I have these a-ha moments - that's what the code does, that's what it's supposed to do, hmm, okay.  Ditto with working on the other coders' code.  The result here being hard to measure - I understand the code better, so I'm not writing bugs in the first place! 

I suppose someone might say, "Why not have some discipline and make sure you understand code you're changing?  You don't need TDD for that."  Well, I almost always *think* I understand the code when I change it.  (Unless I'm in one of those, "Eh, let's just see if this works" moods...I suppose I could use more discipline there.)  But do I really?  That's where the TDD really helps.

This is huge, and is a benefit of TDD I don't hear mentioned that often.  Spread the word.

Comments

[joshlee]

This was pretty much my exact experience with TDD. Mediocre as a QC tool, but invaluable as a coder's tool. I found that writing tests first meant that I was writing a client for a class before writing the class itself, which forced me to design interfaces that made sense from the beginning. As a result, the interfaces were so clean that I didn't even really feel bad about exposing extra bits.

The real win was when I had to start working on code that someone else had written. Turns out that running and inspecting unit tests is a much quicker way to learn a class's interface than by digging through twisted source code or out-of-date docs.

[YourMacGames]

Great post and that's very nice to share your experience on this hot topic!
But "false positives", "worst code quality", "90% of bugs not caught"... I will really need to have more positive points than "better code understanding" to spend time on TDD! ;)

[Parveen]

The important question is:

If you didn't employ TDD, would Schizoid be better, worse, or the same?

[greggman]

Or would it have been done faster without TDD?

[Sean Barrett]

What about its role as a safety-net, where you can be more willing to refactor the code because you have a safety net "guaranteeing" that you'll catch refactoring bugs? Even if it doesn't catch any bugs (because you refactor too well), it can still have an effect this way.

[Jeff]

I have to agree with Sean. The TDD cycle (test, code, test, refactor, test) gives you the safety net late in development when you have to refactor or optimize libraries. A failed test immediately sends up a red flag that the refactored function does not work the same way as the new funtion. Even if your assumptions have changed, you'll know to look carefully for any places in the code that may have been depending on that assumption.

In my mind, TDD is all about making sure some of the more simple bugs never get introduced into the code base because of either bad assumptions between programmers or because of hurried bug fixes late in development. Simple bugs waste tester time and they waste coder time. It's better just to make sure they never hit the code base (if that's possible).

Lastly, TDD for me (on a large project anyway) is just one piece of the puzzle. It serves as the initial smoke screen for the stability of your build, but big projects should also think about add integration testing, monkey testing, and the possibility of an automated in engine test (having the game play through several levels automatically). That way, you can make a pretty reasonable guess as to the stability of your build before it even goes to testers, wasting less of their time, and less of the programmer's time.

About code coverage: have you thought about using code coverage tools to let you know what code branches may not be tested? I know I always forget to test failure states / assertion states. A code coverage tool can usually tell you how much of your code isn't being tested.

[Jare]

The benefits you describe seem to be 100% in line with what you should get using the good old assert() in a smart way. asserts are a very good way to document your assumptions, expectations and understanding of what the code should be doing.

asserts have degenerated into a poor (and wrong) technique to validate runtime conditions. To me that is a real shame.

[Jamie Fristrom]

Well...a unit test fails immediately when you violate it, but an assert only fails once that code is later executed, which could be days or weeks later, if you're running a debug build, and you have to rely on whoever happened to see the assert to then report it. Or some sort of automated assert logging system. Which is cool, too. Love those asserts! But not a substitute for unit testing or vice-versa.

[Soren Johnson]

Jamie, how strictly do you adhere to "official" TDD practices? Every time I read up on it, usually from a non-game developer, it just seems so heavy-weight and somewhat backwards that I have a hard time seeing how I could actually use it for game development. I'd love to see some actual examples of it in use.

[martin]

Having shipped games that passed Sony PSX QA with zero bugs... no A, B, C bugs... I can honestly say... TDD is a crock of horse manure. There are far more powerful methods to prevent bugs which are automated. Hand writtening tests, that need to be re-written each time you refactor is anything but automated. Learn to use assert, as the comment by Jare is dead on. Oh, and little secret here... sice you already decouple your render logic from game update logic... run the game at like 1000 fps, with render turned off. Let it play itself overnight, each time you go home. It will find bugs and leaks you never even dreamed of. I'm willing to try TDD... but first, and I've been searching I need to find a project that finished using it correctly, and showed some benefit. And no 20 guys 2 years to make a basic console game is not "successful" unless it is gears of war.