TDD: I just can't stop talking about it!

Jamie, how closely do you adhere to "official" TDD practices, Soren Johnson asks.

Short answer is: not very.

Although I tried to be rigorous at first, putting a test around every function I wrote or every change I made, the number of false positives was disheartening, and I eased off to where I was just testing things that weren't likely to change.  I slacked off too far, and some bugs crept in that could have been prevented, and now I've found an equilibrium.  And the other coders on the team are less rigorous than I, but I'm not being a fascist about it.

Do we write a test for every bug, at least?  Again - no.  I haven't figured out a good way to test multithread bugs, for example.  I considered a test where we'd spawn the troubled function on two threads - but it would behave unpredictably.  A broken function could pass just by luck. 

Other TDD 'best practices' we ignore:
- no file I/O - your file I/O should be mocked out, because it's too slow.  Bah.  It's not that slow - the easiest way to test saving/loading for us is just to save & load and make sure it's the same.
- fast tests - our test suite takes a few minutes to run.
- only one assert per test - whatever.  The asserts catch their prey whether they're clumped or not, and we can do less copying-and-pasting this way.

A good book, by the way, on TDD, which Noel Llopis recommended to me, is *Working With Legacy Code*.

My recommendation, if your code is buggier than you'd like (and whose isn't?), is to try it!  Be rigorous at first, and then find your sweet spot.

And here's something to think about - although it was "just barely a win" for us - we're a bodaciously small team (one full-time coder, two part-time) writing in a high level language.  Not a very bug-prone environment!  Lots of coders and designers working together in a mixture of C++ and your-game-scripting-language-here are going to introduce a lot more bugs than we do - making TDD all the more desirable.

Comments

[Noel Llopis]

Hey Jamie,

I've been meaning to write a bit more about TDD prompted by your recent posts. But I figured I'd jump in and add a comment.

I think the key difference in how I see TDD is that it's *not* about catching bugs. It's all about design. By taking some of those shortcuts (accessing files directly), you're hurting your design.

I'd recommend TDD not to people with buggy code, but to people with a blob-style code and who can't make changes fast enough.

Also, I'm curious by what metric you consider your code to be "worse" than before TDD. And I mean informally, not a hard metric. Do you think the code you write with TDD is pretty much the same you would write without it? I know that's certainly not the case for me!

--Noel

[Bobo]

I'm curious what, if any, testing you do around some of XNA's classes (like mocking Game) and around some of the more difficult areas to test with automation, such as your renderer.

I'd also be interested in hearing about your engine architecture from a bird's eye view. Your overall structure, what parts of XNA are you using, e.g. are you making any use of the components and services, etc.

[Arne Claassen]

The most value i've gotten out of TDD is in the framework pieces of the code. Things that are complicated and vital subsystems so that fixing a bug in there is always scary. Get a bug reported, write a failing test for it and release the fix, knowing the rest of the tests are watching out that your fix didn't break something else.

But I do keep finding places where TDD is not feasible to implement. Used to be just UI, but now I'm dealing with coordinating multiple processes across multiple machines, where the interaction of these is the thing to test (so no mocking them out) and the setup of all the pieces for an automated test just gets out of hand.

So i totally agree on finding the sweet spot approach.