TDD

The last few days I've blown the dust off of a prototype and gotten it up-and-running again to discover that it's embarrassingly buggy.  In fact, as I've been trying to resurrect the project I've been spending more time fixing bugs than making it cooler.  One bug in particular I was having trouble reproducing the situation in which it occurred.  And then I thought -
Maybe this is a good time to try out Noel Llopis's free unit test framework!
I made this decision around 7:30 tonight.  Since then, I downloaded the stuff, compiled and ran the sample, got it working with my project (which required seperating most of my game code into a static library, a surprisingly painless task - that would have really kicked my ass ten years ago), wrote the code to set up the situation I was trying to repro (laced with CHECK() statements), and wrote the test.
Voila.  My fix worked. 
But is it fixed in the wild?  I can't be 100% sure, but the code isn't *that* crazy, so I'm confident.
So this was all done before 10:30.  Three hours.  And now I have a testing framework.  One of the things I resurrected with the project was its old testing framework...which didn't compile.  (I found a note on my buglist:  "Unit testing framework should actually...work.")  So, going forward, I'm going to give test driven development a try. 
I haven't had the greatest luck with unit testing in the past.  I've even mentioned it before.  It's not one of those cut-and-dried things, like source control or bug tracking software where once you implement it you wonder how you ever lived without it.  For me, at least.  Even on Clinton Keith's webpage he mentions that velocity went down when they implemented it.  In theory, it will pay for itself in the back end.  In theory, I will get my prototype to the point where I'm spending more time making it cool and less time fixing legacy bugs.  I sure as hell hope so, because the bug-fixing part of making a prototype is no fun.

Comments

[greggman]

Tell us how it goes. I'm excited to try TDD as well in my next project. It certainly seems like it would be awesome for library / engine development. I'm pretty skeptical that it would be good for "game" development. Well, "good" is probably a loaded word. I guess rather I'm not sure it would be worth it. I just finished up my latest project with no TDD and I can only think of 2 or 3 bugs during the entire project that TDD would have caught and they didn't take anytime to find so clearly even a 5% hit for TDD let alone a 40% hit for TDD would not have been worth it.

At the same time, toward the end we were scared to make changes to the code and if TDD gave us the piece of mind to do it that might be worth something but I'm not sure TDD would have caught the type of bugs we were worried about creating.

As an example like many engines we used a signal system. One object had a both "go forward" and a "go backward" input. Signals could be used in a polling mode or a callback mode. In polling mode it was possible in some extremely rare cases for both of those signals to get set before the next polling intervol at which point the object in question had no idea which way to go, forward or backward, and if it choose wrong players could get stuck waiting for something that never would never happen. We fixed that by changing that object to use the callback method so it could know which signal came last. That fixed that bug in particular but we had no way of knowing if there were any places that counted on the old way of working and I'm not sure TDD would have found that problem. Maybe approaching it from a user point of view that TDD encourages might have helped but we knew about the issue (two signals coming in between polling intervols) when we coded it and we knew which of the two signals in that case took priority. We didn't think that was going to be a problem so I'm not sure we would have put in test that would have found the issue.

One thing "Practices of an Agile Developer" assumes is that the code in never done. That might be true for an engine and it's certainly true for a business apps but for games that't assumption is false. At least for console games the code is 100% done the moment it ships. There are no updates for PSP/PS2/GC/DS games. Xbox and 360 might cop out by supporting patches through live but not everyone subscribes to live so for most Xbox/360 games they also have a 100% done point as well.

[Jare]

The part that drives me crazy about TDD is when you need to test code that relies on a number of other systems, assets and the like. I write tests and use cases during the design phase, but I can never convince myself to go the full TDD route, and just end up with the old contract / assert way of ensuring robustness.

(I got a strange error in xml form while trying to post this)

[Noel Llopis]

Jamie, glad you found UnitTest++ useful.

Applying TDD to an existing codebase can be quite a challenge, and Jare's comment is quite valid. But once the codebase has changed enough, or if it was built with TDD from the ground up, then you don't have to worry about testing code that relies on a number of other systems... because you didn't design it that way.

I keep saying it, but TDD for me is not about making sure you have no bugs, but about designing your code from the outside. And giving you a simple, reliable, and incremental way of changing your code.