TDD theory

Here's a theory for y'all:

"The stuff that's harder to test is the stuff that's likelier to break."

More than once I've written a bunch of tests and skipped some case because it was Special.  Of course, Special often means Touchy, and later it'll turn out to be that one case that breaks.

Comments

[Sammy Larbi]

Depends on whether you can break something that never worked in the first place...

I'd say that anything untested in the first place is more likely to never have worked, and thus it is likelier to break (and presumably, it went untested because it's harder to test).

If you remove that case, I'm not sure what you'd find. If you keep it, I'm almost certain the hypothesis will be "proven."

[jvalenzu]

I'd say the more reticent you are to test it, the more likely it's broken.

[Jare]

It's true that untested software is more likely to be broken, and that hard to test code is more likely to be untested, but I agree that there's also a fundamental correlation.

What makes something "hard" to test? Most likely dependencies. Things you need in order to test it, be it assets, other objects and systems, etc. The more you have, the harder it is to design, build and run the test.

Dependencies also tend to be where a piece of software that "works" one day stops working: it's not that the code has somehow mutated, but that the assumptions and contracts (explicit or not) it relies on are no longer valid, applicable, or happening in practice.

That said, I make plenty of mistakes in my code, which are quickly caught in testing. I don't call that code "broken" because of that, simply "unfinished."

[Dave Mark]

I believe that is seriously what keeps AI designers and programmers away from autonomous agent-based models and the emergent behavior that results.

Now you CAN run hyper-speed world tests to make sure people don't end up with their nose in the corner or other such behaviors after lengthy periods to time... but at some point, you have to simply trust that the perception and decision models that you have created will all play nice together.

[Tynam]

I have to agree with this, and I don't think it's as trivial as it sounds.

Cheesy Metaphor: Testing functionality (greeneggs_sidedishes), it isn't possible to test all of the hundreds of possible situations. So you set up test cases for sample situations (spam), (ham) and (cans). You have to leave the majority of situations out, and you don't include case (nand) because it's _much_ harder to test, needing some awkward and difficult calculation, but it's less important an example than the other cases.

This is a sign that you should have included it. The relative awkwardness of testing it is a giveaway that there's some interaction going on here that doesn't happen in the other cases. In the real world where you can't test everything, the test case that you want to leave out because it's complicated and marginal is _precisely_ the one that you're likely to need.

I'm in medical rather than game software testing, so the validation priorities are sometimes a little different anyway, but I ran into precisely this situation in a recent project.

(I agree dependencies are a common reason for something to be 'difficult' to test, but far from the only issue. One common problem we have is situations where functionality is easy to run tests on, but hard to find a standard to test _against_. The AI problem Dave mentioned is an obvious example of this.)