« Inside Game Design | Main | Interview »

January 28, 2008

Comments

Sammy Larbi

Depends on whether you can break something that never worked in the first place...

I'd say that anything untested in the first place is more likely to never have worked, and thus it is likelier to break (and presumably, it went untested because it's harder to test).

If you remove that case, I'm not sure what you'd find. If you keep it, I'm almost certain the hypothesis will be "proven."

jvalenzu

I'd say the more reticent you are to test it, the more likely it's broken.

Jare

It's true that untested software is more likely to be broken, and that hard to test code is more likely to be untested, but I agree that there's also a fundamental correlation.

What makes something "hard" to test? Most likely dependencies. Things you need in order to test it, be it assets, other objects and systems, etc. The more you have, the harder it is to design, build and run the test.

Dependencies also tend to be where a piece of software that "works" one day stops working: it's not that the code has somehow mutated, but that the assumptions and contracts (explicit or not) it relies on are no longer valid, applicable, or happening in practice.

That said, I make plenty of mistakes in my code, which are quickly caught in testing. I don't call that code "broken" because of that, simply "unfinished."

Dave Mark

I believe that is seriously what keeps AI designers and programmers away from autonomous agent-based models and the emergent behavior that results.

Now you CAN run hyper-speed world tests to make sure people don't end up with their nose in the corner or other such behaviors after lengthy periods to time... but at some point, you have to simply trust that the perception and decision models that you have created will all play nice together.

Tynam

I have to agree with this, and I don't think it's as trivial as it sounds.

Cheesy Metaphor: Testing functionality (greeneggs_sidedishes), it isn't possible to test all of the hundreds of possible situations. So you set up test cases for sample situations (spam), (ham) and (cans). You have to leave the majority of situations out, and you don't include case (nand) because it's _much_ harder to test, needing some awkward and difficult calculation, but it's less important an example than the other cases.

This is a sign that you should have included it. The relative awkwardness of testing it is a giveaway that there's some interaction going on here that doesn't happen in the other cases. In the real world where you can't test everything, the test case that you want to leave out because it's complicated and marginal is _precisely_ the one that you're likely to need.

I'm in medical rather than game software testing, so the validation priorities are sometimes a little different anyway, but I ran into precisely this situation in a recent project.

(I agree dependencies are a common reason for something to be 'difficult' to test, but far from the only issue. One common problem we have is situations where functionality is easy to run tests on, but hard to find a standard to test _against_. The AI problem Dave mentioned is an obvious example of this.)

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Jamie's Bragging Rights

  • Spider-Man 2
    The best superhero games of all time Game Informer
    Top five games of all time Yahtzee Croshaw
    Top five superhero games of all time MSNBC
    Top 100 PS2 games of all time Official Playstation 2 Magazine
    1001 Games You Must Play Before You Die Nomination for Excellence in Gameplay Engineering Academy of Interactive Arts & Sciences
  • Schizoid
    Penny Arcade PAX 10 Award
    Nominated for XBLA Best Original Game
    Nominated for XBLA Best Co-Op Game