GameDevBlog

TDD: etc

"I'm curious by what metric you consider your code to be "worse" than before TDD," Noel Llopis asks.  (Cool new logo, btw!)

I said the quality was worse but that was actually a horrible word-choice.  I'll go back and fix that post right now, and nobody will ever know.  What I meant was that my architecture, design, and style are worse.

I expose things I wouldn't ordinarily expose so they can be visible to the test suites.  This could be seen as a problem with C# - there is no 'friend' function, and it's nontrivial to control access to a unit test suite.  On the other hand, with C#, I've been using public a lot more often than I would with C++, because C# supports properties - if I decide that thing I made public shouldn't be exposed, I can turn it into a property that wraps the thing I want to hide.

On the other extreme, TDD is making me wrap and/or abstract things I wouldn't ordinarily wrap and abstract!  Noel would say this is a good thing, because now I'm getting away from my blob-style code.  But I'm a big 'speculative generality is bad', 'premature abstraction is bad' guy, and putting in these abstractions before I actually need them, in my mind, just adds these annoying levels of indirection to the code.

At some point those wrappers will be useful, when we port to other platforms, but I'd rather wait until I'm actually doing the port, and then refactor.

And I guess this is something Noel and I disagree on.  For me, TDD is all about:

- Stopping bugs early when they're cheapest to fix.  Code quality, here, for me, means fewer bugs.

- The euphoria that comes from writing code that does exactly what it's supposed to.  (Chuck Tolman, at Treyarch, writes code so solid that once he's finished it, he usually doesn't ever have to come back to it again, unless somebody wants a new feature.  I called it "Chuckian code".  The ability to do this just comes to him naturally.  I need the test suite.)

- Putting scaffolds in that run quickly so I can iterate on the function I'm working on rapidly.

- Improved understanding of the code in general.

- Having the courage to make fairly scary changes even when deliverables are due.

Bobo wants to know what XNA stuff we mock or fake:

We have a very light wrapper around some of GraphicsDevice.  We wrap SoundDevice.  We wrap the controller, and I'd do that even if we weren't using TDD, because the XNAGS controller class doesn't quite behave the way I'd like.  We wrap the certification and network library.  Most of this is "faking" rather than "mocking", and it's there to prevent the XNAGS stuff from doing, well, any XNAGS stuff in our test suite.  We want to test our code, and so we just want XNAGS to not be there at that time.

Arne mentions how problematic it is to get network games under a test suite.  I haven't done it either, but I do know a guy who works on an online game, and he's the test engineer for the network code.  So he has a master server that runs test scripts on the dev kits to make sure they can host, connect, and play - automated testing, boom.

TDD: I just can't stop talking about it!

Jamie, how closely do you adhere to "official" TDD practices, Soren Johnson asks.

Short answer is: not very.

Although I tried to be rigorous at first, putting a test around every function I wrote or every change I made, the number of false positives was disheartening, and I eased off to where I was just testing things that weren't likely to change.  I slacked off too far, and some bugs crept in that could have been prevented, and now I've found an equilibrium.  And the other coders on the team are less rigorous than I, but I'm not being a fascist about it.

Do we write a test for every bug, at least?  Again - no.  I haven't figured out a good way to test multithread bugs, for example.  I considered a test where we'd spawn the troubled function on two threads - but it would behave unpredictably.  A broken function could pass just by luck. 

Other TDD 'best practices' we ignore:
- no file I/O - your file I/O should be mocked out, because it's too slow.  Bah.  It's not that slow - the easiest way to test saving/loading for us is just to save & load and make sure it's the same.
- fast tests - our test suite takes a few minutes to run.
- only one assert per test - whatever.  The asserts catch their prey whether they're clumped or not, and we can do less copying-and-pasting this way.

A good book, by the way, on TDD, which Noel Llopis recommended to me, is *Working With Legacy Code*.

My recommendation, if your code is buggier than you'd like (and whose isn't?), is to try it!  Be rigorous at first, and then find your sweet spot.

And here's something to think about - although it was "just barely a win" for us - we're a bodaciously small team (one full-time coder, two part-time) writing in a high level language.  Not a very bug-prone environment!  Lots of coders and designers working together in a mixture of C++ and your-game-scripting-language-here are going to introduce a lot more bugs than we do - making TDD all the more desirable.

Gamefest Talks Are Up

We talked a bit in the Microsoft keynote:

http://creators.xna.com/Headlines/presentations/archive/2007/08/25/Gamefest-2007-Keynote.aspx

And then we had our own talk:

http://creators.xna.com/Headlines/presentations/archive/2007/08/25/Gamefest-2007_3A00_-Pre_2D00_Mortem_3A00_-Torpex-Games_2700_-Schizoid.aspx

Apparently, you need IE rather than Firefox to view the links.

We got a few laughs here and there, so there are worse ways you can spend your time than watching us. 

Random side notes:

I actually looked up that Patton quote that I mangled.  The real quote is:  "A good plan, violently executed now, is better than a perfect plan next week."  Makes a lot more sense that way, but I do mean what I said - when it comes to inventing game mechanics, anyway.

I implied it was I who suggested using Illustrator to make levels.  It was Chip.  Just wanted to give credit where credit's due, even if we didn't actually implement that idea.

TDD addendum; end-to-end builds; soak tests

Some asked "Would Schizoid have been better/faster without TDD?"  In other words:  "Was it a win?"  That's what I was trying to say in the last post:  pretty sure it's a win.  Higher quality product (in the fewer bugs sense) developed faster (because a quality / stop-the-line mentality -> long term speed). But it's not a clear, obvious, "how did we ever live without this?" win - and although I feel like it's a win, maybe I'm wrong - impossible to prove, one way or the other, without a change-one-variable, control-group experiment.  (Two identical teams develop Schizoid - one uses TDD, the other doesn't.)

The "safety net" -> "more courage" thing is definitely true.  (And using C# is also a good safety net!)  We're quite fearless about making changes - quite a difference from our Spider-Man days, where we'd lock down days before a big milestone and only allow people to check in one at a time.

Jeff mentioned monkey & integration testing.  Soak tests, monkey tests, and end-to-end builds ARE clear, obvious, "how did we ever live without this" wins.  Maybe not a silver bullet, but definitely a copper one, a best practice you can't afford not to do.  Which is why most people already do it.  If anyone's considering TDD who doesn't do soak tests or have an automated build system in place - get your soak tests and build system going first!

Here's an old article I wrote on how we used to do it.  The key takeaway is that "daily builds" are so five years ago!  It's all about the "end-to-end" build these days. 

We do end-to-end builds with Schizoid, using Parabuild on our build server (took under a day for us to get it up and running with Parabuild - quite a difference from the weeks put into the Spider-Man 2 build scripts), and it runs NUnit rather than running the actual game. We haven't been 100% disciplined about it - the build server broke down and it took us a few weeks to replace it, at one point.  And we run soak tests manually - sometimes with the AI playing the game and sometimes with external random monkey.

Test-Driven Development, A Year Later

So what do I think of TDD after I've been using it for over a year?  (And on two projects, since I abandoned that prototype to work on Schizoid.) 

Frankly, I'm disappointed.  I was hoping for a bigger, more obvious win. 

The number of "false positives" - tests that break because my assumptions changed - greatly outnumbers the number of real bugs really caught. 

Also, they say it improves your code style, design, and architecture  Frankly, my code architecture is worse, as I frequently expose things I wouldn't have exposed so I can get them under test more easily - and other abuses.

And we still end up introducing lots of bugs into the code.  Partly because we're not covering everything (not only are we not covering cosmetic features, "is this sprite/menu item/whatever in the right place?", we're not covering experiments, "Will it be more fun this way?" even if those experiments end up sticking) and partly because you just can't cover all the cases of all the functions, and partly because the tests don't catch multithread problems.  The number of bugs still introduced is an order of magnitude higher than the number of times a unit test correctly catches a real mistake.  (Although a more optimistic way of looking at it is:  unit tests are catching almost 10% of our bugs!)

Not a silver bullet. 

All that said, I'M NOT GOING BACK.

Although the code quality may be worse, my understanding of the code is very high.  Code I wrote months and months ago, code that I've totally forgotten how it was supposed to work...I write tests, and the tests fail, and I have to dig in to understand why the tests fail, and then I have these a-ha moments - that's what the code does, that's what it's supposed to do, hmm, okay.  Ditto with working on the other coders' code.  The result here being hard to measure - I understand the code better, so I'm not writing bugs in the first place! 

I suppose someone might say, "Why not have some discipline and make sure you understand code you're changing?  You don't need TDD for that."  Well, I almost always *think* I understand the code when I change it.  (Unless I'm in one of those, "Eh, let's just see if this works" moods...I suppose I could use more discipline there.)  But do I really?  That's where the TDD really helps.

This is huge, and is a benefit of TDD I don't hear mentioned that often.  Spread the word.

Why Are Games Important? One Of Many Reasons

This post has been a long time coming.

Just one of the many reasons games are important is supposedly education and training.  But games almost never correctly train you in what they ostensibly are supposed to:  SimCity doesn't prepare you for a career of urban planning, Civilization doesn't prepare you for a career of world leader, Guitar Hero doesn't prepare you to be a rock guitarist.  I heard the term "negative training" from a Full Spectrum Warrior guy - meaning a lot of games actually teach you bad habits.  A lot of first person shooters, for example, teach you to run around your enemy in circles, jumping, while you wildly shoot at them.  Probably not great practice for a soldier in the field.

So if games aren't great at teaching specifics, what's the point? 

The point is, let's call it meta-learning:  learning how to learn.  Training in the process of learning.  I have only anecdotal evidence for this belief, but I think that people raised on a diet of videogames are better at picking up new skills than those who weren't.  All of those skills I learned to amateur level...typing, writing, guitar playing, surfing, coding (oh, hey, that one I went pro)...all stem back to playing Defender on the Atari 2600.

Recently read George Leonard's *Mastery* which is all about the process of getting good at something.  Gamers learn a lot of what he preaches, in the field:

- Practice, practice, practice.  People who play videogames know that what looks like an impossible challenge today (the expert level on Guitar Hero?) will be doable months down the road.

- Various kinds of mentoring:  read the FAQ, ask your friends, get a teacher.  I'm a "read the FAQ" type, mostly, myself.  Although some consider hintbooking cheating I usually find when I need to consult a FAQ it's not my fault, it's the game's.  I have two books on surfing, speaking of skills you can't learn from books.  They did help, a little.

- Breaking out of a local minima - getting off a plateau - means you might get worse for a while as you change your approach.  (Switching from all downstrokes to up-down - another Guitar Hero example.)

Something that's occuring to me just now, another thing games teach:

- Keep score.  Measure your progress.  Leaderboards are useful not only because you're competing with everyone else but also because you're competing with yourself, the old you.
This is something I might not have really put into words before, but I knew it on some level, because even with surfing I'd count the number of waves I'd caught in a day.

But one thing Leonard's book talks about that videogames didn't prepare me for:

- Love the plateau.

Leonard describes the "hack" - the guy who gets good enough at something to get by, hits a plateau, and then gets bored and looks for something else to learn.  Hm.  Sounds familiar.  Could videogames be to blame, here?  I think so - though some of the videogames of my youth may teach patience (you have to play the whole game over to get to the level you're having trouble with - talk about practice, practice, practice - if the game doesn't get slowly burned into muscle memory that way, I don't know what'll do it), the videogames of today are engineered to give you the illusion of progress whether or not you're actually getting better.  And there's more than enough games out there - if I get stuck, and gamefaqs isn't any help, I can just turn to a different game.

So there's the quote for the anti-gamers to take out of context.  Despite that one wart on games-as-training-trainers, I'll stick to my guns and say, on the whole, a Good Thing.  In fact, one thing that games have going for them that other media don't - even a bad game will teach these sorts of lessons.  Even a bad game usually has a way of keeping score;  a bad game will usually have strategies that are tougher to learn but allow you to reach new levels of skill;  a bad game will act as a reminder that practice and patience will see you through.  So there's some value in even playing bad games.  (Though no reason to play a bad game when there's so many good ones out there.)  But watch a bad movie...you've just wasted your time.

 

GameFest

Today was a big day for Torpex Games, as we finally showed off Schizoid a little at the Gamefest keynote.  I should have had a link to the webcast up this morning, but I couldn't actually find it. Anyhow, here it is now.

Eventually there'll be slides and archived footage, I understand.

Our website has a new look too! And if you search, you can find some Schizoid gameplay footage there.

More Press

Nickolas Landry has written an in-depth, big-ass article on XNA Game Studio at
http://www.code-magazine.com/Article.aspx?quickid=0709041
in which Schizoid is mentioned many times and I am quoted often.  I give the impression that I unreservedly love Game Studio - the truth is that I reservedly love Game Studio.  We'll be talking at GameFest - as I understand it there's going to be a webcam and archives and slides - I'll try to provide links for all of that stuff when I know more about it, and some of my reservations will come out there.  But I do still love.