GameDevBlog

Regex

I know Regex is cool.

But.

Here's the problem:

Everytime I come across a problem where I think, "Regex would be perfect for this."

I have to learn it all over again.

Because these problems only come up once or twice a year for me.

State Machine Hell

I know "all programs are state machines" and blah blah blah but isn't there something better than this recurring idiom:

void Update()
{
  switch( myState )
  {
    case DoingTheFirstThing:
       ...
    case DoingTheSecondThing:
       ...
    case DoingTheLastThing:
       ...
  }
}

To me, these state machines are spaghetti code:  even less readable than a function laced with gotos.  Taking each state and making it a class makes it even worse - now I have to go digging through half a dozen source files to figure out what the code does.  I've done both many times, of course, and always been mystified by my code later.

What's the alternative?  Threads?  Although the code reads cleaner it can't possibly worth the issues multithreading introduces.

I've got nothing, and I'm still grudgingly doing it as above.  Anyone?

TDD triumphs again!

We had a bug once where some enemies weren't moving with their correct starting trajectories.  I wrote a unit test and fixed it.  Today - the test failed!  And it was for real, because of a change I just made that had unforeseen consequences.  If the test hadn't been there, I would have checked in, and the bug would have sat there, probably until Richard or Skaff played the problem levels and noticed - which could easily have ended up being the night before our IGF submission is due...  whew.

TDD: etc

"I'm curious by what metric you consider your code to be "worse" than before TDD," Noel Llopis asks.  (Cool new logo, btw!)

I said the quality was worse but that was actually a horrible word-choice.  I'll go back and fix that post right now, and nobody will ever know.  What I meant was that my architecture, design, and style are worse.

I expose things I wouldn't ordinarily expose so they can be visible to the test suites.  This could be seen as a problem with C# - there is no 'friend' function, and it's nontrivial to control access to a unit test suite.  On the other hand, with C#, I've been using public a lot more often than I would with C++, because C# supports properties - if I decide that thing I made public shouldn't be exposed, I can turn it into a property that wraps the thing I want to hide.

On the other extreme, TDD is making me wrap and/or abstract things I wouldn't ordinarily wrap and abstract!  Noel would say this is a good thing, because now I'm getting away from my blob-style code.  But I'm a big 'speculative generality is bad', 'premature abstraction is bad' guy, and putting in these abstractions before I actually need them, in my mind, just adds these annoying levels of indirection to the code.

At some point those wrappers will be useful, when we port to other platforms, but I'd rather wait until I'm actually doing the port, and then refactor.

And I guess this is something Noel and I disagree on.  For me, TDD is all about:

- Stopping bugs early when they're cheapest to fix.  Code quality, here, for me, means fewer bugs.

- The euphoria that comes from writing code that does exactly what it's supposed to.  (Chuck Tolman, at Treyarch, writes code so solid that once he's finished it, he usually doesn't ever have to come back to it again, unless somebody wants a new feature.  I called it "Chuckian code".  The ability to do this just comes to him naturally.  I need the test suite.)

- Putting scaffolds in that run quickly so I can iterate on the function I'm working on rapidly.

- Improved understanding of the code in general.

- Having the courage to make fairly scary changes even when deliverables are due.

Bobo wants to know what XNA stuff we mock or fake:

We have a very light wrapper around some of GraphicsDevice.  We wrap SoundDevice.  We wrap the controller, and I'd do that even if we weren't using TDD, because the XNAGS controller class doesn't quite behave the way I'd like.  We wrap the certification and network library.  Most of this is "faking" rather than "mocking", and it's there to prevent the XNAGS stuff from doing, well, any XNAGS stuff in our test suite.  We want to test our code, and so we just want XNAGS to not be there at that time.

Arne mentions how problematic it is to get network games under a test suite.  I haven't done it either, but I do know a guy who works on an online game, and he's the test engineer for the network code.  So he has a master server that runs test scripts on the dev kits to make sure they can host, connect, and play - automated testing, boom.

TDD: I just can't stop talking about it!

Jamie, how closely do you adhere to "official" TDD practices, Soren Johnson asks.

Short answer is: not very.

Although I tried to be rigorous at first, putting a test around every function I wrote or every change I made, the number of false positives was disheartening, and I eased off to where I was just testing things that weren't likely to change.  I slacked off too far, and some bugs crept in that could have been prevented, and now I've found an equilibrium.  And the other coders on the team are less rigorous than I, but I'm not being a fascist about it.

Do we write a test for every bug, at least?  Again - no.  I haven't figured out a good way to test multithread bugs, for example.  I considered a test where we'd spawn the troubled function on two threads - but it would behave unpredictably.  A broken function could pass just by luck. 

Other TDD 'best practices' we ignore:
- no file I/O - your file I/O should be mocked out, because it's too slow.  Bah.  It's not that slow - the easiest way to test saving/loading for us is just to save & load and make sure it's the same.
- fast tests - our test suite takes a few minutes to run.
- only one assert per test - whatever.  The asserts catch their prey whether they're clumped or not, and we can do less copying-and-pasting this way.

A good book, by the way, on TDD, which Noel Llopis recommended to me, is *Working With Legacy Code*.

My recommendation, if your code is buggier than you'd like (and whose isn't?), is to try it!  Be rigorous at first, and then find your sweet spot.

And here's something to think about - although it was "just barely a win" for us - we're a bodaciously small team (one full-time coder, two part-time) writing in a high level language.  Not a very bug-prone environment!  Lots of coders and designers working together in a mixture of C++ and your-game-scripting-language-here are going to introduce a lot more bugs than we do - making TDD all the more desirable.

Gamefest Talks Are Up

We talked a bit in the Microsoft keynote:

http://creators.xna.com/Headlines/presentations/archive/2007/08/25/Gamefest-2007-Keynote.aspx

And then we had our own talk:

http://creators.xna.com/Headlines/presentations/archive/2007/08/25/Gamefest-2007_3A00_-Pre_2D00_Mortem_3A00_-Torpex-Games_2700_-Schizoid.aspx

Apparently, you need IE rather than Firefox to view the links.

We got a few laughs here and there, so there are worse ways you can spend your time than watching us. 

Random side notes:

I actually looked up that Patton quote that I mangled.  The real quote is:  "A good plan, violently executed now, is better than a perfect plan next week."  Makes a lot more sense that way, but I do mean what I said - when it comes to inventing game mechanics, anyway.

I implied it was I who suggested using Illustrator to make levels.  It was Chip.  Just wanted to give credit where credit's due, even if we didn't actually implement that idea.

TDD addendum; end-to-end builds; soak tests

Some asked "Would Schizoid have been better/faster without TDD?"  In other words:  "Was it a win?"  That's what I was trying to say in the last post:  pretty sure it's a win.  Higher quality product (in the fewer bugs sense) developed faster (because a quality / stop-the-line mentality -> long term speed). But it's not a clear, obvious, "how did we ever live without this?" win - and although I feel like it's a win, maybe I'm wrong - impossible to prove, one way or the other, without a change-one-variable, control-group experiment.  (Two identical teams develop Schizoid - one uses TDD, the other doesn't.)

The "safety net" -> "more courage" thing is definitely true.  (And using C# is also a good safety net!)  We're quite fearless about making changes - quite a difference from our Spider-Man days, where we'd lock down days before a big milestone and only allow people to check in one at a time.

Jeff mentioned monkey & integration testing.  Soak tests, monkey tests, and end-to-end builds ARE clear, obvious, "how did we ever live without this" wins.  Maybe not a silver bullet, but definitely a copper one, a best practice you can't afford not to do.  Which is why most people already do it.  If anyone's considering TDD who doesn't do soak tests or have an automated build system in place - get your soak tests and build system going first!

Here's an old article I wrote on how we used to do it.  The key takeaway is that "daily builds" are so five years ago!  It's all about the "end-to-end" build these days. 

We do end-to-end builds with Schizoid, using Parabuild on our build server (took under a day for us to get it up and running with Parabuild - quite a difference from the weeks put into the Spider-Man 2 build scripts), and it runs NUnit rather than running the actual game. We haven't been 100% disciplined about it - the build server broke down and it took us a few weeks to replace it, at one point.  And we run soak tests manually - sometimes with the AI playing the game and sometimes with external random monkey.

Test-Driven Development, A Year Later

So what do I think of TDD after I've been using it for over a year?  (And on two projects, since I abandoned that prototype to work on Schizoid.) 

Frankly, I'm disappointed.  I was hoping for a bigger, more obvious win. 

The number of "false positives" - tests that break because my assumptions changed - greatly outnumbers the number of real bugs really caught. 

Also, they say it improves your code style, design, and architecture  Frankly, my code architecture is worse, as I frequently expose things I wouldn't have exposed so I can get them under test more easily - and other abuses.

And we still end up introducing lots of bugs into the code.  Partly because we're not covering everything (not only are we not covering cosmetic features, "is this sprite/menu item/whatever in the right place?", we're not covering experiments, "Will it be more fun this way?" even if those experiments end up sticking) and partly because you just can't cover all the cases of all the functions, and partly because the tests don't catch multithread problems.  The number of bugs still introduced is an order of magnitude higher than the number of times a unit test correctly catches a real mistake.  (Although a more optimistic way of looking at it is:  unit tests are catching almost 10% of our bugs!)

Not a silver bullet. 

All that said, I'M NOT GOING BACK.

Although the code quality may be worse, my understanding of the code is very high.  Code I wrote months and months ago, code that I've totally forgotten how it was supposed to work...I write tests, and the tests fail, and I have to dig in to understand why the tests fail, and then I have these a-ha moments - that's what the code does, that's what it's supposed to do, hmm, okay.  Ditto with working on the other coders' code.  The result here being hard to measure - I understand the code better, so I'm not writing bugs in the first place! 

I suppose someone might say, "Why not have some discipline and make sure you understand code you're changing?  You don't need TDD for that."  Well, I almost always *think* I understand the code when I change it.  (Unless I'm in one of those, "Eh, let's just see if this works" moods...I suppose I could use more discipline there.)  But do I really?  That's where the TDD really helps.

This is huge, and is a benefit of TDD I don't hear mentioned that often.  Spread the word.

GameFest

Today was a big day for Torpex Games, as we finally showed off Schizoid a little at the Gamefest keynote.  I should have had a link to the webcast up this morning, but I couldn't actually find it. Anyhow, here it is now.

Eventually there'll be slides and archived footage, I understand.

Our website has a new look too! And if you search, you can find some Schizoid gameplay footage there.

Stupid C# Tricks: Dealing With Non-Generational Garbage Collection

Mostly for Nathan:

Better men than me have already discussed non-generational GC with XNA Xbox, and a few things you can do to deal with it. 

Trying to build upon their work - your best buddy is going to be the CLRProfiler.  Once you figure out how to use it, it's an awesome tool that tells you what's allocating the most garbage, when and where.

Once you've found trouble spots, there's a variety of things you can do.  Using structs instead of classes is often what you want - structs are put on the stack;  and an array of structs is just one object.

If there are moments in your game where pauses make sense, (when the user hits the pause button, for example - or in a turn-based game, the beginning of every turn, maybe?) - that's a good time to force a garbage collection.

Earlier I said we're using fixed pools.  Not anymore.  What we've done is, we've written our own SpecialNew and SpecialDelete functions for some of our problem classes.  Delete doesn't delete;  it just puts the object at the front of a linked list.  Thus, the object never gets collected.  New first checks that linked list to see if an old object is available, clears it out and hands it back, only actually calling the C# new if no object is available.  Voila, no garbage collection, and no fixed-pool restrictions on number of objects allocated.  It seems so simple once you've seen it, I don't know why I didn't think of it;  it was one of our other programmers who showed me the light.

One problem with it (and fixed-pools) is suddenly you lose the advantage of a constructor that is guaranteed to set your object to a known initialized state.  Just like with C++, if you forget to initialize a variable in your SpecialNew...then that object's behavior is undefined.  We get around this with reflection.  We don't actually reflect through every object when it's SpecialNewed to initialize everything...that would probably be too slow - what we do, is in our debug builds, we reflect through the object to make sure it matches a canonical initialized object that was newed the regular way, and fire an assertion if it doesn't.  This has caught a handful of omissions.

One might ask: why use C# to make our game if one of the most useful features of C#, the garbage collection, shouldn't be used on the Xbox?  Well, in C++, if you forget to delete an object when it goes out of scope, you've got a memory leak that will eventually crash your game.  With C# on the Xbox, you do the same thing, you've got a memory "leak" that will eventually cause a garbage collection and make your game drop a frame and maybe have a noticeable hitch.  I'll take the latter.  It allows you to fix on an as-needed basis:  you can play fast-and-loose with the heap for your early product development, and then fix your problem classes as needed.

As for delegates and closures:  I love, love, love delegates and it's the main reason I like working with C#.  If I found out I shouldn't use them on the Xbox I would go into a Black Hole of Despair.  That said, you do have to massage them sometimes.  A closure, obviously, will put an object on the heap every time it's created.  So if you, say, create a closure and pass it to a function for every entity every frame, that's probably going to create too much garbage.  Sigh.  A not-so-obvious case is if you pass a member function to a delegate parameter, that member function gets...wrapped...somehow...(I really need to look deeper into it, there are some websites that discuss it in high detail)...and so it puts garbage on the heap also.  Here you can protect yourself by creating a static delegate variable and assigning the member function to it...that way the wrap only happens once and never gets collected.  Still, I'm happy to play fast-and-loose with delegates and closures and then fix them once I discover the problem cases.

Brief plug - I'll be speaking at the next GameFest about this stuff and more!