GameDevBlog

My Year In Positive Psychology 3: Notes on Creativity, Inc and Feedback

Continued from Part 1 & Part 2.

So now I'm on to Creativity, Inc (I may be consuming these faster than I'm able to blog about them) and - back the truck up! Pixar's CTO invented the z-buffer!? Why, he must be the same Catmull of Catmull-Rom splines! Respect!

I'm enjoying the book a great deal both as storytelling and as an advice column but there is of course no cure for werewolves here - a lot of the stories sound very similar to corporate game development. "No matter what we try our movies always suck at first and we have to go through hell in order to fix them." I was particularly horrified by the story that, during the Toy Story 2 hell, a Pixar employee forgot to drop his kid off at daycare on the way to work one morning during the Toy Story 2 hell, left him in the car, and the child was unconscious by the time he remembered. Would that employee have made the same mistake if he wasn't overworked? No way to know for sure, but just imagining the possibility is terrifying.

That said, it's clear they're trying and I just checked Pixar on Glassdoor and it does seem like people are happy there. (4.0. So, it's not Microsoft, but it's pretty good. On that note, this was my first week at Microsoft! I am stoked.) 

What really makes me think is the section on critical feedback. I have a troubled relationship with criticism. I thought it was essential up until I was in my forties (in fact, back when I ran a creative writing workshop in Venice I assumed everyone was there because they wanted the gut punch of brutal criticism and had a 'no defending your work' rule. Membership dwindled to a fraction of its former glory under my enlightened leadership. Go figure.) and then started to get tired of people telling me the myriad of ways my games could be better or I could be a better person... more ways than I could ever possibly address! And when I read Johnstone's Impro, where he rails against criticism as a destroyer of art and killer of babies, it resonated with me deeply. Yes, having an art tutor in middle school was why I stopped making art. Yes, instead of working on my novel for ten years until it was 'perfect' I could have written a half dozen novels and one of them was likely to be just as good as that 'perfect' one and surely would have sold more copies. Yes, there is only a slightly-above-average chance that even addressing someone's criticism will actually make a product better. Yes, we wouldn't have abandoned the stealth gameplay from Spider-Man 3 that IMO presaged the Arkham games if internal critics hadn't pounced on it too early.

And to top it all off it just hurts. It is not fun, and if you fill your days with it it probably means your days or not fun. I unfortunately forget which of the books I've read lately mentioned it--perhaps it was The Happiness Advantage?--but there's also some negative literature about performance reviews, those days that come once or twice a year where we shit on each other and the whole company needs to spend a few days recovering from the morale hit. Cerny talked about the 'stench of failure' that can affect projects that are revealed too early. Wow, I can just go on about it, can't I? Side note: my girlfriend and I pretty much have a tacit keep-your-mouth-shut-when-you-want-to-criticize-unless-it's-absolutely-essential understanding and we've been together for years, drama free. Best relationship ever. :)

And then Ed Catmull comes along and says that at Pixar, their brain trust of executives that candidly points out flaws in their movies is absolutely essential and one of the key ingredients of Pixar films. 

Huh.

It clearly works for Pixar. 

So what's up?

There are a few things about the brain trust review at Pixar that Catmull talks about that have been missing from the criticism I'm familiar with. Catmull acknowledges that criticism causes a possibly instinctual defensive reaction and that can make soliciting criticism like a trip to the dentist: necessary but painful. 

I'm going to have to re-listen to the chapter, but some things stood out to me that are different about the Pixar way. 

• One thing was it is trusted peers giving the criticism, not necessarily bosses. (Obviously Lasseter is in charge and involved in these meetings but he's earned it, yeah?) These are other successful movie-makers; not a bunch of executives from Activision HQ who have never made a game before on a strafing run to say 'ur combat sux.' 
• Another thing is that the creator knows the criticism is coming and can not only 'clench up' but gets to choose what they submit for criticism. They're not ambushed with 'hey, we need to talk to you about your level which we pulled down from perforce and tried last night without you knowing' or whatever.
• The creator gets to make some disclaimers beforehand. "This isn't ready, I know this still needs work, but this area I think is ready for judgment." So time isn't wasted beating any already dead horses. I bet this is more important than it sounds. When I get criticized for something I already know is wrong I feel this nearly uncontrollable urge to point out 'yeah, I know that needs work' to rebalance the status equation.
• It's understood that it is a process to make the product better, not a warning shot prior to cancellation. The product is being protected.
• Perfect world it's a criticism of the work and not the person but that's easier said than done. I think we probably need constant reminders of 'you're awesome' and to paraphrase Jake the Dog, 'dude, sucky work is the first step to sorta good work.'

So I think I'll let some criticism back into my life. Hell, I'll have to now that I don't work for myself anymore. But I will try my damnedest to remember the above stuff when giving criticism, and when receiving I'll try to guide the critical along the same path. "Would you like feedback on your criticism?" :)

My Year in Positive Psychology, Part 2: Notes on Essentialism

Continued from Part 1.

The last book I attempted to digest was Essentialism by Greg McKeown.

In a nutshell, Essentialism says to cut your life down to the bone, keeping only what is absolutely essential. 

Some positive psychology books are filled with common sense wisdom and I find myself asking 'why am I reading this stuff when most of it is obvious?' but I have to remind myself that the point is that being constantly reminded with positive words and helpful advice keeps it conscious and keeps me acting on it.

So while I think of myself as pretty good at optimizing my time and prioritizing my values this acted as a reminder that maybe it would be good to give myself a priority tune-up.

Is spending time on Twitter essential? As I mentioned before I've definitely gotten some good stuff out of it over the years, but has that made up for the time I spend on it each day, which is often 15 minutes on the PC and I'm not sure how much on my phone?

What about blogging? This is actually the second time I'm writing this - the first time I thought I saved a draft and did not. I went years without so at some level that makes it clear it's not essential.

And what about Dungeon Life, the game I went to keep working on as a side project just to keep my hand in on the game design side? It looks like I just can't bring myself to let go of that one. 

The fact that I'm writing this--and when I'm done, will undoubtedly tweet that it's here to read--shows that I'm not bringing myself to make any deep cuts. The way I'm rationalizing it is that blogging for me is really a way of thinking. By composing blog posts in my head and getting them down on paper I am wrestling with my thoughts and assimilating them. Publishing them for other people to read is actually a side effect. (But thanks for reading anyway!) So it doesn't actually matter that I lost my original draft of this article. :)

So on the whole I agree with Essentialism, think it was good to have the reminder to check my 'Essentialist Health', and in particular like McKeown's criteria for determining what is essential (most books that say focus don't really give you a good guide towards what to focus on): those things in which you can have the greatest impact.

There was one thing that really gave me pause though. (Allow me to indulge my negativity bias as I talk about the one part I disagreed with. :P) He seems to be a strong advocate for passing up questionable opportunities, and quotes a passage from Chiksentmihalyi's Creativity in which Peter Drucker turns down an interview with Chiksentmihalyi saying that he is an admirer of Chiksentmihalyi's work but has to decline because what keeps him productive is a laserlike focus on a small number of things. 

I personally believe luck is something of a skill rather than magic: the sort of person who takes advantage of opportunities is 'lucky'. If I turned down nonessential opportunities there are speaking engagements I wouldn't have taken and, therefore, foreign cities I never saw. Sixty Second Shooter Prime for the Xbox One would never have happened, which was, in a way, the biggest success story of my indie career. And I wouldn't have gotten to collect an income for playing with Roblox for the past couple of years. 

To put it another way, I think seizing promising opportunities is essential. For me anyway. Keeping an open mind, saying 'yes' to things, and at least tasting them before making a decision and giving a 'no' verdict brings happiness and variety.

As for Dungeon Life, I think keeping my range wide and continuing to be a generalist will be beneficial to my career as a whole and not just a time-waster. So as I was consuming Essentialism I thought about what the most essential thing to work on with Dungeon Life would be in the last couple weeks before I start at Microsoft. My conclusion? I spent the time playing with my kids.

My Year in Positive Psychology, Part 1

Why do you think I named my company Happion Labs? My first and foremost priority with it was happiness, not money. Happiness unfortunately doesn't always come that easily to me (though a lot easier to me than some), and Happion Labs in the end was something of a failed experiment that I maybe stuck with longer than I should have. And as I gave up on it and relegated game development to the category of 'hobby' or 'side project' I went through the sads.But the year wasn't all bad; for one thing Daniel Cook turned me on to this habit-forming thing at the beginning of the year:

First week of habit forming complete using @ncasenmare's post-it habit forming technique. So good. https://t.co/sBgEOeiJA6

— Daniel Cook (@danctheduck) January 5, 2019

And I've been doing it ever since, using the Habit Tracker app for Android rather than post-its. When I went into it it was with just a couple of fairly well established habits (running a few times a week and taking supplements every day) but at this point I'm also doing spaced repetition learning with ankidroid, meditating 9 minutes a day (sounds so much easier than 10), getting up earlier, drinking less, and getting at least 7 hours of sleep a night. (I'm shooting for 8 but my body just won't listen.) 

I also went back into therapy this year. (Which scares me a bit to write; I guess I'm still worried about the therapy stigma. But here I go, trying to normalize mental illness!) I didn't stick with it for long because I didn't feel like I was getting that much out of it, but my therapist gave me one stellar life-changing piece of advice which was to continually listen to positive psychology audiobooks. I was listening to fiction while running, driving, and doing chores and when I swapped in audiobooks it 'gave me a diet of positivity' as she put it.

A couple of these audiobooks were magical thinking, the modern equivalent of Think And Grow Rich (I'm talking particularly about Joe Dispenza here, don't tell my ex-wife because she's the one who recommended him) but most of what I listened to was inspiring, motivating, and affirming. I made various decisions and strategies this year that directly came out of these books.

My best source for a list of positive psychology books is Scott Crabtree's blog:

If you want to read a book that will not just entertain you, but might just make you happier for the rest of your life, I suggest these books.

Have you read any of these? What did you think? What books am I missing from my list that I should read next? https://t.co/5UC9FayEfg

— Happy Brain Science (@ScottCrab) April 22, 2019

(Side note: while social media is correlated with depression and I've definitely had my share of triggers on twitter this year the handful of positive psychology or mindfulness tweets I've seen has more than made up for it.)

One of the important and early ones was Succeed by Heidi Halvorson. This book pointed out that the mindset we have when trying to attain goals is very important, and although the magical-thinking Think And Grow Rich / The Secret type books might say 'just be positive' the evidence-based Succeed points out that the mindset is to both believe we can do it but at the same time believe it will not be easy. People who think achieving their goals will be easy, whether it's weight loss or finishing a novel or what, are more likely to fail than those who don't, because they might get complacent; they might allow themselves to be tempted because they're overconfident in their ability to resist temptation; and other reasons.

So after reading Succeed I reframed some of my goals. Dungeon Life became a thing that I was going to work on just to learn, not to make money or to necessarily become popular. (When you approach a goal with the mindset of learning rather than some criteria of 'success' you're more resistant to critical feedback, for one thing, and game developers sure get a lot of critical feedback.) I've been working on Dungeon Life ever since, part-time, and been much happier with my work on it than before.

The other goal I reframed was getting a real job-job-type-job. To be honest, I thought restarting my career after doing the indie game dev thing would be easy. I mean, like, I was the tech lead on Spider-Man 2 and stuff! But nobody was hiring for gameplay programming or lead positions and when I branched out to possible other areas I would either be ghosted or choke on the interviews. I realized: getting a job was going to be hard. I read in one book that 1 out of 10 onsites result in a hire. So to succeed at my goal I created new habits: one was to apply to a new position or follow up on an existing query every day, and another was to study how to take job interviews: I read Cracking the Coding Interview, practiced the algorithmic stuff on LeetCode almost every day. I also hit up my friends for referrals. It took many months but eventually I landed what I expect to be an awesome position: a lead coder on the Minecraft team at Microsoft! I don't know if I would have landed it if it hadn't been for Succeed, or, for that matter, my friend Max Szlagor.

My most recent read, Better Than Before, by Gretchen Rubin, was a surprise. It turned out it's a book about habit forming (look, I'm reincorporating. It's a storyteller thing) and I recognized a ton of myself in it, for example how I stopped strength training after my last Spartan race. That's the 'finish line' effect and why I've also stopped doing leetcode once I got offered the Minecraft gig. (Not ashamed of quitting leetcode though, as much as I enjoy doing it, if I'm going to have habits I want them to be the most important habits I can have and I'm not sure leetcode makes the cut.) It turns out goal setting can be harmful to habit forming unless you set a new goal as soon as you reach the old one. Hey, is that why I never wrote a second novel?

One strategy from Better Than Before is the 'clean slate' strategy: when facing a life change, it's an opportunity to commit to other habits as well that will be more likely to stick. How convenient that I'm starting a new job soon. I've been afraid that I'll let habits like running and meditation slip once faced with possibly long days and commutes, but Better Than Before has me thinking that if I build my cardio habit into my new schedule it will stick. I'll let you know.

Another thing Better Than Before reminded me of is that I'm a believer in a low carb diet, but I've let my dieting habits slip over the past few years. I want to recommit to a low carb diet again. So I've been keeping it down to around 100g a day for the past few days...and am pretty much hungry all the time now no matter how much protein I eat. :P But armed with the habit tracker I think I'll be able to stick with it. Again, I'll let you know.

Well...I've ran out of time for today, so I'm going to stop here. I've read a bunch of other books and I'll talk about how they've affected me and what I think of them in Part 2.

Easy To Implement Wall-Running

Did you know I open-sourced Energy Hook? As you'd expect, I'm proud of parts of the codebase and mortally embarrassed by other parts, but I'm getting over my embarrassment for anyone who wants to check it out. You can download the project here, the link is at the bottom of the page: http://www.happionlabs.com/energy-hook/ So if you want to dig through for the code that goes with this little article, you can. 

Anyways. Not too long ago I did a job interview where they asked me, "Can you talk about a time when you simplified a system?"

On the spot, I couldn't come up with anything good--my best answer was about how the monster leveling in Dungeon Life went from a complex and enigmatic algorithm that tried to gauge the strength of the heroes and multiplied it by how long the current heroes had been playing to a simpler system that just gradually increased the power of the monsters as the game went on--but in hindsight I could have mentioned Energy Hook's wall-running.

When I first implemented the wall-running, I did something rather complex - once I detected that a player wanted to wall-run (are they colliding with a wall and holding down the wall-run button?) I would constrain the avatar to that wall. Much like when swinging I would update the player position and then correct (as described here: https://gamedevelopment.tutsplus.com/tutorials/swinging-physics-for-player-movement-as-seen-in-spider-man-2-and-energy-hook--gamedev-8782) but instead of correcting by tethering to a sphere I corrected by constraining to the wall plane. Once the player collided with another wall or wasn't colliding with anything (they'd run past the edge) I'd turn the constraint off. The code was not pretty.

And then it occurred to me: I only have one entity in my game and Unity has this global gravity vector that we can change. So I stripped out all that code and simply changed gravity when the player is wall-running; the wall becomes the floor (or a sloping floor, really, because I didn't want the avatar to be perfectly vertical) and all the code that works for running on slopes and surfaces just worked and had fewer artifacts. I still had to be able to tell when the wall run was over - if you ended up catching air, or if your velocity suddenly changed (from bonking into something) I'd return to gravity to normal.

How would I do it with Roblox, you might be wondering. Roblox's gravity constant always points straight down, for one thing, and for another Roblox is multiplayer by default. How would I give each character their own gravity? So it wouldn't be as simple but I would probably do it by setting the global gravity to nothing and give each character or entity I wanted to process physics on a gravity BodyForce, so when a given entity needed to wallrun I could adjust its own personal gravity.

I'm sure other devs have independently done the same thing, but that didn't make the insight any less pleasing.

 

AB Testing vs Baseline Testing

Continuing my posts about n00b game design data science: https://www.gamedevblog.com/2019/07/a-n00b-does-game-design-data-science.html https://www.gamedevblog.com/2019/08/finally-ab-test-victories.html

Talking to the tech director of a fast growing up-and-coming always-online game I asked if they were doing AB testing with their live population. They weren't, and he said that they didn't really need to because they could tell if an update sucked by looking at the metrics before-and-after. I was surprised because they were a pretty big, well funded game and I thought all the cool kids were doing the real thing these days.

I learned long ago when getting my psych degree that this kind of testing, or 'baseline testing' - making a change to your test and then measuring to see if results changed - is not great. Without having proper control and experimental groups, factors beyond your control might be influencing your results. With a videogame update it might be having longer session lengths because it's a holiday; it might be more popular because a youtuber has told people it's cool; and so on.

To mitigate that you can do 'double baseline' testing, which is to take the change away and see if behavior returns to what it was. But who wants to implement a feature and then take it away once they see a spike in player engagement, just to make sure? 

So enter AB testing. It is fairly easy to assign your online players to different test groups and have legit 'control' and 'experimental' groups, and now you've introduced some scientific rigor. 

But there's another reason I like proper AB testing better than baseline testing and this is probably the clincher for me: it's being able to test lots of changes independently but still release them all at the same time. If you do, say, an update every week with a bunch of changes, and you see your metrics change, you don't know which changes in that update were responsible. Maybe one change made things way better and another change made things a little worse. You won't know. If you want to be scientific you have to slow down, make one change at a time, wait and see how people react, take them away again, yikes. 

With AB testing, however, you can independently test all the features in your update, and even investigate to see if features interact in interesting ways. This didn't really occur to me until I started doing it and then I wished I had gotten started much sooner. Soooo worth the small amount of time I invested. 

One thing I'm concerned about and don't have a good answer for yet - sometimes I'll make a change and when I do the AB testing the change, the 'B' group, is definitely a success when compared to the A group, but looking at my overall numbers things didn't seem to improve. For example I changed how leveling works on some Dungeon Life servers - before the change I was seeing average session lengths over 600.  After the change in the control group I started seeing session lengths of 400 and the new leveling system still had session lengths around 600. WTF? So clearly the new leveling system is better...but what is going on? Did I accidentally introduce some bug in the control group? Playing in the control group myself to check I didn't see any obvious problems. So that's going to remain one of life's little mysteries...because obviously I'm keeping the new system after all that.

That said, all in all I'm really enjoying the sort of objective rigor that this bringing to my design process. When I try something will make the game more appreciated or engaging and get to have some objective measure afterwards of whether I'm right or wrong, and with the AB testing I can continue developing at close to the same speed as before (minus a little overhead for measurement and more overhead for throwing out the changes that didn't work :/). I feel I'm learning about game design at a much faster rate than I have the rest of my life!

Finally, AB Test Victories

Continuing my n00b data science. With most of the AB testing I've been doing on Dungeon Life so far the only thing I've learned is that my ideas for making it more engaging or happy-making don't seem to have any effect and sometimes even make the game worse. I experimented with making the UI buttons 'juicier' - more animated and responsive - and to my surprise that lowered retention. (Alpha of 10%.) I experimented with moving people to the front of the queue to be heroes when they were first-time players. I experimented with hiding extraneous UI buttons for new players. A whole lot of nothing.

But finally I've made some useful changes!

One was adding VO that I did my own damn self for the tutorial. Just a few lines of me speaking in bad-roleplayer voice. At first it seemed like there was no effect, but then I filtered for people playing the game in English and the effect was super-clear. Tutorial VO improved all of my metrics with practically twice the retention! Not bad for someone who was once told "Welp. You're not a voice actor" when doing temp audio for Spider-Man. :)

I'd realized for a while that it takes too long to get to the final boss in a given 7-floor dungeon. My original conception of the game was that it would be endless until it got too hard and the hero would die. Later I decided that in case some players got OP I needed some way to cap the experience so somebody else could have a turn. Enter the superboss, a way to 'win' - but mostly a way to give other players a turn. I eventually concluded that 7 floors was still too long though. But how short to make it?

This required a little new tech. Up until now, I'm only testing differences between players. Two players with different experiences, for example one with VO and one without, could play on the same server and it was fine. Now I needed different server test groups. Doing that was different enough, but now the queries looked a bit different. Frex:

-- this automatically filters out players who played before the test group came into play, because if the category didn't exist for their first session it won't be in the join
-- thus it is only providing the first impressions for people playing that server and nothing else
SELECT value, rating FROM (
(SELECT server_key, category, action, value, time FROM events where category='ServerTestGroup' and action = 'NumDungeonLevels') as test_group
JOIN
(SELECT DISTINCT ON(player_key) player_key, value AS rating, time, server_key FROM events WHERE category='PlayerRating' ORDER BY player_key, time ) AS session_length
ON session_length.server_key = test_group.server_key
)
ORDER BY value

At first, as I was watching this test as it went along I noticed that it seemed to just have a trend for better session lengths the smaller the number of levels got. As the creator of Sixty Second Shooter this didn't really surprise me - I've been a firm believer that the shorter you can make your sessions the more engaging your game will be...all things being equal. So I made some more changes and made it so people could play anywhere from a 2-floor dungeon (one regular floor, one boss floor) to the old standby. Then, after some difficulties where the server was down while I was doing onsite interviews every day and too busy to check, I finally got a pile of data. There were thousands of test cases here:

This is all totally significant; t-tests and chi squares below 5%. 

So *that's* weird - though there is a trend towards better retention and ratings as the number of levels get shorter, session length is kind of all over the place. And there's this odd-even effect. If there are an even number of levels retention is up but ratings are down and vice-versa. I can only hazard a guess as to what that is - something to do with how long it takes monsters to build up their resources to do cooler things as far as they're concerned but less cool as far as the heroes are concerned? 

So now I'm in a quandary. Do I make the game a quick two-level experience because that's the favorite, even though the session length is lower? Or do I go for the best session length even though that version is less well liked? I've been talking a good game about how player experience is the most important thing to me but now that the choice between engagement and joy is staring me in the face I want the engagement.

So I decide I'm going to put my money where my mouth is and make the two floor game.

And after playing it for a while I start to feel despair.

I HATE it.

It's broken, for one thing. During the game the monsters are gradually building up their Dungeon Points which, by the third level (and sometimes the second if they're good) they can become mid-level bosses. With only two levels it's very hard to do that. A whole part of the game has been practically eliminated. 

The balance feels terrible also. The first floor is a cakewalk, like first dungeon floors usually are. Then the second floor has a boss that a new player was never intended to be able to beat. It goes from too easy to sudden death instantly.

So WTF? Why do my players seem to prefer this mode.

Here is how I've rationalized it: the ratings I'm looking at are actually first impressions. They've been playing for in the neighborhood of 5-20 minutes. They're enjoying the one hit kills, whether it's from being a hero on the first floor of a dungeon or getting lucky and becoming the superboss on the second floor. They have yet to realize that the game lacks depth this way. If I'm going to maximize the joy this game provides, I need to do different queries; a player's peak rating and their average rating are more important than their first. But their peak rating and average rating can't be tied to a single server, so I am just going to ignore their ratings this time.

And I decided to go with the 4 level dungeon, which is my favorite length and a good compromise for session length and retention. Yes, it gives a worse first impression, but since players are more likely to come back in this scenario, maybe their opinion will improve.

Since I've made the changes permanent more people have been playing DL, but I'm afraid that has nothing to with me. It's currently in the international featured sort, and getting a little more play than it usually does. When that's over, we'll see who sticks around.

A N00b Does Game Design Data Science

Because I'd rather Do Several Things Poorly™ than specialize, for fun and learnings I've been playing around with machine learning and data science with my last game, Dungeon Life. I want to share what I learn both because I like to write and hopefully educate but also because maybe you'll help catch me when I make mistakes. 

One of the cool things about Roblox is that it's a place you can do a data science side project without breaking the bank because the cost of user acquisition is so low. But I was not satisfied with either the Google Analytics or Game Analytics plug-ins because they weren't letting me see my own data and weren't letting me do the queries I wanted to do. Sure, I could look at my engagement, change things, and see if my changes seemed to have an effect, but could never tell for sure if the effects were due to my changes or other factors such as it being a holiday or what. 

I hope to share how I set up my metrics system in detail in another post - in brief, it uses Postgres SQL (it used to use MongoDB but I hit the memory cap and switched to something free; also I thought brushing up on my SQL would be a more marketable skill than learning Mongo's weird query language) with NodeJS on the backend and Roblox http service on the game servers. With just a handful of concurrent users at any moment I can run the backend on my desktop at home; I had no need to get another computer or set something up in the cloud.

My data packets currently contain these fields:

server_key: a guid generated by Roblox's HttpService:GenerateGUID.

session_key: ditto.

category: the class of event, such as 'Spawn' 'Kill' 'Tutorial' 'PlayerAdded' 'FrameDuration' - there are a bunch. 

action, label: additional info about the event.

value: numeric info about the event. Can be real. 

time: timestamp.

Once I had it set up, and once I'd made my way through a chunk of this online ML class (https://www.coursera.org/learn/machine-learning/) I started seeing what I could learn. Like most game devs I'm interested in session length (how long did they play) and retention (did they come back?) but lately I've also been interested in intrinsic motivation and whether a game brings joy or not. I believe it's quite possible for an activity to make you desire to keep doing it without it actually bringing you happiness or joy; they're separate neural subsystems, dopamine and endorphins, and humans are notoriously bad about desiring to do things that don't make them happy. (For more on that you can read Stumbling On Happiness by Daniel Gilbert.)

In my attempt to include joy as a metric to optimize, Dungeon Life pops up a dialog box at various intervals asking "How are you enjoying Dungeon Life lately?" and allows an input of 1-5 stars. Self-reporting isn't perfect when it comes to measuring happiness but what else can you do?

The way I approached it was by trying to create predictive models using linear and logistic regression. (That class is free and can give a far better explanation than I can so jump on that if you want to dive deeper.) 

Some features I thought might be relevant to a player's first Dungeon Life experience included:

framerate, language, screen resolution, percentage of successful hits with melee & ranged weapons, amount of currency earned, death rate, and kill rate. I needed to be careful to make sure I didn't just count deaths or kills, because obviously more deaths and kills would be correlated with a longer session. There are a lot more features that might be relevant but this seemed like a good start.

I've loved a good scatterplot ever since reading Tufte's Visual Display of Quantitative Information; Excel is a little twitchy when I try to make scatterplots with it, so I've been using Matlab just like I learned in my ML class.

Here's a graph of here's time played vs. frame duration:

So, not too surprisingly, players running the game on toasters do not hang around. What surprised me is how slow the game ran for so many people; it's not a very high-resource game. I think this may go some way to explaining something I wondered about Roblox: it seems the most popular games are not that graphically intensive, but sometimes you find real works of art languishing in low popularity. My conclusion is the market on Roblox has a lot of players with very low-end machines and approaching Roblox development with the attitude that your game will have more detailed graphics than the competition might be exactly wrong.

Using just these features I was surprised at how accurate a session-length prediction algorithm ended up being. A graph of predicted session length against actual session length for a test set looked like this:

The line represents perfect predictions. I personally was impressed that the cloud seemed to hover near the line. (Doing logistic regression on retention, however, has yet to be terribly accurate.)

From looking at that data I could see which features contributed the most to the session length prediction. Death rate was one of them.

FWIW, this is what my death query looked like:

-- this query gets the count of death events in the first session

  SELECT pkey_deaths, COUNT(death_time) as first_session_deaths FROM (

        -- this subquery gets all the death events before the end of the first session

        SELECT * FROM

            -- this subquery gets all the deaths we ever tracked

            (SELECT player_key as pkey_deaths, time as death_time FROM events WHERE category='Death' ) AS deaths 

        JOIN

            -- this subquery gets the end of the first session (I hope)

            (SELECT DISTINCT ON(player_key) player_key as pkey2, time as session_end FROM events WHERE category='SessionLength' ORDER BY player_key, time ) AS session_end_times

        ON pkey_deaths = pkey2 AND death_time <= session_end

  ) AS first_deaths

  GROUP BY pkey_deaths

(I divided by time spent playing and took the logarithm on the Matlab side.)

Death rate made some interesting graphs. Death rate was not a bell curve; some players quit before ever dying at all, some died once early on and quit right away. Applying a logarithm to it made it look nicer and ended up being what I put into the predictive model.

This reminded me of Csikszentmihalyi; die too much you quit, don't die enough you also quit. The green pluses means players who came back.

So, yay, I've got some insights. What do I do with them?

This is when I implemented A/B testing. I wrote some new code to assign players to groups (in typescript in case you lua people are wondering wtf):

export namespace GameplayTestService {

   let testGroupData =

   [

       {

           name: "InstantHeroExpress",

           values: 2,

       },

       {

           name: "ClassesLocked",

           values: 3  // 0: nothing locked; 1: mage locked; 2: mage & rogue locked

       },

       {

           name: "DelayPlaceButtonReveal",

           values: 2

       }

   ]

 

   let testGroups = new Map<Player,TestGroups>()

 

   export function playerAdded( player: Player, testInfoHolder: TestInfoHolder  ) {

       if( !testInfoHolder.testGroups ) {

           // choose test groups for new player

           testInfoHolder.testGroups = new Map<string,number>()

       }

       let existingGroups = testInfoHolder.testGroups

       for( let testGroup of testGroupData ) {

           if( !existingGroups.has( testGroup.name ) ) {

               let value = MathXL.RandomInteger( 0, testGroup.values-1 )

               existingGroups.set( testGroup.name, value )

               Analytics.ReportEvent( player, "AssignTestGroup", testGroup.name, "", value )

           }

       }

   }

}

The 'testInfoHolder' is stored in the datastore so we'll never forgot what test group a player belongs to, and is where the game keeps track of what code track to send a given player down.

So now I can query various metrics against test groups, frex:

SELECT test_group_value, COUNT(first_return_day), COUNT(day0pkey) FROM (

SELECT * FROM ( -- 1) get everyone who joined after the latest test was introduced

(SELECT player_key AS day0pkey, time AS day0start FROM events WHERE category='Day' AND value=0) as day0

JOIN

(SELECT player_key AS test_group_pkey, time AS test_group_time, action AS test_group, value AS test_group_value FROM events WHERE category='AssignTestGroup' AND action='ClassesLocked') as test_group

ON day0pkey=test_group_pkey ) AS tbl

WHERE test_group_time <= day0start ) AS relevant_samples

LEFT JOIN

(SELECT DISTINCT ON(player_key) player_key AS r_pkey, value AS first_return_day, time FROM events WHERE category='Day' AND value>0 ORDER BY player_key, time ) AS retained

ON r_pkey=day0pkey 

GROUP BY test_group_value

"ClassesLocked" is a change I made where instead of being able to pick from Warrior, Rogue, and Mage right off the bat, the choices are limited. This was inspired a bit by learning about the death rate. I had noticed playing the game that players liked to pick Mage, but Mage is actually harder to play; they're fragile and when they run out of mana they are inferior fighters. Even though the game recommends starting with Warrior it seemed like players didn't read that text. In addition, this would give the players motivation to keep playing to unlock more classes. A lot of games do it that way, maybe that's the reason. So what was the result? 

Classes Locked Out	Retained	Total	Expected
None	71	2413	81.07047
Mage	77	2425	81.47364
Mage and Rogue	98	2484	83.45589
	246	7322

 

With my naked eye that looks significant but people tend to see significance when it's not really there. I'd forgotten how to do a chi-square from my psych major days but happily Excel has it built in. I calculated the expected results by taking the retained total, multiplying by the line item total, and dividing by the total sum, then ran CHISQ.TEST on 71,77,98 vs 81,81,83. The result: 0.13. That means that there's a 13% chance we could have seen those results just from luck. (Correct me if I'm wrong, please, n00b here.)

So that wouldn't be good enough to publish a paper in a psych journal, but it is enough to convince me to leave that change in!

What about my other metrics? For those, I query all the results, pull them into Excel and do t-tests. Excel also happily has a button for that, but I don't know how to do multivariate statistical tests it turns out. So I just test locking out 'mage & rogue' vs 'none' - anybody who wants to tell me where to look to do the multivariate equivalent of a t test please do.

The mean of the player ratings is a skoch higher (3.90 stars instead of 3.86) but as I expected, not significant with a sample size of 979 and a t-stat of -.44.

And that's when I get the bad news.

t-Test: Two-Sample Assuming Unequal Variances
	Variable 1	Variable 2
Mean	509.463	427.967
Variance	508881	316605
Observations	2370	2454
Hypothesized Mean Difference	0
df	4502
t Stat	4.3957
P(T<=t) one-tail	5.6E-06
t Critical one-tail	1.6451
P(T<=t) two-tail	1.1E-05
t Critical two-tail	1.9604

The session length of the players with the unlocked classes vs. the locked classes is the opposite of what I want to see. I haven't had this significant a result since I began playing around with t-tests. I was shocked enough that I triple-checked that I had the test set up correctly. 

Wah wah wah.

What am I to conclude here? Players see the mage is locked out and decide to quit right then? 

And then it occurs to me: it's not immediately obvious to the player that the rogue & mage will unlock after they've played; they have the same padlock icon on them that the premium classes, the priest & barbarian, have. Maybe players are thinking "I have to pay for the mage? That's bullshit!"  So. I'll change that, and test some more!

I won't go into as much detail with other things I A/B tested; a few seemed to have no effect, but another thing that hurt was taking out the first Daily Prize that is thrown in the players' faces when they boot up the game for the very first time. Not only were they more likely to come back if they got one (alpha 5%) they might even have rated it higher (alpha 30%.) I thought I didn't need the GUI clutter, but apparently I do. 

So! That's been my experience with data science and game-making so far. I've yet to have a real win--mostly the value so far has been in catching mistakes--but one of these days I hope to legit improve the game.

I appreciate you reading all the way to the end; hope it was useful to you, and if I screwed something up let me know!

 

Sorted sets in C# and performance. A mystery remains.

I've recently become a bit obsessed for no good reason with different implementations for sorted sets in C#. I was looking into different ways to squeeze more performance out of a chunk of code and discovered to my confusion that, in my particular circumstances, I was able to provide a faster one than any out of the box. I'm not that much of an algorithmic badass, so there must be something going on I don't understand. So I decided to do some quick-and-dirty testing of various possibilities.

There's a few things we can do if we find ourselves needing a high-performance sorted set:

We can use the SortedSet<> class. Probably a good call, after all, what are the odds that we're going to be able to write an implementation that beats the pros?

That said, since SortedSet<> uses a binary tree for its implementation (presumably, it's actually not that well documented afaict) it's possible for smaller n's that some kind of implementation built on a List<> would be faster.  Unlike SortedDictionary<> which has its SortedList<> equivalent, there is no List equivalent for a Set. But, hey, we can use SortedList, and for a key use whatever we wanted to sort the elements of the set on in the first place.

We can do the same thing with a SortedDictionary as well - that's probably not going to be faster than SortedSet, but there are no guarantees in perf so I threw it into the mix just to see.

And I implemented my own quick-and-dirty List implementation of a Set, by BinarySearching into a List and inserting members myself. This has an instant advantage over the other sets in that with the other built-in C# sets I either have to check to see if the element is already in there before adding it, or put the Add in a try/catch block. With BinarySearch I could find the insertion point and tell if there's one already there at the same time.

There's an additional concern: should I use structs or classes for the elements in this set? Particularly with List-based implementations, cache-friendly structs might be just what the doctor ordered. Taking a look at Jackson Dunstan's example - https://jacksondunstan.com/articles/3860 - I was thinking that surely value based structs would be awesome! So l tried everything with both classes and structs and a variety of sample sizes to see what would happen.

In particular I was interested in the changeovers - at what sample size does it make sense to switch from a O(n) where the operation is dirt-cheap to O(log n) where it's not so much? Where does the cache friendliness of the array of structs start to mean less than having to move a large volume of those structs for each Insert? 

So here are my results for the Insert. These are with test structs/classes that contain 4 ints. 

This may be hard to read at first but the upshot is that with n of around 5000 the SortedSet is the winner, but with an n of 2500 my hand-rolled binary insert wins; the break-even point is somewhere between 2500 and 5000. Which was kind of surprising to me: log2 of 4096 is 12; are you really telling me the 12 operations the SortedSet is doing on average comes out to about the same as the 4000 'operations' the BinaryInsert is doing? 

Here's something I didn't get, though, which is what happened when I looked at iterating over these structures:

The list iteration is great, and the cache friendly struct version of the list is even better. I thought I'd see the SortedList performance in the ballpark of my List, though. After all, isn't it basically just a List? And why was the struct version of the SortedList so much slower than the class version? (It's hard to tell from the picture, but it's better with small n's, in the < 500 range, but then it falls off hard.)

I re-read the documentation (https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.sortedlist-2?view=netframework-4.8.) Hmm, it's a list of KeyValuePairs--is that making it less cache-friendly?--but according to the docs those are structs too. So, ok, it's doing an extra dereference. (I knew that, it's right there in the iterator code.) That'll take time. But that much time?

At this point I feel like I've spent too much time digging into this. I think about stepping through the code in the Visual Studio debugger to see what it's doing. Then I remember that dotnet core is open source. I can do something I've never done before; look at the source code of the language I'm using. Holy crap, there it is! https://github.com/dotnet/corefx/blob/master/src/System.Collections/src/System/Collections/Generic/SortedList.cs

Ohhhhhhhhhhhhh. It is two lists, and when I iterate over it using foreach(), it has to create a KeyValuePair for me. In other words, I am doing it wrong; I should be iterating over just the values. Let's compare:

Huh. So when I iterate over just the values it is much better than iterating over the pairs as we'd expect, but it's still puzzling that it's that much slower than the vanilla list. At this point it should be identical. Back to puzzling over the code…  I dig up the MoveNext for List<> and after looking back and forth between the MoveNext for the sorted list values and the MoveNext for List I couldn't tell you why one is slower than the other.

In the meantime, it seems like using an enumerator, which is just indexing into the list anyway, is doing extra work which good old for( int i … ) wouldn't do. So I gave that a whirl.

Yikes. I really wasn't expecting such a huge difference. I'm usually a 'premature optimization is the root of all evil' type but I might never use foreach again. 

So my main take-home lessons from this experience are: 

• if performance is an issue, maybe reinventing the wheel can actually be a good thing
• even if n is in the thousands O(n) might be better than O(log n)
• foreach.  yeesh.

This dive into C# performance turned out to be a rabbit hole - I'm not sure when we're going to hit bottom. For example, maybe I should now test it in Mono and Visual Studio to see if the same performance characteristics carry over. If I'd known how long this would take I might not have bothered, but it was a learning experience! But I'm going to pull myself away now and maybe see if I can learn something from y'all instead of digging in further myself.

What I really don't get is that now I'm indexing into the two different kinds of list myself, why is the one still so much slower than the other? 

I throw it to you. Can you shed light on this for me?

(One week later: mystery solved? I reread the [] operator for both the List and the SortedList.Values - the SortedList.Values makes another function call; seems like that's probably it.)

I would be completely unsurprised (but still a little embarrassed) if there's just an obvious bug in the code. I've put it up to play with in github: https://github.com/JamieFristrom/SortedContainerTests

My Roblox Years: Dungeon Life and New Product Science

After fourteen years I'm throwing in the towel as an indie game developer.

Now that I am quitting as an indie I can make blogging more of a priority. I've been feeling that itch to write, to communicate with other devs rather than just players, and in more than just the sporadic tweet. 

For the past three-plus years I've been doing something very unusual for an indie dev my age: developing games on Roblox. I was lucky enough, a few years ago, to meet an investor who had seen how much money some kids were making on Roblox and interested in seeing what would happen if he funded professional developers to make games for it. We came at the whole thing very arrogantly! We were sure we would be able to beat the best game they had on offer--after all, we were professionals!--and while we did beat most of them as far as production values go, we failed to make anything popular. Our games would have had to do at least a hundred times better than they did to be profitable.

In hindsight we discovered that there were actually plenty of games that have high production values on the platform but for whatever reason very few of those games became very popular. Perhaps Roblox is a platform where that je ne sais quois is truly what matters?

But just because we weren't able to figure out the market doesn't mean other devs should ignore it. While there are many reasons to not make games for Roblox that in turn means that there isn't that much competition on the platform. Unlike most entrenched platforms these days it's fairly easy to get noticed. So there are a lot of young, inexperienced developers who are making enough money with Roblox to fund small teams. Making a game on any platform is a highly risky proposition; it may be less risky on Roblox, with both smaller budgets and larger chances of success.

For me, possibly the most interesting thing about Roblox is that the costs of advertising on the platform are supernaturally low. On Roblox you can buy a click for your game for about two cents. I'd love to be corrected if I'm wrong but as far as I know that's unheard of anywhere else. This has allowed me to use Roblox as a petri dish for studying player behavior.

Roblox is therefore a pretty great place to apply some new product science, in particular the idea of going from a handful of prototypes / pilots / proofs of concept and winnowing them down until you have one product that has the best shot. After making Castleheart and Legend of You, products that we spent several months on before unveiling them to the public, I wanted to fail earlier. So I made several prototypes (most of which are still available on my Roblox profile page if you can find someone to play them with you) and launched them all before test audiences.

These prototypes all took around one to four weeks to make; the average was about two. They had no monetization component: they were not viable products in that sense. I also tried to keep extrinsic motivation out of them: I wanted to find activities that were intrinsically fun, that players wanted to play and come back to simply for the joy of playing. The prototype that became Dungeon Life, for example, was permadeath. Later, once I decided that it was the most promising prototype, I added its leveling system.

You can never be sure if a pilot program does poorly because it's a bad idea or because of poor execution, but by trying several prototypes and looking at their retention I was able to make decisions about which product to pursue which didn't just rely on gut feelings.

I wish I could report to you that thanks to that careful strategy I was able to create a game that was financially successful. As things built up to launch the signs were very promising: Dungeon Life was earning more than I was spending on advertising; it was getting over 80% thumbs up for its ratings; and it had longer average session lengths and more retention than any game I'd worked on yet: about 15 minutes and 10%. Roblox agreed to feature the game--note: currently I'm pretty sure that any experienced developer who puts in an honest few months of work can get their game featured--and my game was on the front page with my hopes high.

And then the game tanked. The ratings fell to 65%, the retention dropped to 5%, the average session length fell under 10 minutes. I spent the next month feverishly trying to improve it but to no avail; my changes were marginal improvements at best. The game earned a few hardcore fans who had lots of ideas for improvement but those were generally ideas that did not cater to the new players who were giving it low ratings and not coming back.

One of the best things I did that month was implement a method to provide feedback directly within the game; people's feedback would be turned into an event that was posted to Google Analytics, so I could simply read the analytics events page to see what the most popular complaints were. Movement speed was one of the main ones at the time--characters in Dungeon Life moved slow by Roblox standards--and bringing it back up to normal was an easy fix that noticeably improved engagement but not enough to save the game.

So this may make it sound like my attempts to apply new product science were pointless, but here's the thing: although Dungeon Life was a financial failure I know that none of the other ideas I had were necessarily any better. If I hadn't done that process of testing prototypes I might be swayed to go back and try to give another one of my game ideas a go; instead, I know that they're just as likely to flop as Dungeon Life. It makes more sense for me to tweak Dungeon Life and see if I can get to a tipping point with it than to try and breathe life into one of the less promising prototypes.

Unfortunately I'm at a point where I feel it's foolish to continue to pay the opportunity cost of being an indie game developer and plan to be back doing full-time employed work soon. (Honestly, I don't exactly remember why I wanted to be an indie game dev in the first place. At one point I thought it would be my dream job but aside from the actual development part it hasn't brought me much joy.) 

In the meantime though I continue to noodle with Dungeon Life as a testbed for learning: it's given me the opportunity to learn typescript (thanks to https://roblox-ts.github.io/), backend coding (I've made my own analytics system where I get to actually look at the data I gather, unlike with, say, Google Analytics or Game Analytics. Crazy, I know!), and some machine learning and data science. (I use my analytics data for a test set.) I hope to write more about this stuff in the near future. 

I don't regret the time I spent as an indie, or the time I spent working on Roblox. The creative part has always been a joy and I've made some games that I think are really cool even if they weren't financially successful. You might give Dungeon Life a go: it's a dungeon crawler where you take turns playing the heroes and the monsters, a bit like Crawl, but it's a third-person action RPG with persistent levels and loot and online multiplayer. (https://www.roblox.com/games/2184151436/Dungeon-Life)

If you have questions about developing for Roblox fire away; hopefully I'll make a habit of blogging again and be able to use your questions for future articles.

Have I really not blogged in three years? So sad.