Analyzing Starcraft 2's Ranking System
Saturday, July 24, 2010 at 12:29PM 
Let's talk about Starcraft 2's ranking system, specifically the bonus pool system, the focus on ranked matches, and the division system. Before we get into all that, I'll give some background info from three years ago when I talked with Rob Pardo (VP of Game Design at Blizzard) about me possibly taking on the role of ranking-and-tournament systems designer at Blizzard. I ended up having to back out of that race because the Street Fighter HD Remix project suddenly became a reality, and I couldn't pass that up.
The reason I want to tell you about the ranking stuff from back then is to illustrate two points of view--mine and Pardo's--and to explain how it took me three years to understand that his point of view is probably best after all.
TrueSkill vs. Monkeying With Rankings
He asked me to come up with a ranking system for a game, we'll just call it Game X. My first response was that this is easy because it's already a solved problem: Microsoft solved it with what they call TrueSkill. TrueSkill is a refinement of the well-known ELO system used in Chess. One of TrueSkill's main features is that it can apply to games with more than 2 players, while ELO can't. Also, TrueSkill uses a bell curve rather than a single point when referring to a player's skill level. As the system gets more information about the player, it becomes more "certain" that the skill ranking is accurate, and that player's personal bell curve shrinks.
More important than any of those details though, is Microsoft's philosophy about rankings. The premise of their whole system is that players will have the most fun if the ranking system can give matches as close to 50-50 as possible. Yeah it's fun to have to have a few matches that are easy and some that are hard (and you will because of the inevitable variance), but you really do want the ranking system to try to give you close matches. Consider a matchmaking system that simply gave you random opponents, and how from the perspective of a bad player, he just gets stomped repeatedly then probably quits. He is better served by getting evenly matched with other bad players. Over time, he might become a good player rather than quitting.
Microsoft makes another good point here that ONLY winning and losing can be allowed to affect these stats. You can't adjust the matchmaking stat by "experience points" or even by any skill-based stats such as headshots, number of kills, time to finish a lap in racing, etc. All those stats can be gamed, and you will end up trying to get more headshots or something instead of winning. Any formula that equates number of headshots (or any other stat besides wins/losses) with how likely you are to win or lose introduces a layer of imperfect simulation. If we want to know how likely you are to beat someone, we should only consider your wins and losses, and not any in-game stats.
Short version: Microsoft's philosophy is correct (or is it...?), our ranking system should only consider wins and losses and should maximize close matches using TrueSkill, or a close implementation.
But Pardo was not quite on board. Such a ranking system is harsh and while hardcores are ok with it, regular people just get crushed and discouraged. They don't get to see any real progress, he said.
Yes that's true, I responded. And yet competitive games flourish and top the sales charts. We shouldn't ruin our ranking system with some experience point system to help people's egos because that will necessarily make the matchmaking system worse. No matter what anyone claims to want, they will be happier with close matches as those are memorable and intense, and that trumps anything that deviates from the TrueSkill philosophy, I said.
Pardo said that he really wanted what he wanted though, and if he's asking for a system that is more friendly to non-hardcores, then I should be able to come up with one. "If someone put a gun to your head and said to give some sort of incentive to keep people going in a ranking system, what would you do?" Yes, fair enough. If he's asking a question, I should be able to provide an answer. But I still thought the question was more like, "should we ruin the perfect matchmaking a little, a medium amount, or a lot?" while I was thinking, "let's not ruin it at all!"
Later, I presented a system that used a TrueSkill-like number for matchmaking behind the scenes, and that did not deviate by allowing any nonsense stats to affect it. But the UI would prominently display 1 main experience level stat (not the skill-based stat) and 2 other stats that were destined to go up over time as well. (Think "headshots", though it wasn't exactly that. Stats that naturally increase over time.) These secondary stats were dressed up a bit in the form of levels, too, so you basically had three meters going up. The fourth stat, the REAL one, was also displayed (and used for matchmaking!) but it was displayed less prominently. Players had various progress bars to level up, but still perfect matchmaking.
Pardo thought I at least gave it a try, but that I had not tried hard enough. He said "we don't need hyper-accurate matchmaking out of a ranking system. That's not its primary function." This is around the time when Street Fighter HD Remix started, so I did not get the chance to submit a revised system.
What did he mean by that last statement though? Isn't "hyper-accurate matchmaking" exactly the thing we do want from a matchmaking system? Years later I’m willing to say, “maybe not.” It could be a good trade-off to have merely “very good” matchmaking if in exchange for losing “hyper-accuracy” we also gain some fun or keep people playing when they otherwise would have given up. Microsoft’s premise was that hyper-accuracy gives you the most fun possible, but perhaps that premise is not strictly true. Maybe there is a way to add fun without losing much at all in the way of perfect matchmaking. Starcraft 2’s bonus point pool system could be that way.
Starcraft 2’s Bonus Point Pools
The bonus point system is similar to the “rest” system from World of Wacraft. Players accumulate a pool of bonus points over time, capped at some maximum. In World of Warcraft, the function is to give you double the experience points each time you kill a monster, until your bonus points run out. In Starcraft 2 the function is to give you double the *ranking points* each time you get a win, until your bonus points run out. For example, if you were facing a stronger opponent such that winning gives you 10 points and losing would subtract 5 points, then winning would actually give you 20 points. 10 points would drain out of your bonus pool and go into your ranking points (to match the 10 points you’d normally get for winning here). If you lose, you would lose 5 points as usual with no further adjustment.
What is the effect of this system? Some people claim it rewards those who play a lot, but this is looking at it the wrong way. Those people would claim that play-a-lot guy gets more ranking points overall because he uses up all his bonus points all the time, while a play-a-little guy would not play enough to use them up, so he’d get fewer points overall for his wins. True, but missing the point.
The actual play-a-lot guy will play way more than his bonus point pool can hold. Most of his matches will be played without any bonus because the maximum size of the bonus pool must be fairly low for this system to even make sense. Meanwhile play-a-little guy will play most or all of his matches with boosts from these bonus points. So if anything, play-a-little guy benefits. (Though the play-hardly-at-all guy is disadvantaged because he doesn’t use up all those bonus points.) But even THAT isn’t the point.
I think there are two real points to this system. First, it gives you an incentive to play when you otherwise wouldn’t. If you haven’t played in a few days or a week or whatever, you will think “hmm, I could play right now and get the benefit of those bonus points!” And if you wait until the points reach their maximum (in World of Warcraft that’s two weeks, but unknown as of this writing in Starcraft 2), then you’ll have another reason to play: that you would be “wasting” bonus points by not playing. If you just played enough to use up some of the bonus points, you’d be able to keep accumulating more for later. Yes it’s a psychological trick, but at least it gets more people active, which basically helps everyone involved (more opponents).
The second real point hits at something Pardo was originally asking me for: a way to have at least some feeling of progress in the harsh world where actual progress is damn hard. Imagine that Alice, Bob, and Charlie all have 1100 ranking points and that they are all of equal skill, so any matches between them will have a 50-50 chance of going either way. Alice doesn’t play for a while (but her skills don’t decline). Bob and Charlie play each other, and also play even more players of the same ranking/skill. After a while, Alice will still be at 1100 (she didn’t even play). Bob and Charlie will have a HIGHER number of ranking points though, due to the bonus pool. Even if they have even records of 30 wins, 30 losses against each other, they are gaining more points than they are losing because bonus points are added for wins, but there’s no extra penalty for losing. They have the feeling of progress here, even if it’s illusory.
Does this screw up the actual rankings? Well, Bob and Charlie might find themselves at 1300 ranking points (I’m just making these numbers up), so you could say their ranking is inflated. But...not really, because all the other people of their same skill out there have inflated rankings by the same amount. Really, it just stretched the scale out. It’s true that Alice will have bad matchmaking when she decides to play again though. Her rating is 1100, but because of the point inflation that’s gone on, she really “deserves” a higher rating now. That will correct itself pretty quickly though, it just means she’ll have a few matches that are easier than expected when she returns. This is the imperfection in matchmaking that we have to accept if we want the good features of: 1) more people tend to play more, and 2) ego boosts even when you go even against evenly matched opponents.
Also consider that this ego boost is not changing your ranking against a list of specific competitors. If you’re the 1000th best player, you’ll still be that, it’s just that everyone’s points are creeping higher over time. (There’s a reset every season, so it won’t go infinite.) Since your REAL rank as 1000th best player is intact without too much monkeying, it seems ok that the underlying stat increase, as a psychological motivator. We still know who is actually better, so who really cares if everyone’s underlying stat is one that goes up over time or not? Really only your ego cares, so we might as well throw you a bone.
To put it another way, I was thinking about how I have not played Street Fighter HD Remix online in the last two weeks. If I had some bonus pool points waiting for me there, would I have? Probably yes, as dumb as that is. And yet if that made me and thousands of other people play a bit more, it helps the whole scene, so why not.
Focus on Ranked Matches
Microsoft has a clear philosophy on this one too: sometimes people want to be ranked because they are seeking out competition and a way to measure themselves against others, but sometimes people want to avoid all that and play without any fear of shame or losing. Xbox games must have both a ranked mode and an unranked mode (badly named “player matches”). In ranked mode, developers can show any kind of stats to the player they want, while in unranked matches, they are supposed to hide these stats as much as possible. This makes total sense to me, as even I don’t want some record of how bad I am at some game I’m just messing around with. And sometimes I don’t want a win/loss record shoved in my face either.
There’s a tension though, and it goes back to matchmaking. We really want you to play ranked matches a lot so that we can accurately determine your skill level and give you close matches. In Kongai, a card game I designed, we did what some other games do by just renaming the modes. The modes used to be called Ranked and Unranked, but now the ranked match is just called Play Game. This shifts the perception of which one is the one you’re supposed to play. While Ranked sounds ominous and serious, Play Game sounds like it’s for everyone.
Starcraft 2 takes a similar approach here. The default mode is ranked, and it’s not named some scary thing, it’s just presented as the normal way to play the game. The screen where you get into a game devotes only a small space in the corner to creating / joining a Custom Game, which is the equivalent of unranked.
The notion that you’re always playing ranked could turn people off, but it does help matchmaking, like I said. Overall I’m weakly in favor of it.
Divisions
Next, Starcraft 2 does a kind of weird thing though. Within each of the 5 zones, there are a zillion players, so they break up these players into groups of 100. You are giving a ranking relative to those 100 people. So you might be “#25 in the silver league” for example, which is a polite way of saying “#52 billion overall.” I get the point here, it’s another psychological trick to keep you going. Being ranked out of 100 doesn’t sound so bad, and it’s a small enough set that your ranking won’t change wildly for no reason. If you saw your real overall rank (#52 billion), it would change all the time from everyone else moving around you, even if you did nothing.
I think, not sure on this, that Blizzard is also trying to give you a set of people small enough to be in your monkeysphere (more than 150 people is too much for your brain to understand). What’s a bit confusing about this is that based on the beta, you don’t seem to play against these 100 people any more than anyone else. I’m not sure if that was the intent, or an artifact of the beta, or what. Maybe those people happen to not be online when you are looking for a match. For whatever reason, it seemed rare to actually encounter them in the beta, which means you’re really being ranked against 100 random people who you have no idea about. It does sound a bit better if the matchmaking mostly matched you with them, so you got to know them and felt more of a personal connection with your subcommunity, which consists of people at least fairly closely ranked with you. We’ll have to see how this works out in the real game.
One thing I will say about the divisions though, is that the top division SHOULD be designed differently. While average joe is probably very happy with this division system, I think the very top players will not be. Anyone way, way up there wants to know the real, cold, hard truth. They want to know who is #1 and who is #10, etc. They are dying to know this, even. So for the very top bracket only, I suggest that all players are in just one division. I’m not sure if this would be best for the top 1,000 players or 10,000 or 5%, or what the magic number is, but I’m saying people at that end actually want the opposite of what the division system is offering. They really do want to know exactly where they stand relative to each other.
Conclusion
Of all these things, it’s really the bonus point pool system I wanted to mention the most. As soon as I realized that system would have made me play Street Fighter HD Remix in the last couple weeks instead of not playing, I realized that its effect of enticing people to play every so often is more important than any loss of “hyper-accurate matchmaking.” So I think Pardo was right in what he was asking for all those years ago.Sirlin
Followup: Some people have said that the matchmaking stat is not affected by the bonus pool points, and that the bonus pool points are only used for display purposes, like where you appear in your 100 man division. This is not the impression I got from Blizzard's statements, but that claim could be correct. It's also possible the answer to that changed at some point, as who knows what adjustments went on during the beta. If the claim is true, the system matches you by a different criteria than it actually rates you. I don't really know what to think of that.
Sirlin
More information here that the bonus pool points do not affect the "real" underlying stat used for matchmaking, only the ("fake"?) displayed ranking in your division.
Reader Comments (68)
i gotthe game and its awesome but i just cant host games in bnet... i mean i can host games but noone joins them, anyone else has that problem ?
I would have to say that supporting Activision Blizzard at all is Playing To Lose for the games industry overall.
Enlightening.
Except for the last paragraph. I want to live knowing it's that good, instead of the standard TrueSkill.
I'm sorry it doesn't sound to me like you know half the things you are talking about.
MS had a VERY accurate, very competitive ranking system in Halo 2 (it also had a a type of bonus tool points). But half the reason the system was so accurate so that is took a really long time to rank up. A couple weeks to a month to get to your correct rank. After all the system was based on a exp ladder system.
Because of the time it took to accumulate exp MS designed, TrueSkill. TrueSkill ranks you based on the ranks of the players you are fighting instead of giving each player exp after a game. Because of this it give you a decent rank VERY fast, in a few games, similar to the practice games in SC. But the big problem with this system (and SC2) is that if you are new to the game & place poorly initially then the grind to get a correct rank becomes exponentially more difficult. Plus because you aren't seeing much progress, the game's ranking system becomes even more discouraging to the player.
Alex S,
Nice that you chose to open with an insult. I know all the points you raised, so it seems you're just trying to be antagonistic. Very strange that you open with "you know half the things you talk about" then "correct" me by repeating exactly what I know and have said.
I agree with the flaws of TrueSkill that you said. If winning your 100th game against a particular ranked opponent gives X points, then winning your 10,000th game against that same ranked opponent should STILL give X points. After years of experience with ranking systems that take away your ability to move in the ranks because they are "sure" of where you belong, I think they are a bad idea.
I'd just like to say that the whole SC2 ranking system has actually stopped me from playing the game. I was only a gold level player, but I am competitive, and when I play I want to see improvement - in real skill, not some fairyland number conjured up by a *cough* 'ranking system'.
But it quickly became apparent that my rank and points in SC2 was no indication of my true skill, and hence I had no motivation or enjoyment in continuing to compete. I'm fine if they have a default 'fake' ranking which a player can change to view their true ranking, but the depths blizzard has gone to to hide a player's true ranking even when the player is trying to find it out is pretty sickening, and leaves a bad taste in my mouth for Blizzard. The fact that they are trying to trick their players in continuing to play just feels extremely condescending, like they believe we are just babies who will crawl after candy that's held in front of us.
Then again, I guess this game design is highly successfully - just look at WoW.
The problem here, Sirlin, in addition to having a totally hidden match criteria (which they eventually decided to display in WoW for example), is that there is actually no discovered cap on bonus pool thus far. People's teams that have been inactive since August (http://us.battle.net/sc2/en/profile/322715/1/Clydemist/ladder/10067#current-rank) have around 1900 bonus pool accumulated and it's still growing. As long as you eventually play at any time before they do one of their planned ladder resets, everyone basically gets the same free ride up to whatever level the bonus pool has reached (since everyone has gained an identical number of bonus pool points). People have basically taken to ignoring the bonus pool since it does nothing but inflate everyone the exact same amount, and you can see how much bonus pool they haven't spent yet in their profile and add it on to their rating.
You say: "Most of his matches will be played without any bonus because the maximum size of the bonus pool must be fairly low for this system to even make sense." But they've gone and put no maximum size on the bonus pool whatsoever.
READ UP, AND LEARN, I GOT IT MOSTLY CALCULATED AFTER SEVERAL TRIAL AND ERRORS
******************************************************************************************************************
============================================================================
i understand the fun factor, i do the same thing, 1% of the time ill play my true skill level because after thousands of games u get bored with the repetitive winning procedure and u dont improve ur skills cept in that category i like to explore all options and i always choose random so im always fully remembering what im against and dont get rusty, i was winning 2v2s earlier today by rushing command centers in under 3 minutes then turning them into planetary after plant, or killing people with ghosts only or ravens only, or infestors only etc... obviously that is gonna fail many times but its fun learning how to use the unit to its maximum potential and what to expect when u encounter one, and its funny when u do win with silly shit like that.
I do random games with random people mostly, box of chocolates style, well i been back for 4 days after a 36 week break, and i made up an amazing terran strat, not gonna tell about it, its still underground, not many seen it, but it had a 30-8 win/loss record, that includes the first time i tried it and the tweaking and fixing errors, once i go tit down pat it was nearly flawless. So my first 5 games back 5-0 with 2v2 random, got diamond, why not masters weird.. still dont get that anyways, maybe cause my teamate was never in diamond division in the past or possibly ever, so its not gonna take him from gold/silver to master he would get destroyed without me, btw i was 1st place diamond before masters existed s1 in all brackets except 1v1 i dont like 1v1 im a social person when i play, i played a few games in 1v1 and i was top 8 platinum but never cared to learn the trade, its all simple mathematics anyways jus if ur bored enough to properly execute each action and not promote a laugh halfway through the game when u hear "nuclear launch detected" i like to have fun when i play. So im a past top diamond player he was gold/silver, it cant bump him that high, itll handicap him in matchmaking when hes by himself, so we 5-0 diamond, did 3s, 5-0 diamond again, same issue playing with someone whos never worked his way up the ladder in pts and wins to earn his promotion like s1 did, i remember starting at bronze, winning 30 games at 1st place finally promotion, 30 games at silver 1st place finally promotion, 30 games is estimate but u get the idea i climbed the ladder it wasnt given, he never went thru that, however i believed if i had played with 2 masters on my team for 3s and we even went 4-1 we prolly woulda hit masters considering a friend of mine went 0-5 in 1v1 ladder and somehow has diamond because back when he climbed it and he at least put up a fight, NOW THIS IS WHEN i started realizing maybe it has nothing to do with any of that.
heres what i learned last night, i logged on im 3v3 diamond, played 3 games won all 3, demoted to platinum, 4th game i got demoted to gold and i won that one as well, 5th game i won, stayed gold, 6th game i lost, and then it put me in silver, now im confused as hell, because i used to be diamond top 8 when masters didnt exist and i climbed from gold 1st to plat 1st to diamond 1st, so that must not be it.. heres the thing: we would win our games, sometimes id have top score in all brackets sometimes not, but i always did goofy things, diff strat each time, never repetitive, and id always try new ways of building and id make new strategies, i had fun and put myself through a possible loss to learn new strategies or kill someone with ravens etc.. cause fuck.. im already silver now, whats the worst that could happen, well im 1st place in the 4s silver and keep winnign never losing, my pts were like 500 and 2nd place in my division was like 300, i was dominating why no move..
then one game i was in a 3s and i was pissed off because i truly believed if i kept winning i would get promoted, and my 2 teamates were REALLY bad, so i pulled some masters level skill out and i played serious and killed all 3 enemies myself without help 1 by 1, well when i did this and when that game ended, it boosted me from 4v4 silver, to 4v4 diamond and i was 3rd place in the bracket, now ud think its all about the score because i pulled a 3v1 but no, one of the guys on the other team had a higher score than me from beating my teamates up allgame, and making more units than i did, i just made smarter units, so i placed 2nd in score, but then it said promotion 4v4 diamond 3rd place.. so wtf just happened, its self explanatory if u think about it, aka playing for fun or building properly, making every unit count, using multiples and losing few while killing many with macro, lots of expansions, never letting them take them, just playing like my life depended on it, so think about that, and u should be able to figure out what the trick is, this by the way was my first serious non slack game in i dnno, the last 100 and was the only time i moved up. i had upgrades, expansions, minerals, no unspent minerals, very minimal if at all mistakes were made.
so these are the steps ive taken, things ive tried, and then finally the thing that worked, its self explanatory, the point system monitors a whole lot more than displayed i believe, its extremely complex as if its being judged by an AI, each move is important, or each move not made that should be is as well, and it places u by the way u play, not score, not none of that, cause when i had fun and won with random stuff, i never got anything or ever got a raise, 30 games id play win each one nothing.. so it places u with people who are willing and will play to the potential and move and build to the skill u can present, so it keeps the fun players at a low, and the serious ones will get the ratings rather they win or lose, if they built properly due to the default standardized skill level such as if u play like an insanity computer, u get masters, if u play as if u were a very hard computer ur diamond and so on, i think it times ur building stages, unit productions etc, all is calculated.
thats my 2 cents, and what worked for me, enjoy, and i still play for fun i cant get into it seriously unless theres money on it or something, so ill prolly stay in silvers, but thats ok, division means nothing really, i can still kill several master players with a cannon rush, just gotta rethink ur strategy til u get it to work, those rumors are false, just because he has a perfect 4 gate build, doesnt mean he will get to use it, or i wont triple nuke his base while hes walled in and im seiged in, etc.. fun can be had, but even tho im silver and beat that master, i didnt do it to insanitys measures on expected build orders etc.. and proper units to use against rivals, im making ghosts while hes got colossus.. it didnt look good, win or lose and it put under the "luck" category, i think theres ur hidden point system.