Analyzing Starcraft 2's Ranking System
Saturday, July 24, 2010 at 12:29PM 
Let's talk about Starcraft 2's ranking system, specifically the bonus pool system, the focus on ranked matches, and the division system. Before we get into all that, I'll give some background info from three years ago when I talked with Rob Pardo (VP of Game Design at Blizzard) about me possibly taking on the role of ranking-and-tournament systems designer at Blizzard. I ended up having to back out of that race because the Street Fighter HD Remix project suddenly became a reality, and I couldn't pass that up.
The reason I want to tell you about the ranking stuff from back then is to illustrate two points of view--mine and Pardo's--and to explain how it took me three years to understand that his point of view is probably best after all.
TrueSkill vs. Monkeying With Rankings
He asked me to come up with a ranking system for a game, we'll just call it Game X. My first response was that this is easy because it's already a solved problem: Microsoft solved it with what they call TrueSkill. TrueSkill is a refinement of the well-known ELO system used in Chess. One of TrueSkill's main features is that it can apply to games with more than 2 players, while ELO can't. Also, TrueSkill uses a bell curve rather than a single point when referring to a player's skill level. As the system gets more information about the player, it becomes more "certain" that the skill ranking is accurate, and that player's personal bell curve shrinks.
More important than any of those details though, is Microsoft's philosophy about rankings. The premise of their whole system is that players will have the most fun if the ranking system can give matches as close to 50-50 as possible. Yeah it's fun to have to have a few matches that are easy and some that are hard (and you will because of the inevitable variance), but you really do want the ranking system to try to give you close matches. Consider a matchmaking system that simply gave you random opponents, and how from the perspective of a bad player, he just gets stomped repeatedly then probably quits. He is better served by getting evenly matched with other bad players. Over time, he might become a good player rather than quitting.
Microsoft makes another good point here that ONLY winning and losing can be allowed to affect these stats. You can't adjust the matchmaking stat by "experience points" or even by any skill-based stats such as headshots, number of kills, time to finish a lap in racing, etc. All those stats can be gamed, and you will end up trying to get more headshots or something instead of winning. Any formula that equates number of headshots (or any other stat besides wins/losses) with how likely you are to win or lose introduces a layer of imperfect simulation. If we want to know how likely you are to beat someone, we should only consider your wins and losses, and not any in-game stats.
Short version: Microsoft's philosophy is correct (or is it...?), our ranking system should only consider wins and losses and should maximize close matches using TrueSkill, or a close implementation.
But Pardo was not quite on board. Such a ranking system is harsh and while hardcores are ok with it, regular people just get crushed and discouraged. They don't get to see any real progress, he said.
Yes that's true, I responded. And yet competitive games flourish and top the sales charts. We shouldn't ruin our ranking system with some experience point system to help people's egos because that will necessarily make the matchmaking system worse. No matter what anyone claims to want, they will be happier with close matches as those are memorable and intense, and that trumps anything that deviates from the TrueSkill philosophy, I said.
Pardo said that he really wanted what he wanted though, and if he's asking for a system that is more friendly to non-hardcores, then I should be able to come up with one. "If someone put a gun to your head and said to give some sort of incentive to keep people going in a ranking system, what would you do?" Yes, fair enough. If he's asking a question, I should be able to provide an answer. But I still thought the question was more like, "should we ruin the perfect matchmaking a little, a medium amount, or a lot?" while I was thinking, "let's not ruin it at all!"
Later, I presented a system that used a TrueSkill-like number for matchmaking behind the scenes, and that did not deviate by allowing any nonsense stats to affect it. But the UI would prominently display 1 main experience level stat (not the skill-based stat) and 2 other stats that were destined to go up over time as well. (Think "headshots", though it wasn't exactly that. Stats that naturally increase over time.) These secondary stats were dressed up a bit in the form of levels, too, so you basically had three meters going up. The fourth stat, the REAL one, was also displayed (and used for matchmaking!) but it was displayed less prominently. Players had various progress bars to level up, but still perfect matchmaking.
Pardo thought I at least gave it a try, but that I had not tried hard enough. He said "we don't need hyper-accurate matchmaking out of a ranking system. That's not its primary function." This is around the time when Street Fighter HD Remix started, so I did not get the chance to submit a revised system.
What did he mean by that last statement though? Isn't "hyper-accurate matchmaking" exactly the thing we do want from a matchmaking system? Years later I’m willing to say, “maybe not.” It could be a good trade-off to have merely “very good” matchmaking if in exchange for losing “hyper-accuracy” we also gain some fun or keep people playing when they otherwise would have given up. Microsoft’s premise was that hyper-accuracy gives you the most fun possible, but perhaps that premise is not strictly true. Maybe there is a way to add fun without losing much at all in the way of perfect matchmaking. Starcraft 2’s bonus point pool system could be that way.
Starcraft 2’s Bonus Point Pools
The bonus point system is similar to the “rest” system from World of Wacraft. Players accumulate a pool of bonus points over time, capped at some maximum. In World of Warcraft, the function is to give you double the experience points each time you kill a monster, until your bonus points run out. In Starcraft 2 the function is to give you double the *ranking points* each time you get a win, until your bonus points run out. For example, if you were facing a stronger opponent such that winning gives you 10 points and losing would subtract 5 points, then winning would actually give you 20 points. 10 points would drain out of your bonus pool and go into your ranking points (to match the 10 points you’d normally get for winning here). If you lose, you would lose 5 points as usual with no further adjustment.
What is the effect of this system? Some people claim it rewards those who play a lot, but this is looking at it the wrong way. Those people would claim that play-a-lot guy gets more ranking points overall because he uses up all his bonus points all the time, while a play-a-little guy would not play enough to use them up, so he’d get fewer points overall for his wins. True, but missing the point.
The actual play-a-lot guy will play way more than his bonus point pool can hold. Most of his matches will be played without any bonus because the maximum size of the bonus pool must be fairly low for this system to even make sense. Meanwhile play-a-little guy will play most or all of his matches with boosts from these bonus points. So if anything, play-a-little guy benefits. (Though the play-hardly-at-all guy is disadvantaged because he doesn’t use up all those bonus points.) But even THAT isn’t the point.
I think there are two real points to this system. First, it gives you an incentive to play when you otherwise wouldn’t. If you haven’t played in a few days or a week or whatever, you will think “hmm, I could play right now and get the benefit of those bonus points!” And if you wait until the points reach their maximum (in World of Warcraft that’s two weeks, but unknown as of this writing in Starcraft 2), then you’ll have another reason to play: that you would be “wasting” bonus points by not playing. If you just played enough to use up some of the bonus points, you’d be able to keep accumulating more for later. Yes it’s a psychological trick, but at least it gets more people active, which basically helps everyone involved (more opponents).
The second real point hits at something Pardo was originally asking me for: a way to have at least some feeling of progress in the harsh world where actual progress is damn hard. Imagine that Alice, Bob, and Charlie all have 1100 ranking points and that they are all of equal skill, so any matches between them will have a 50-50 chance of going either way. Alice doesn’t play for a while (but her skills don’t decline). Bob and Charlie play each other, and also play even more players of the same ranking/skill. After a while, Alice will still be at 1100 (she didn’t even play). Bob and Charlie will have a HIGHER number of ranking points though, due to the bonus pool. Even if they have even records of 30 wins, 30 losses against each other, they are gaining more points than they are losing because bonus points are added for wins, but there’s no extra penalty for losing. They have the feeling of progress here, even if it’s illusory.
Does this screw up the actual rankings? Well, Bob and Charlie might find themselves at 1300 ranking points (I’m just making these numbers up), so you could say their ranking is inflated. But...not really, because all the other people of their same skill out there have inflated rankings by the same amount. Really, it just stretched the scale out. It’s true that Alice will have bad matchmaking when she decides to play again though. Her rating is 1100, but because of the point inflation that’s gone on, she really “deserves” a higher rating now. That will correct itself pretty quickly though, it just means she’ll have a few matches that are easier than expected when she returns. This is the imperfection in matchmaking that we have to accept if we want the good features of: 1) more people tend to play more, and 2) ego boosts even when you go even against evenly matched opponents.
Also consider that this ego boost is not changing your ranking against a list of specific competitors. If you’re the 1000th best player, you’ll still be that, it’s just that everyone’s points are creeping higher over time. (There’s a reset every season, so it won’t go infinite.) Since your REAL rank as 1000th best player is intact without too much monkeying, it seems ok that the underlying stat increase, as a psychological motivator. We still know who is actually better, so who really cares if everyone’s underlying stat is one that goes up over time or not? Really only your ego cares, so we might as well throw you a bone.
To put it another way, I was thinking about how I have not played Street Fighter HD Remix online in the last two weeks. If I had some bonus pool points waiting for me there, would I have? Probably yes, as dumb as that is. And yet if that made me and thousands of other people play a bit more, it helps the whole scene, so why not.
Focus on Ranked Matches
Microsoft has a clear philosophy on this one too: sometimes people want to be ranked because they are seeking out competition and a way to measure themselves against others, but sometimes people want to avoid all that and play without any fear of shame or losing. Xbox games must have both a ranked mode and an unranked mode (badly named “player matches”). In ranked mode, developers can show any kind of stats to the player they want, while in unranked matches, they are supposed to hide these stats as much as possible. This makes total sense to me, as even I don’t want some record of how bad I am at some game I’m just messing around with. And sometimes I don’t want a win/loss record shoved in my face either.
There’s a tension though, and it goes back to matchmaking. We really want you to play ranked matches a lot so that we can accurately determine your skill level and give you close matches. In Kongai, a card game I designed, we did what some other games do by just renaming the modes. The modes used to be called Ranked and Unranked, but now the ranked match is just called Play Game. This shifts the perception of which one is the one you’re supposed to play. While Ranked sounds ominous and serious, Play Game sounds like it’s for everyone.
Starcraft 2 takes a similar approach here. The default mode is ranked, and it’s not named some scary thing, it’s just presented as the normal way to play the game. The screen where you get into a game devotes only a small space in the corner to creating / joining a Custom Game, which is the equivalent of unranked.
The notion that you’re always playing ranked could turn people off, but it does help matchmaking, like I said. Overall I’m weakly in favor of it.
Divisions
Next, Starcraft 2 does a kind of weird thing though. Within each of the 5 zones, there are a zillion players, so they break up these players into groups of 100. You are giving a ranking relative to those 100 people. So you might be “#25 in the silver league” for example, which is a polite way of saying “#52 billion overall.” I get the point here, it’s another psychological trick to keep you going. Being ranked out of 100 doesn’t sound so bad, and it’s a small enough set that your ranking won’t change wildly for no reason. If you saw your real overall rank (#52 billion), it would change all the time from everyone else moving around you, even if you did nothing.
I think, not sure on this, that Blizzard is also trying to give you a set of people small enough to be in your monkeysphere (more than 150 people is too much for your brain to understand). What’s a bit confusing about this is that based on the beta, you don’t seem to play against these 100 people any more than anyone else. I’m not sure if that was the intent, or an artifact of the beta, or what. Maybe those people happen to not be online when you are looking for a match. For whatever reason, it seemed rare to actually encounter them in the beta, which means you’re really being ranked against 100 random people who you have no idea about. It does sound a bit better if the matchmaking mostly matched you with them, so you got to know them and felt more of a personal connection with your subcommunity, which consists of people at least fairly closely ranked with you. We’ll have to see how this works out in the real game.
One thing I will say about the divisions though, is that the top division SHOULD be designed differently. While average joe is probably very happy with this division system, I think the very top players will not be. Anyone way, way up there wants to know the real, cold, hard truth. They want to know who is #1 and who is #10, etc. They are dying to know this, even. So for the very top bracket only, I suggest that all players are in just one division. I’m not sure if this would be best for the top 1,000 players or 10,000 or 5%, or what the magic number is, but I’m saying people at that end actually want the opposite of what the division system is offering. They really do want to know exactly where they stand relative to each other.
Conclusion
Of all these things, it’s really the bonus point pool system I wanted to mention the most. As soon as I realized that system would have made me play Street Fighter HD Remix in the last couple weeks instead of not playing, I realized that its effect of enticing people to play every so often is more important than any loss of “hyper-accurate matchmaking.” So I think Pardo was right in what he was asking for all those years ago.Sirlin
Followup: Some people have said that the matchmaking stat is not affected by the bonus pool points, and that the bonus pool points are only used for display purposes, like where you appear in your 100 man division. This is not the impression I got from Blizzard's statements, but that claim could be correct. It's also possible the answer to that changed at some point, as who knows what adjustments went on during the beta. If the claim is true, the system matches you by a different criteria than it actually rates you. I don't really know what to think of that.
Sirlin
More information here that the bonus pool points do not affect the "real" underlying stat used for matchmaking, only the ("fake"?) displayed ranking in your division.
Reader Comments (68)
One thing I would add is that, not only should the top league consist of only one divison, but that it should use a TrueSkill-type system. Even if we accept that the psychological benefits of a steady point increase outweigh the importance of precise matchmaking and rankings for most users, elite players are different for the reasons you mention. They don't need a thinly-veiled trick to induce them to play the game or to make them feel better about themselves, and are probably more interested in accurate information about their standing.
The whole time I read this I couldn't figure out why the thing you use to indicate progress needs to also be the thing you use for matchmaking. Why can't you earn ranks or something and increase your Hey-You-Played-Starcraft Points as your Total Wins goes up, but still use the win/loss record for matchmaking? Maybe I'm just thick, but I don't think you really explained why the Bonus Point Pool would be better than what you proposed? (Apart from the loss-aversion aspect which isn't really making it more fun so much as just making you play anyway, right?)
What exactly do you mean by... fun?
Assuming that what you're displaying is simply a fact, then does the way in which facts are displayed have something to do with the enjoyment factor of a game? Besides adding rainbows, sparkles and gratuitous sound effects.. uhh well Peggle is pretty damn fun to watch haha.
Seriously, do only "hardcore" players appreciate an accurate representation of their gameplay results? I think not. The danger of artificially inflating a user's display of progress is that you will risk the ire of any player who wants their game to be HONEST with them.
Consider the alternative: instead of artificially inflating any kind of factual display of data or altering an actual ranking according to set rules, consider ACTUALLY inflating the abilities of users as they decrease in rank and slowly deflating abilities as you rank up. I think that would keep things interesting and allow new players jumping right with an advantage which slowly decreases as they rank up. I'm basically stealing this idea from Mario Cart for the Wii which gives players in last place a I-Win bullet and players in first place only get bananas for bonuses. This not only removes the need for teirs, but also gives poorer users the sense of accomplishment and gives hardcore users nothing more than what they already had because their relative rankings will remain the same. This is, of course, assuming the player bonuses don't drastically cause terrible players to jump immediately to the top. :)
Great blog, regular reader, this is my first comment. Keep up the great work.
First off I'm going to come right out and say I strongly dislike Trueskill. I'm happy for strong matchmaking systems, but Trueskill seems evil to me. The 'certainty it gets about your skill is really annoying at times, especially when you're a veteran of the genre, but new to a game. In a situation where a player will start out with slow improvements then rapidly improve, you can get stuck feeling like you have to 'grind' your rating to get it up to where it should be. At least this has been my Trueskill experience.
As for Blizzards system, I love it. First I'll note that they've promised a 'proleague' which hopefully will be a global ranking for top players, but we'll have to wait and see.
The thing I wanted to point out about Blizzard's system is that the bonus pool may not influence matchmaking, it's not entirely clear. Players in Blizzard's system have a publically displayed rating based on their currently division and a hidden ranking based on your matchmaking skill. It's not entirely clear if these two ratings have any real relationship with eachother. What is clear is that the bonus pool points apply to the displayed rating. So it's possible they don't apply to the hidden rating and instead only influence the displayed rating. This means that you can have the incentive to keep playing while preserving the quality of the matchmaking system by not having it inflate.
Overall the bonus pool system is mediocre to me. I'm not entirely sure how well it influences players to keep playing, but I'm sure some people do enjoy using up their bonus pool points. My perspective is skewed because I play a lot of games so I never have bonus pool.
The system that does work for me though is the 100-man leagues and the divisions overall. The divisions are really nice for setting immediate obtainable goals and then feeling good about accomplishing them. Maybe you want to get into Silver league, or be #1 gold, or top #10 diamond. Whatever goal you set it's right there for you to grab. Likewise with the 100 man leagues it provides an incentive to keep playing when combined with the bonus pool. Since the public ratings (which determines your position in your league) inflate due to bonus pool it means you need to keep playing to maintain your spot. You might be all proud about your #1 platinum spot right now, but if you don't play for a few days some loser will usurp your throne.
Likewise when comparing yourself to your friends there's something about the league system that gives that rivalry a little more weight. It's fun to be able to compare your rating or league with your friends rather than just a global ranking.
Overall I think Blizzard did an excellent job with their matchmaking, especially since the behind the scenes ranking works wonderfully even before the bonus pool and league system. It's 100% the best ranking system I've used so far.
Another way to look at it is that the whole ladder ranking system is a race. Not only in terms of time, but in skill. In your example (with a competitive game), it's highly unlikely Alice will go back to the game after a few weeks at the same skill level. She'll need some "catch up time". And even then, she'll be behind in practice to those who didn't take a break.
Once players recognize ELO is a "race", as soon as they pause for a moment they're disincentived from coming back. They've fallen behind. It feels like they can't catch up. The bonus pool/rest systems gives Alice the illusion that it's worth her while to jump back in, because she's getting a bonus, a bonus none of the other racers got. Emotionally, it catalyzes you back over the hump.
But they also balance out mechanically pretty well too, because being at the top of the skill cap (usually) requires constant practice. Yes, if you could keep winning only when you have bonus points you'd get ahead. So actually, waiting for bonus points is actually costing you wins because you aren't practicing, and puts you behind when you jump back in. If balanced correctly, (in addition to spreading out the rankings), they often prevent you from getting too punished for trying to get back on the horse, rather then helping you "win more".
I think it's genius too.
If you get X number of bonus points a day, then to raise your rank, the optimal strategy is to only play games when you have bonus points. "I can play against Bob today and win 10 points, or I can play against Bob tomorrow and win 20 points." Beating Bob is the exact same challenge today or tomorrow, but tomorrow the reward is bigger.
I have played two games that had a mechanic in place that encouraged me to play X amount of time per day, no more no less. They weren't ranking mechanics, but anyway: Alteil 2 had a promotion where playing 3 games a day, no more no less, was the strictly optimal thing to do in terms of maximizing resources. And Kingdom of Loathing has this thing every day, where maximizing your score means you play X amount of minutes per day - skipping a day is suboptimal, and playing too much in one day is also suboptimal.
The result is that I have trouble staying in flow while playing, because I'm less focused on enjoying the game itself and more focused on completing my daily goal. Even though I used to play 3 games of Alteil 2 a day anyway, it became less fun when I "had" to do it to complete a goal. And I felt bad playing a 4th game in one day, because that 4th game did not have as much "reward" as it would've had if I had waited a day.
I agree that Bonus Points will increase the number of people playing the game, but I know that for myself, it will reduce the amount of fun I have playing the game. I have no idea how much of the population I represent though.
I find something innately disturbing about the assumption that having only matches with people close to you is more fun. Maybe I'm just a masochist though.
Very interesting.
I've had a similar experience with SF4's Championship Mode which also sacrifices some fairness for progress and provides goals at different scales. Would like to hear your thoughts on that system.
One thing that you haven't addressed though is what are the psychological effects for players on the losing side of unevenness, having been catapulted by their rest xp into a position of frequent losses?
Very insightful blog post, Sirlin. I had wondered why Blizzard decided against a single
TrueSkill-like rating before, and your post clears that up nicely.
Personally I really like the way Trackmania handles your (global) rank by displaying it
relative to your location:
#314159 in the World
#8192 in your country
#1573 in your state
#99 in your city
Getting completely slaughtered doesnt usually get you anything. Having to fight for your win or lose not crushingly so, but convincingly often highlights what you did wrong and what you need to improve.
Small incremental steps in difficulty with the occasional jump is a lot more fun than effortless wins or demolishing losses.
Note that "fun" in this context means an enjoyable, faster progress in skill.
It's good to read such a well thought out article on the matter.
I'd say another thing about the 100 man leagues is that nobody wants to be #100, you strive for #1 and get the feeling you'll drop off the end if you hit 100, at least that's the feeling I get even if it is just carrot on a stick.
The rankings are actually available with a bit of data mining. Full rankings for Diamond division across all realms can be found here:
http://sc2.vacau.com/sc2/index.php
However, it seems to have been a conscious decision by Blizzard to obfuscate them. The funny thing is that the owner of the site reports quite a lot of traffic from Blizzard's own IP addresses, presumably employees who also want to find out their global rankings.
@Divison part
They should just add "GLOBAL RANK" for all player's and that would be nice. Not just Diamond players are dieing for knowing where they are actualy there are many other players who want's to know where they are in (world) server stats.
ps: nice article.
i always wondered why blizzard didnt design the system like it is at iccup. on the other hand iccup probably is designed for the pros, more or less. as a casual player u probably wont even get a D or D+ ranking. well may be they gonna do something similiar for the top 2000 or something, like u suggested. good read btw ;)
One especially nice aspect of a "bonus point" system is that the ratings inflate while you don't play, so when you DO play you're against slightly easier opponents, letting you "warm up" and recover the skills lost from lack of practice. By the time you catch up to your "true" skill you'll be recovered in ability enough to be even win/loss at that level, but if you were dumped back in to where you were you'd probably take a ratings hit (and be discouraged) as you lose 4, 5 matches in a row before getting back in the groove of things.
tl;dr your skill isn't static but degrades over time, but nobody likes their numbers dropping, so instead the numbers of active players inflate.
Another trick that might be worth considering is keeping the real ranking, and then matchmaking based on real rank - grinded rank. That way people can feel like they are getting better even when they aren't, because they're playing against easier opponents.
I was #1 in my diamond division and it meant nothing.
The division system sounded like a good idea when I read about it before Starcraft II's beta went online, but after having played hundreds of games, I can say I've never felt a sense of community with the people in my division. The "trick" is not successful. It's completely transparent. I don't feel like I'm progressing significantly. I don't feel like I'm competing directly with familiar names. And to have my actual rank hidden from me makes me care less, not more, about progression on the ladder.
This doesn't apply only to high level players, either. I don't consider myself to be a high level player, and many people of all skill levels have brought this criticism against the division system. Regardless of skill, someone participating in a competition is likely to be interested in where he places. Given that StarCraft II is designed in part to cater to competitive, esports-minded players, it's disappointing that the only stat of consequence is completely hidden.
If the goal is to keep casual players interested, the best method (aside from good matchmaking) is the achievement system. Right now, simply accumulating wins (not a high win/loss ratio, just wins) grants access to unit portraits that are displayed next to the username. As someone who is generally critical of achievements, I actually enjoyed working toward each new portrait. They were fun to get and fun to show off, and they now go up to 1000 wins for each of the four race selections in both 1V1 and team play. That's 8000 wins worth of simple achievement content that already does an excellent job of motivating casual gamers to continue playing. Some people were even begging to hold onto their beta achievements after release.
The way I see it, someone who doesn't care where he ranks doesn't need the system built around him, and someone who does care isn't going to be fooled by a false ranking. Right now, the only outward indicator of a player's skill level in StarCraft II is his league placement, which is too vague to state anything but the obvious.
Nice blog, and I have to say I agree to a lot of it, as a midrange player (before the beta closed I was in the bottom of platinum league) I find it realy fun to "climb" the ladder. Something that these small divisions lets me. There are ofc occasions where I want to see the big picture but that could be handled outside of the game.
I have one thought on the bonus system though, couldn't the bonus points come from your own points? A player that doesn't play should get worse, right? So if I dont play for 1 day, i lose 5 points, but instead of completly loosing them, I gain them back in bonus points. That would solve the inflation I belive.
A big choice with the ranking system was the introduction of placement matches. Placement matches are matches that you have to perform to be placed in a certain division. And more interestingly, the choice to directly place players in the high division is quite unique to the SC2 matchmaking system. It's like if you were asked to perform a few fights in WoW and then you were given a level based on that. It's really good, and at the same time it would be nice to know (as a player) how the division you're put in is calculated, because right now it looks like a black box. And the division you're placed in has a huge impact on your psychology as well. It's cool to start high in division but at the same time it's scarry. And it's cool to be placed in a low division so you have progression. Those must have been real difficult design decisions to take.