Analyzing Starcraft 2's Ranking System
Saturday, July 24, 2010 at 12:29PM 
Let's talk about Starcraft 2's ranking system, specifically the bonus pool system, the focus on ranked matches, and the division system. Before we get into all that, I'll give some background info from three years ago when I talked with Rob Pardo (VP of Game Design at Blizzard) about me possibly taking on the role of ranking-and-tournament systems designer at Blizzard. I ended up having to back out of that race because the Street Fighter HD Remix project suddenly became a reality, and I couldn't pass that up.
The reason I want to tell you about the ranking stuff from back then is to illustrate two points of view--mine and Pardo's--and to explain how it took me three years to understand that his point of view is probably best after all.
TrueSkill vs. Monkeying With Rankings
He asked me to come up with a ranking system for a game, we'll just call it Game X. My first response was that this is easy because it's already a solved problem: Microsoft solved it with what they call TrueSkill. TrueSkill is a refinement of the well-known ELO system used in Chess. One of TrueSkill's main features is that it can apply to games with more than 2 players, while ELO can't. Also, TrueSkill uses a bell curve rather than a single point when referring to a player's skill level. As the system gets more information about the player, it becomes more "certain" that the skill ranking is accurate, and that player's personal bell curve shrinks.
More important than any of those details though, is Microsoft's philosophy about rankings. The premise of their whole system is that players will have the most fun if the ranking system can give matches as close to 50-50 as possible. Yeah it's fun to have to have a few matches that are easy and some that are hard (and you will because of the inevitable variance), but you really do want the ranking system to try to give you close matches. Consider a matchmaking system that simply gave you random opponents, and how from the perspective of a bad player, he just gets stomped repeatedly then probably quits. He is better served by getting evenly matched with other bad players. Over time, he might become a good player rather than quitting.
Microsoft makes another good point here that ONLY winning and losing can be allowed to affect these stats. You can't adjust the matchmaking stat by "experience points" or even by any skill-based stats such as headshots, number of kills, time to finish a lap in racing, etc. All those stats can be gamed, and you will end up trying to get more headshots or something instead of winning. Any formula that equates number of headshots (or any other stat besides wins/losses) with how likely you are to win or lose introduces a layer of imperfect simulation. If we want to know how likely you are to beat someone, we should only consider your wins and losses, and not any in-game stats.
Short version: Microsoft's philosophy is correct (or is it...?), our ranking system should only consider wins and losses and should maximize close matches using TrueSkill, or a close implementation.
But Pardo was not quite on board. Such a ranking system is harsh and while hardcores are ok with it, regular people just get crushed and discouraged. They don't get to see any real progress, he said.
Yes that's true, I responded. And yet competitive games flourish and top the sales charts. We shouldn't ruin our ranking system with some experience point system to help people's egos because that will necessarily make the matchmaking system worse. No matter what anyone claims to want, they will be happier with close matches as those are memorable and intense, and that trumps anything that deviates from the TrueSkill philosophy, I said.
Pardo said that he really wanted what he wanted though, and if he's asking for a system that is more friendly to non-hardcores, then I should be able to come up with one. "If someone put a gun to your head and said to give some sort of incentive to keep people going in a ranking system, what would you do?" Yes, fair enough. If he's asking a question, I should be able to provide an answer. But I still thought the question was more like, "should we ruin the perfect matchmaking a little, a medium amount, or a lot?" while I was thinking, "let's not ruin it at all!"
Later, I presented a system that used a TrueSkill-like number for matchmaking behind the scenes, and that did not deviate by allowing any nonsense stats to affect it. But the UI would prominently display 1 main experience level stat (not the skill-based stat) and 2 other stats that were destined to go up over time as well. (Think "headshots", though it wasn't exactly that. Stats that naturally increase over time.) These secondary stats were dressed up a bit in the form of levels, too, so you basically had three meters going up. The fourth stat, the REAL one, was also displayed (and used for matchmaking!) but it was displayed less prominently. Players had various progress bars to level up, but still perfect matchmaking.
Pardo thought I at least gave it a try, but that I had not tried hard enough. He said "we don't need hyper-accurate matchmaking out of a ranking system. That's not its primary function." This is around the time when Street Fighter HD Remix started, so I did not get the chance to submit a revised system.
What did he mean by that last statement though? Isn't "hyper-accurate matchmaking" exactly the thing we do want from a matchmaking system? Years later I’m willing to say, “maybe not.” It could be a good trade-off to have merely “very good” matchmaking if in exchange for losing “hyper-accuracy” we also gain some fun or keep people playing when they otherwise would have given up. Microsoft’s premise was that hyper-accuracy gives you the most fun possible, but perhaps that premise is not strictly true. Maybe there is a way to add fun without losing much at all in the way of perfect matchmaking. Starcraft 2’s bonus point pool system could be that way.
Starcraft 2’s Bonus Point Pools
The bonus point system is similar to the “rest” system from World of Wacraft. Players accumulate a pool of bonus points over time, capped at some maximum. In World of Warcraft, the function is to give you double the experience points each time you kill a monster, until your bonus points run out. In Starcraft 2 the function is to give you double the *ranking points* each time you get a win, until your bonus points run out. For example, if you were facing a stronger opponent such that winning gives you 10 points and losing would subtract 5 points, then winning would actually give you 20 points. 10 points would drain out of your bonus pool and go into your ranking points (to match the 10 points you’d normally get for winning here). If you lose, you would lose 5 points as usual with no further adjustment.
What is the effect of this system? Some people claim it rewards those who play a lot, but this is looking at it the wrong way. Those people would claim that play-a-lot guy gets more ranking points overall because he uses up all his bonus points all the time, while a play-a-little guy would not play enough to use them up, so he’d get fewer points overall for his wins. True, but missing the point.
The actual play-a-lot guy will play way more than his bonus point pool can hold. Most of his matches will be played without any bonus because the maximum size of the bonus pool must be fairly low for this system to even make sense. Meanwhile play-a-little guy will play most or all of his matches with boosts from these bonus points. So if anything, play-a-little guy benefits. (Though the play-hardly-at-all guy is disadvantaged because he doesn’t use up all those bonus points.) But even THAT isn’t the point.
I think there are two real points to this system. First, it gives you an incentive to play when you otherwise wouldn’t. If you haven’t played in a few days or a week or whatever, you will think “hmm, I could play right now and get the benefit of those bonus points!” And if you wait until the points reach their maximum (in World of Warcraft that’s two weeks, but unknown as of this writing in Starcraft 2), then you’ll have another reason to play: that you would be “wasting” bonus points by not playing. If you just played enough to use up some of the bonus points, you’d be able to keep accumulating more for later. Yes it’s a psychological trick, but at least it gets more people active, which basically helps everyone involved (more opponents).
The second real point hits at something Pardo was originally asking me for: a way to have at least some feeling of progress in the harsh world where actual progress is damn hard. Imagine that Alice, Bob, and Charlie all have 1100 ranking points and that they are all of equal skill, so any matches between them will have a 50-50 chance of going either way. Alice doesn’t play for a while (but her skills don’t decline). Bob and Charlie play each other, and also play even more players of the same ranking/skill. After a while, Alice will still be at 1100 (she didn’t even play). Bob and Charlie will have a HIGHER number of ranking points though, due to the bonus pool. Even if they have even records of 30 wins, 30 losses against each other, they are gaining more points than they are losing because bonus points are added for wins, but there’s no extra penalty for losing. They have the feeling of progress here, even if it’s illusory.
Does this screw up the actual rankings? Well, Bob and Charlie might find themselves at 1300 ranking points (I’m just making these numbers up), so you could say their ranking is inflated. But...not really, because all the other people of their same skill out there have inflated rankings by the same amount. Really, it just stretched the scale out. It’s true that Alice will have bad matchmaking when she decides to play again though. Her rating is 1100, but because of the point inflation that’s gone on, she really “deserves” a higher rating now. That will correct itself pretty quickly though, it just means she’ll have a few matches that are easier than expected when she returns. This is the imperfection in matchmaking that we have to accept if we want the good features of: 1) more people tend to play more, and 2) ego boosts even when you go even against evenly matched opponents.
Also consider that this ego boost is not changing your ranking against a list of specific competitors. If you’re the 1000th best player, you’ll still be that, it’s just that everyone’s points are creeping higher over time. (There’s a reset every season, so it won’t go infinite.) Since your REAL rank as 1000th best player is intact without too much monkeying, it seems ok that the underlying stat increase, as a psychological motivator. We still know who is actually better, so who really cares if everyone’s underlying stat is one that goes up over time or not? Really only your ego cares, so we might as well throw you a bone.
To put it another way, I was thinking about how I have not played Street Fighter HD Remix online in the last two weeks. If I had some bonus pool points waiting for me there, would I have? Probably yes, as dumb as that is. And yet if that made me and thousands of other people play a bit more, it helps the whole scene, so why not.
Focus on Ranked Matches
Microsoft has a clear philosophy on this one too: sometimes people want to be ranked because they are seeking out competition and a way to measure themselves against others, but sometimes people want to avoid all that and play without any fear of shame or losing. Xbox games must have both a ranked mode and an unranked mode (badly named “player matches”). In ranked mode, developers can show any kind of stats to the player they want, while in unranked matches, they are supposed to hide these stats as much as possible. This makes total sense to me, as even I don’t want some record of how bad I am at some game I’m just messing around with. And sometimes I don’t want a win/loss record shoved in my face either.
There’s a tension though, and it goes back to matchmaking. We really want you to play ranked matches a lot so that we can accurately determine your skill level and give you close matches. In Kongai, a card game I designed, we did what some other games do by just renaming the modes. The modes used to be called Ranked and Unranked, but now the ranked match is just called Play Game. This shifts the perception of which one is the one you’re supposed to play. While Ranked sounds ominous and serious, Play Game sounds like it’s for everyone.
Starcraft 2 takes a similar approach here. The default mode is ranked, and it’s not named some scary thing, it’s just presented as the normal way to play the game. The screen where you get into a game devotes only a small space in the corner to creating / joining a Custom Game, which is the equivalent of unranked.
The notion that you’re always playing ranked could turn people off, but it does help matchmaking, like I said. Overall I’m weakly in favor of it.
Divisions
Next, Starcraft 2 does a kind of weird thing though. Within each of the 5 zones, there are a zillion players, so they break up these players into groups of 100. You are giving a ranking relative to those 100 people. So you might be “#25 in the silver league” for example, which is a polite way of saying “#52 billion overall.” I get the point here, it’s another psychological trick to keep you going. Being ranked out of 100 doesn’t sound so bad, and it’s a small enough set that your ranking won’t change wildly for no reason. If you saw your real overall rank (#52 billion), it would change all the time from everyone else moving around you, even if you did nothing.
I think, not sure on this, that Blizzard is also trying to give you a set of people small enough to be in your monkeysphere (more than 150 people is too much for your brain to understand). What’s a bit confusing about this is that based on the beta, you don’t seem to play against these 100 people any more than anyone else. I’m not sure if that was the intent, or an artifact of the beta, or what. Maybe those people happen to not be online when you are looking for a match. For whatever reason, it seemed rare to actually encounter them in the beta, which means you’re really being ranked against 100 random people who you have no idea about. It does sound a bit better if the matchmaking mostly matched you with them, so you got to know them and felt more of a personal connection with your subcommunity, which consists of people at least fairly closely ranked with you. We’ll have to see how this works out in the real game.
One thing I will say about the divisions though, is that the top division SHOULD be designed differently. While average joe is probably very happy with this division system, I think the very top players will not be. Anyone way, way up there wants to know the real, cold, hard truth. They want to know who is #1 and who is #10, etc. They are dying to know this, even. So for the very top bracket only, I suggest that all players are in just one division. I’m not sure if this would be best for the top 1,000 players or 10,000 or 5%, or what the magic number is, but I’m saying people at that end actually want the opposite of what the division system is offering. They really do want to know exactly where they stand relative to each other.
Conclusion
Of all these things, it’s really the bonus point pool system I wanted to mention the most. As soon as I realized that system would have made me play Street Fighter HD Remix in the last couple weeks instead of not playing, I realized that its effect of enticing people to play every so often is more important than any loss of “hyper-accurate matchmaking.” So I think Pardo was right in what he was asking for all those years ago.Sirlin
Followup: Some people have said that the matchmaking stat is not affected by the bonus pool points, and that the bonus pool points are only used for display purposes, like where you appear in your 100 man division. This is not the impression I got from Blizzard's statements, but that claim could be correct. It's also possible the answer to that changed at some point, as who knows what adjustments went on during the beta. If the claim is true, the system matches you by a different criteria than it actually rates you. I don't really know what to think of that.
Sirlin
More information here that the bonus pool points do not affect the "real" underlying stat used for matchmaking, only the ("fake"?) displayed ranking in your division.
Reader Comments (68)
Wow, thank you so much for this article. I am going to be thinking so much about rankings and systems now.
How can you defend a matchmaking system that is not only condescending in nature, but impractical, vague, and convoluted? There is nothing simpler in this world than a structured hierarchy. People climb the ladder in business, military, and politics. There are ranks associated with their position within the structure, and they know exactly where they stand relative to eachother. This is the point of a ladder. Imagine being in the army and finally getting a much deserved promotion. You are now a Captain. But not really. You are a Captain relative to 99 people you don't know, or have had any any interaction with whatsoever. Not only that, but compared to other Captains you're actually greatly over or under qualified for your newly acquired position. This means your rank now has lost all meaning. Do you see how this is actually counterproductive to the goal of getting people to put in more hours? Their achievements aren't based on merit! You might as well randomize the system entirely.
Blizzard has somehow managed to, at the very least, make people consider the possibility that a square is actually a more suitable shape for a wheel than a circle. Just because it's Blizzard doesn't mean they don't screw up. This, this is a screw up. Don't rationalize it, don't try to understand their motives behind it -- they screwed up.
When it comes to cheap psychological trickery (something StarCraft II absolutely does not need), the achievement system works considerable better. It still needs to go, though.
Awesome read up... I actually LOVE the idea of rank points inflation. Because when youre not playing, youll get worse. So when you play again youll probably end up playing against kind of even opponents until youll get back on your feet and be on your regular level which will be pretty soon because youre getting bonus points for every win.
As far as divisions are concerened i have no idea why theyre not making the visual ranking system a little more readable... In my opinion having divisions as a simple way of lowering the amount of information would be the best way to deal with it. so gold league division 1 > division 2 > division 3 > division 4
You still might end up competing with other people in your division at least on higher levels (its not like atm youll be able to find someone that matches your skill inside your division out of 1000 bronze league divisions^^)
And youll be able to see your progress pretty clearly by progressing through those divisions.
So if you ask me their idea of a ranking system is pretty perfect, their idea of visualizing that... less than optimal.
@MikeCosta:
"If you get X number of bonus points a day, then to raise your rank, the optimal strategy is to only play games when you have bonus points."
Well, no, that's not true. You assume that in addition to wanting to rise through the ranks
- the player has a goal of playing as little as possible
- the player has complete control over when he can play
This just isn't true for most people. An hour of play tonight can't necessarily be combined with an hour tomorrow night to give you two hours.
For any normal person the optimal strategy is to play as much as you would anyway, and then a few more times if your bonus points are nearing their maximum with that level of activity. You're not 'losing out' on points by playing when you don't have rest bonuses available.
The bonus point system sounds like a good way to get people to jump in when they might not otherwise... as long as it's not a system that encourages people to keep playing when they don't want to, which this obviously isn't.
I agree with Arzey in that if your goal is to encourage lower-tier players, having achievements and other non-gameplay-related unlocks might be a better approach. Even bad players aren't as dumb as you might think - I'm pretty sure most of them can figure out that their illusion of progress is meaningless. On the other hand, giving them stuff to work towards that doesn't even require them to win might help them experience more facets of the game and actually help them improve.
This article is based on a faulty understanding of SC2's matchmaking.
Matchmaking has nothing at all to do with league or rating, but rather uses a hidden rating that is most probably some sort of ELO like system. That's precisely why you will sometimes see things like silver players being matched up with platinum players.
So the current system is rather similar to the one that Sirlin suggested in the end, just with the "true" rating being completely invisible.
A carrot (in the way of bonus points pool) can work amazinly well for getting people to play more than they'd normally would. If you look at wow they have several of those carrots, while most other mmos don't. (ie. daily quests and instances that can only be done every x days)
Interesting to read about divisions as well, doesn't seem to make that much sense though if you don't play against your 100 more than other people, not much point in being ranked amongst 100 random people, though I guess it is better than being #52 billion.
I'm pretty sure blizzard is attempting to do this. At the beginning of the beta, they announce a 'Pro League' which will only have one division, and will work almost exactly as you have described here. Details about the Pro League have not been confirmed yet, but it is rumored to be invite-only.
"Matchmaking has nothing at all to do with league or rating, but rather uses a hidden rating that is most probably some sort of ELO like system"
DrunkenJedi, do you have a source for this? It's what I've been reading on the battle.net forums but I've yet to hear it from an official Blizzard representative. From Sirlin's article, I assumed the forumites were wrong.
1. Like WC3, SC2 uses a "hidden ELO" for matchmaking. Platinum players will encounter Diamond players.
2. Bonus pool is a "positive version" of the "lose points if you just sit at the top and don't play" function seen in WC3. This is solely because it causes inflation. Just like in investing, if you have a lot of money and let it sit, it'll lose value. Top players must continue playing to keep up.
ranking system is screwy if you ask me! During ladder resets, I've placed in Platinum, Gold, Silver and Copper. I fight opponents from random rankings. I have no idea what the "metal" league even signifies other than another kind of separate division.
A good article, with one mistake that I noticed:
You seem to be under the belief that the displayed rating on your Ladder screen in SC2 is what is used in matchmaking - it is not.
The system operates almost identically to the Arena matchmaking system in WoW. The displayed rating, which is affected by the bonus pool, has absolutely no relevance outside of ranking you within the division (ingame, at least - players will obviously parse the database and create massive "everyone" rankings).
However, you have a second rating that has no interaction with the bonus pool, which operates, basically, like TrueSkill. This is the rating which determines matchmaking decisions. Thus, the bonus pool cannot in any way impact matchmaking.
@Cory
I believe the goal of playing as little as possible may become a reality when people realize that playing as little as possible gives you the maximum number of points for the minimum amount of work. I play an hour tonight and an hour tomorrow, and get 200 points. Or I play two hours tonight and get 150 points. Of course I will wait until tomorrow, to do the same amount of work to get a greater reward. By playing when you don't have Bonus Points, you are investing the same amount of time and work for a lesser payoff.
If your argument is that people will ignore the carrot-on-a-stick that says "Don't play anymore today", then I bet those people will also ignore the rank inflation carrot-on-a-stick from Bonus Points. In which case why have Bonus Points at all?
@PoisonDagger
Actually I think "encouraging people to play when they wouldn't want to" is exactly what this system can do. There is a cap on how many Bonus Points you can have, so if you don't play for say 5 days, you stop accruing Bonus Points. Thus even if you don't want to play for the next 2 weeks, you are encouraged to play in order to deplete your pool of Points, so that you have room to accrue more Points. If you don't play, then you lose those points that you could'e accrued.
For more information about the SC2 ranking system, this is an amazing post:
http://www.teamliquid.net/forum/viewmessage.php?topic_id=118212
Here's the basic rundown:
SC2 uses two ratings, a hidden one (MMR) and a displayed one (ELO). You can read that post above for more information about how they interact, but basically the hidden rating is what determines who you are pitted against while your displayed rating is more of a carrot on a stick.
While the game always tries to find players who have the same MMR, your ELOs can be drastically different (because MMR moves much faster than ELO). However, your ELO discrepancy determines the Favor system as well as how many ELO points you gain or lose for a match.
So if Bob starts out at 100 rating, but has an MMR of 500, he will find himself playing against other players with MMRs of 500. After the game has been out long enough, your MMR and your ELO will meet, so it's reasonable that he will be playing against people who have both a 500 MMR and a 500 ELO. That means that his opponents will be Favored, and Bob will gain the most points from a win and lose almost no points for a loss.
Here's the kicker: the bonus pool only applies to your ELO.
So in Bob's case, lets say he has a 100 bonus pool. His MMR is 500 and his ELO is 100, and he's winning half his games. His ELO is quickly jumping to meet his MMR, because he's getting 40 points for a win and only losing one point for a loss. Eventually he gets to 500 ELO, and now he's getting the same number of points for a win as he does for a loss and fighting against Even ranked opponents.
So where might inflation come in? Well, what if Bob is already at 500 ELO and 500 MMR, and he's saved up a 100 bonus pool? He plays a bunch and gets his ELO up to 600, but he hasn't actually improved as a player so his MMR is still at 500. It may seem that his ELO has been inflated, but now Bob is fighing in matches where *he* is favored. Now Bob is losing a ton of points for each loss, and pretty soon his ELO will normalize back to 500.
Therefore, the bonus pool does not cause ELO inflation.
Just a general disclaimer: I didn't come up with this, I'm just paraphrasing the link above.
In response to Chronocide's post above linking my Ladder Analysis thread on TL, I don't think Sirlin has it wrong in a fundamental sense. The matchmaking rating that is used to pair players is akin to the fourth rating that he proposed in his initial design, only the MMR is hidden. Ultimately, the MMR is not relevant to the player -- it's an arbitrary value that is relative to the overall player pool and it would vary too wildly just like it does in WoW Arena. The only thing that matters to players is the tangible ladder rating that changes as a result of games played and ranks you within your division. In the grand scheme of things, your MMR and ladder rating will ultimately converge because the system will properly place you where it estimates you to be.
I helped with the analysis of the ladder system for SC2 that was presented in Excal's thread from TL. First and foremost, the analysis we did is not necessarily correct (obviously). Our analysis stemmed in large part from observations about WoW's arena system. Obviously they have made refinements, but the MMR system in arena worked well and it makes sense that they would use a similar system in SC2.
There are some interesting points in this blog post that are making me think. I think there is strong evidence that there is a hidden matchmaking rating (MMR) that governs the actual matchmaking. I am unsure precisely how it is now connected to your displayed rating. Do bonus points affect your MMR?
The case for MMR is most apparent in that during beta you could move up from one league to another without being anywhere near the top of your league/division in displayed rating. So your example of Alice taking a break and coming back to easier opponents would be invalid with the MMR system put forth by Excal and myself (and again, it may be incorrect). She would face people matched based off her MMR and her displayed rating would correct simply due to stored bonus points. The MMR-displayed rating system actually functions similarly to your original design of displaying a psychologically appealing (and mostly accurate, so long as people maintain a certain level of participation) statistic, with a hidden rating that does the real matchmaking.
I see. I guess we don't quite know for sure if the bonus points are affecting the matchmaking stat or not, then. You're saying that your educated guess is that they don't affect matchmaking. I could imagine a case for doing it either way, I guess. There's something to be said for leaving matchmaking untouched by this other stuff, but there's also something to be said for keeping matchmaking and displayed stat more in sync. The side effect of Alice getting a few easier matches (and consequently some other poor soul getting a bit harder matches) is not so terrible of a thing anyway (if it worked how I guessed), because I suspect it wouldn't be *that* far off anyway.
We'll probably know the real answer for sure pretty soon.
Sara, and of course there's also the sadist position. :)
I definitely think SC2 has it right; If you assume skill inflation, rank inflation almost necessarily follows. "Average" players over time have more experience and thus more "skill" compared to an "average" new player, so it makes sense that new players start at the 50th percentile at launch and, years later, start closer to the 90th percentile.
Look at the Team Liquid link that CHRONOCIDE posted, and especially the quote block at the end of that post. If that information is correct,
-There is a hidden rating that works just like in other games, with no gimmicks (matchmaking rating, MMR)
-There is a public rating that the bonus points do affect (displayed rating, DR)
-The DR changes are calculated based on your DR vs your opponent's MMR
-> DR's converge towards MMR's, which do not inflate
-> Bonus points give a temporary boost to your DR, but eventually the inflated rating will get dragged down to the correct level
This is essentially the same as what CHRONOCIDE said, but I didn't really get it until I read the original, so I'll try a shorter version. Looks like a really slick system to me.