QuizMe: New Player Skill Rating

Thanks to Daniel, there will be a new world order on our quiz's leaderboard. In his own words (mostly), here's how our new play skill rating will work:

Currently, we rank players by the total number of points earned. After seeing the results of our different trials online, it was obvious that a major problem with using the total number of points was that it is largely dependent on the number of questions answered. It would only be an accurate comparison between users that have answered a similar number of questions.

Other flawed measures: Point average and percent correct -- both ignore the difficulty of a question and percent correct ignores how fast a question is answered.

Our new skill rating addressees all of these problems. Similar to the Elo rating system, our skill rating system takes into account a number of factors to determine how good a player is at answering questions.

The system works something like this:

Each player starts with a skill rating of 1500. Whenever a player answers a question, their current skill rating is compared to the difficulty rating of the question (which is determined by how other people answered the same question) to determine what the expected score for this particular user answering this particular question is. This predicted score is then compared to the score the player actually earned, and the player's rating is adjusted up or down depending on if the player scores higher or lower than expected.

For those of you who are interested in the actual formula:

To determine the expected score:

P = Player's rating before answering the question
Q = The questions difficulty rating

Expected score = 1 / (1 + 10^((Q-P)/800))

The adjustment to the players rating is made with the following equation:

P = Players original rating
E = expected score
S = actual score
K = a scaler that is dependent on the number of questions a player has answered: it ranges from 25 for new players to 10 for players who have answered a lot of questions.

New player rating = P + K(S - E)

Leave a comment