Elo, Glicko and Trueskill ratings on replays.springrts.com

[djmad]

Howto add the Trueskill into Spads for Balancing

may someone can do this absolutly ultimative best awesome
(sh*t im talking like apple) move to code this in

[hoijui]

stop the shroomin man! apple no tak!

[muckl]

Fehler: Server nicht gefunden
Der Server unter replays.springrts.com konnte nicht gefunden werden.

the server seems not working?

[abma]

Fehler: Server nicht gefunden
Der Server unter replays.springrts.com konnte nicht gefunden werden.

the server seems not working?

http://replays.springrts.com/ works fine here... i guess dns has/had some problems for you?!

[dansan]

Howto add the Trueskill into Spads for Balancing :roll:

may someone can do this absolutly ultimative best awesome
(sh*t im talking like apple) move to code this in :-)

Oh sorry... didn't see this posting until now.

A month ago vbs asked me about such a feature, and I implemented a little demo, but didn't publish anything, because he said bibim was already working on something, and I didn't want to put pressure on him, so.... anyway, the server side has a basic implementation, that can be extended if wished:

Code: Select all

$ python
>>> from xmlrpclib import ServerProxy
>>> proxy = ServerProxy('http://replays.springrts.com/xmlrpc/')
>>> for num in [5890, 168247, 198943, 102095, 204720, 38429]:
...   proxy.xmlrpc_rate_single_user(num, "BA", "T")
... 
38.2120494267062
40.4091281341573
40.1996157605119
17.7110639373982
24.1400297020706
26.847548034585

Function is:

Code: Select all

xmlrpc_rate_single_user(accountid, game, match_type)
game == "BA" or "CD" (chicken def) "NOTA" "RD" "S1944" "TA" "XTA" "ZK"
match_type == "1", "T", "F", "G"

(see models.py#L364 regarding "match_type"
see "Games" on the right sidepanel on the replays site regarding "game")

From the function name you can see that I assume a function to fetch multiple records with one call is needed later for better performance. But at that moment I wanted to make a demo only, so that bibim or vbs can play around with it, and develop the interface/requirements they desire, and then I'll look into it again.

I tried a little Perl copy & paste and trial & error (I cannot code in Perl), and the following works:

Code: Select all

$ perl
use RPC::XML::Client;
my $client = new RPC::XML::Client('http://replays.springrts.com/xmlrpc/');
my $res = $client->send_request('xmlrpc_rate_single_user', 5890, 'BA', '1');
print $res->value;
1820.53324626803

required Perl module: http://search.cpan.org/dist/RPC-XML/

[klapmongool]

I noticed a bug on the replay site while i had this topic open (am too lazy to make new topic for it).

In the description of a battle the side a player played corresponds with the side the player picked in the lobby instead of the side the player actually played.

[dansan]

I noticed a bug on the replay site while i had this topic open (am too lazy to make new topic for it).

In the description of a battle the side a player played corresponds with the side the player picked in the lobby instead of the side the player actually played.

That cannot be fixed atm :(

I'm using only the info available from the demofile header, and that is mostly created at spring start time.
When I have time I'll steal koshis full command stream parsing code and then I'll be able to detect lots of interesting stuff like this one.

Thanx for reporting, I created a ticket: https://github.com/dansan/spring-replay-site/issues/28

[dansan]

BTW: On each replays page you can see now the TrueSkill (or Elo for 1v1) for each player and the sum for the complete team before and after the match. I noticed, that the team with the higher TS wins more often than it looses, so the relative skill estimation is good.... but I didn't make any statistics yet... just am impression... I'll try to run some stats soon.

[very_bad_soldier]

A month ago vbs asked me about such a feature, and I implemented a little demo, but didn't publish anything, because he said bibim was already working on something, and I didn't want to put pressure on him, so.... anyway, the server side has a basic implementation, that can be extended if wished:

Yeah thanks for that! I used it and modded SPADS to use a rank plugin (that can be exchanged) to fetch the ranks. So SPADS was querying your replay site to fetch the TrueSkill values.
But I encountered some other problems:
-how to do smurf detection? SPADS does it and the replay site does it too. How to combine these two mechanisms?
-the system wont be able to detect (well faked) smurfs (which is a huge downside for me and leads to the next point:)
-how to be able to manually chrank?

[dansan]

A month ago vbs asked me about such a feature, and I implemented a little demo, but didn't publish anything, because he said bibim was already working on something, and I didn't want to put pressure on him, so.... anyway, the server side has a basic implementation, that can be extended if wished:

Yeah thanks for that! I used it and modded SPADS to use a rank plugin (that can be exchanged) to fetch the ranks. So SPADS was querying your replay site to fetch the TrueSkill values.

Oh cool! :)
Before going production... or is it already being used? ... I want to setup a 2nd webserver that serves only the ratings, so if I restart the main webserver (which I must do on every code update :( ) you don't get connection errors.

But I encountered some other problems:
-how to do smurf detection? SPADS does it and the replay site does it too. How to combine these two mechanisms?

How does the spads mechanism work? Do you identify two accounts to be the same player and have the same rank, or does it just add rank-points to player accounts?

-the system wont be able to detect (well faked) smurfs (which is a huge downside for me and leads to the next point:)
-how to be able to manually chrank?

On the autohost-side you could just calculate a TrueSkill-rank-mapping (new account starts with 25, a very low TS value is 15, top value are around 40) and assign a value that seems appropriate.... That'd be like rank0=TS15, and then +3 points/rank or similar... or use a reference-players value :)

To propagate a value to the replays-site I can simply write a tiny xmlrpc method to set a players value remotely, or make a HTML-interface for admins (but I think cmdline is the usual spads interface).
IMO the interface should be restricted to known autohosts, to prevent abuse, and chranks could be publicly logged for transparency if desired.

[Jools]

All we need is a way to manually join smurf accounts. They are all known and numebers are not so big.

[dansan]

All we need is a way to manually join smurf accounts. They are all known and numebers are not so big.

That means you have accountID-pairs (or nick-pairs?), that you wish to tell the system that they belong together?

That'd be perfect - on the replays-site it is implemented with accounts: if two or more accounts shall be treated as one, then the oldest one (lowest ID) is used as the "primary_account", of each. Ratings for all accounts are stored only in the "primary_account".

Would a xmlrpc-call like this suffice?

Code: Select all

unify_accounts(accountid1, accountid2)

IMO when unifying the highest rating of each account/game/game_type should be copied to the primary_account.

After an initial smurf data drop, a optional full recalculation of all matches can be run, to really unify everyones accomplishments.

[Silentwings]

Imho manually joining smurf accounts might be useful but as many automated ways as possible will be better. Of course you'll never get them all.

Smurfing isn't a big problem if admins can manually chrank in a way that is compatiable with the rating system. I suggest having two types of chrank:
1) a temporary chrank that effects only the next, say, 30 games of that player, after which time it reverts to whatever rank the automated system has calculated. Of course the chrank rank will affect the calculated rank but as long as the chrank rank is not way off what the real one should be then it won't matter too much.
2) a permanent chrank that's only there as a counter for bugs, etc.

[dansan]

Imho manually joining smurf accounts might be useful but as many automated ways as possible will be better. Of course you'll never get them all.

I have no idea how automatic smurf detection could work....

I have only one idea: If we could use ingame-statistics like how much metal/damage someone produced compared to others, it'd give a good indication if a player is really new, or smurfing. A better indicator may be the big difference between newbies and experienced players (that can easily be read from replays): the MC/s+KP/s.

Smurfing isn't a big problem if admins can manually chrank in a way that is compatiable with the rating system. I suggest having two types of chrank:
1) a temporary chrank that effects only the next, say, 30 games of that player, after which time it reverts to whatever rank the automated system has calculated. Of course the chrank rank will affect the calculated rank but as long as the chrank rank is not way off what the real one should be then it won't matter too much.

TrueSkill is designed to converge very quickly on good values. There is no need to retain a fixed value.

Chranking still helps ofc, as the balancing will be fixed immediately for the next match, smurfing will be generally less useful and the TS algorithm will find a good rating quicker if starting from a more realistic value.
A TrueSkill rating has a "certainty value" (that is not shown on the page, but that is used internally) that adjusts with the players performance fluctuation.
There is a awesome paper on TreuSkill by the maker of the algo the page is using: http://www.moserware.com/2010/03/comput ... skill.html

2) a permanent chrank that's only there as a counter for bugs, etc.

Is it a bug or a feature? ;D
I hope something like that will not be necessary... we'll see when we have chranking.

[Silentwings]

TrueSkill is designed to converge very quickly on good values. There is no need to retain a fixed value.

But it still takes some time to converge. Maybe 30 games is too much and 10 would do.

i have no idea how automated smurf detection could work

Imo trying to do it using game stats will be very unreliable (and temporary chranks by admins are a much better solution for that). At the moment its done using IPs, account ids and so on. That seems to work pretty well and leaves only a few cases to be dealt with manually via chrank.

[dansan]

TrueSkill is designed to converge very quickly on good values. There is no need to retain a fixed value.

But it still takes some time to converge. Maybe 30 games is too much and 10 would do.

OK... but... what is the reason to do this?
If you chrank someone, he'll have some work to do to lower that value if he's intent on smurfing with that account again. If you did connect his account to another account it'll be even more difficult and probably not even desirable.
I'm not against it - I just want to understand it :)

i have no idea how automated smurf detection could work

At the moment its done using IPs, account ids and so on.

Ah OK :)
It'll be no problem to alert the site about the detected smurf (using unify_accounts(accountid1, accountid2)). But to use that data on the autohost(s), either each one must maintain it separately, or we'll need another remote-call to get such information from the replays-site... like get_smurfs(accountid)-->list_of_names_and_IDs or account_info(accountid)-->names+ratings+other_accounts.
That'll be desirable I guess, so that all spads can share a common smurfDB.

[PicassoCT]

How about every new joining board member must send a application to a currently active (and random selected among) boardmember?

Something.. i think therefore i am, and why i actually want to join the spring comunity.

[Silentwings]

OK... but... what is the reason to do this?

If someone smurfs and teh automatic system doesn't catch it, an admin can give him a temporary rank close to his proper one, while te system figures him out.

[dansan]

If someone smurfs and teh automatic system doesn't catch it, an admin can give him a temporary rank close to his proper one, while te system figures him out.

Why not let the system figure him out starting from the rank assigned by the admin, instead if the default value?

[Silentwings]

I'm not sure how the trueskill algorithm works - for all I know it might need a history of games with a self-consistent set of ranks for those games before it can iterate onto the new rank in a sensible way. If it can just relearn at any time from a given value without wild oscillations or other screw ups then, sure.