TrueSkill: ranking, replay upload, inactivity penalties...

TrueSkill: ranking, replay upload, inactivity penalties...

SpringRTS Perl Autohost for Dedicated Server

Moderators: Moderators, Lobby Developers, SPADS AutoHost

Post Reply
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

Built-in TrueSkill support for battle auto-balancing
The built-in TrueSkill is totally different and independant from the TrueSkill computed on the replay site. It uses a slightly modified version of TrueSkill to handle some particularities of RTS games and to take into account inactivity penalties. For each game (BA, ZK...), 4 independant ratings are computed: Duel, FFA, Team and TeamFFA (+ 1 global rating which isn't used for balancing, but for global ranking only). Anyone can check his TrueSkill ranking for all games by saying "!ranking" to the bot named "SLDB" in the lobby.
I have some questions regarding the TS system.

- If I enable trueskill on a host, then disable it later, do the battles made when skillMode = rank not count for trueskill? (imo they shouldn't)

- is there any way i can stop games from counting for the trueskill on the replay site, but still upload the replay?
(currently i added two "unranked" rooms where I set skillMode=rank and disabled the end-of-game replay upload, but it'd be nice if to be able to upload the replays)

- how exactly are the inactivity penalties applied?
on the replays site, the rankings for MF show some people with inactivity, but there doesn't seem to be a way to know what it means exactly. Is the player's inactivity "spree", or was it something applied in the past but no longer relevant?
dansan
Server Owner & Developer
Posts: 1203
Joined: 29 May 2010, 23:40

Re: SPADS AutoHost

Post by dansan »

The replay site does not compute any TS rating anymore. All ratings that are displayed there are calculated by SLDB. If the replays are uploaded or not does not matter at all.

AFAIK SLDB observes all matches played on SPADS autohosts and uses their results for TS calculation. Which skillMode-setting they use or if replays are uploaded does not matter.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

Dansan summed it up well.

The skillMode setting in SPADS is really just a way to configure how SPADS estimates a player's skill when auto-balancing teams. It has no impact on the match being taken into account or not for TS. In the same way, replays being uploaded on the replay site or not doesn't matter, because TS is computed at SLDB side, not on the Spring replay site.
raaar wrote:- how exactly are the inactivity penalties applied?
on the replays site, the rankings for MF show some people with inactivity, but there doesn't seem to be a way to know what it means exactly. Is the player's inactivity "spree", or was it something applied in the past but no longer relevant?
The inactivity penalties system is briefly explained here.

If you want more details, here are some explanations concerning the SLDB inactivity penalty system parameterization:
- "minMu=24" => penalties are only applied to players having "mu" (= "estimated skill") greater than 24
- "maxSigma=25/6" => penalties are only applied to players having "sigma" (= "uncertainty") smaller than 25/6
- "muPenalty=0.03" => each inactivity penalty applied decreases "estimated skill" by 0.03
- "sigmaPenalty=0.01" => each inactivity penalty applied increases "uncertainty" by 0.01
- "maxPenalties=200" => the maximum value for each inactivity penalty counter is 200
- "threshold=5" => if a player doesn't play 5 matches in a month for a given game and if all previous conditions are met, then he receives 5-nbMatchPlayedInMonth penalties for this game (and his inactivity penalty counter is increased). On the contrary, if a player has an inactivity penalty counter > 0 for a given game and has played more than 5 matches in the month for this game, then his inactivity penalty counter is decreased by nbmatchPlayedInMonth-5

Each player has different inactivity penalty counters for each game.

If a player has an inactivity penalty counter = 0, it doesn't necessarily mean he has been active recently. Maybe he just has an estimated skill < 24 or an uncertainty > 25/6 (in these cases, inactivity penalties aren't applied because they don't really make sense).
If a player has an inactivity penalty counter > 0, it means this player received inactivity penalties at some time, and didn't play enough recently to fully recover from this inactivity period.
The inactivity penalty counter can be seen as an estimation of how far a player should currently be from his "normal" skill due to inactivity, some sort of rustiness indicator...
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

Thanks.

There should be an option to have the specific servers not count for TS rank.

In my opinion, the way the current TS ranks are presented is too complicated. I'd highlight the current TS rating and hide currently inactive players from the top X listing.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

raaar wrote:There should be an option to have the specific servers not count for TS rank.
The main goal of the TS system is to help balancing matches by having a skill estimation more accurate than in-game time based ranks. Why should we make this system less efficient by ignoring some legit matches?

The TS balancing system that we use could be entirely hidden. It's not a ranking system for competitive matches, whose goal is to rank players according to strict rules with dedicated ranked matches etc. We don't have such a ranking system currently (even if we display TS based leaderboards for top 20 players, for information only).
raaar wrote:In my opinion, the way the current TS ranks are presented is too complicated. I'd highlight the current TS rating and hide currently inactive players from the top X listing.
TS are just displayed as they are stored/computed, for information. They shouldn't be seen as a competitive ranking system. Also, when a game has sufficient active players, inactivity penalties have quite the same effect as what you describe.
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

I'm wondering again about simple ways to configure some of my spads hosts to not count for TS rank, even if they use existing TS values for balancing teams.
The main goal of the TS system is to help balancing matches by having a skill estimation more accurate than in-game time based ranks. Why should we make this system less efficient by ignoring some legit matches?

(...)
TS are just displayed as they are stored/computed, for information. They shouldn't be seen as a competitive ranking system.
Social reasons. A public, visible player skill number that goes up/down depending on outcome means the match is effectively ranked. This can incentivize players to play some of the time but can also be slightly toxic, indirectly causing people to spectate more and play less. I'm mildly convinced that the lack of "casual" servers has been doing more harm than good for many years.

I think it's also discouraging for new players to be effectively forced to play a "ranked" mode and lose rank by default. Having to "ask" for a casual battle is yet another barrier.

I've been using the feature that battles where someone has resource bonus do not influence TS to make some games casual by adding some bonuses (even if it's 1%), but it's not the same thing. Casual should be the default playing mode.

The TS balancing system that we use could be entirely hidden. It's not a ranking system for competitive matches, whose goal is to rank players according to strict rules with dedicated ranked matches etc. We don't have such a ranking system currently (even if we display TS based leaderboards for top 20 players, for information only).
There could be a privacy mode where only the user has access to its own TS, other people would see "?", but it would only work if TS for that game/type is hidden everywhere, not just on specific servers.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

TS should actually just be seen as a part of the balancing algorithm.

It totally makes sense to prefer hiding the skill values used by the balance algorithm so that players don't think they are in a competitive environment as you said, but then you must hide them from everywhere (all autohosts etc.), which is not easy as everyone can run an autohost and thus get the values. Also it would require some changes in the replay site to hide the values entirely for specific games/mods.

Using TS to balance but not using the results to update the values will progressively make the TS system less accurate and less useful.

That being said, if you really want to bypass the system and make the balancing algorithm less efficient, you can make a plugin which quickly enables cheats and disable them at the start of each game. It should be enough to trigger the anti-cheat system which will flag the game as not suitable to improve players skill estimations.
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

I've set two of my spads hosts to use

rankMode:0
skillMode:rank

(instead of rankMode:account and skillMode:TrueSkill)

and they also run from a modified "spads.pl" which has the "my $sldbLobbyBot=" variable set to something other than 'SLDB'

Unlike my other "ranked" spads host, when they started they didn't show the "TrueSkill service available" log, which was what i wanted.
In the same way, replays being uploaded on the replay site or not doesn't matter, because TS is computed at SLDB side, not on the Spring replay site.
This does not seem to be correct. After my config and spads.pl changes, the games that followed were uploaded to the replays site and shown as modifying TS:

game1 https://replays.springrts.com/replay/0c ... b54b3c3868 13.87 -> 13.85
game2 https://replays.springrts.com/replay/1c ... d47317a4f6 13.85 -> 13.82
game3 https://replays.springrts.com/replay/98 ... 64b8178958 13.82 -> 13.8

What does this mean? does the replay site send the info to SLDB independently, or compute the TS changes? I expected it to show either "13.87 -> 13.87" (no change) or "13.87 -> 13.85" (no lasting change) for all 3 cases....but that's not what happened.


EDIT : i'll try the cheat-toggling plugin method -_-
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

I wrote a simple plugin for it and it's apparently working : the cheat toggle and the warning message show up when the game starts.
CasualMode.txt
CasualMode plugin to disable TS updates
(1.84 KiB) Downloaded 22 times
(renamed from .pm to .txt)
Last edited by raaar on 06 Oct 2021, 22:25, edited 1 time in total.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

raaar wrote: 06 Oct 2021, 03:15 I've wrote a simple plugin for it and it's apparently workin
Of course it works, and it is quite simple (finally you believe me ;) )

I strongly recommend using plugins instead of directly modifying SPADS core, otherwise you have to disable auto-update to keep your modifications.

However, as I said, unfortunately you are basically diverting the balancing system to use it as a competitive ranking system.

As it is implemented, it is clearly not made for this purpose, and by doing that you are both presenting the values used internally for balancing as a valid competitive ranking system (which it is not), and making the balance system much worse by preventing it to improve itself over time as it should.

Hiding totally the internal values used by the balancing system would have been a much better choice than breaking it and presenting it as a competitive ranking system imo.
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

wait a minute... it did NOT work!

https://replays.springrts.com/replay/6b ... 2ed7e8a0c4 25.71 -> 25.72 / 13.68 -> 13.67

The cheat is toggled on/off, but the TS was affected anyway!

is there any condition I missed? must it be on for some time?
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

No you shouldn't need to let it on some time. I will take a look in a few days when I'm back home.
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

The workaround i'm using for now is adding this to the plugin to set 1% bonus to players as they join.

Code: Select all

sub onJoinedBattle {
  my ($self,$userName) = @_;
  queueLobbyCommand(['HANDICAP',$userName,1]);
}

NOTE: that event wasn't handled on the spads version i had (0.12.18) so i had to update to the latest "unstable" (0.12.38).
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

Just took a look at your plugin, it's because you are actually calling Spring cheat commands directly and not through SPADS commands.
I will fix SPADS so that it correctly handles this case also.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

Your plugin should now work as expected with SPADS >= 0.12.39
raaar
Metal Factions Developer
Posts: 1094
Joined: 20 Feb 2010, 12:17

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by raaar »

I've updated spads to 0.12.39, but it doesn't seem to have fixed the issue.

It apparently worked for the first game we've played there after the update:
https://replays.springrts.com/replay/20 ... 64bc1bf16/

But today people played and TS was affected:
https://replays.springrts.com/replay/da ... d47ced07f/
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: TrueSkill: ranking, replay upload, inactivity penalties...

Post by bibim »

raaar wrote: 12 Oct 2021, 23:50 I've updated spads to 0.12.39, but it doesn't seem to have fixed the issue.
It didn't work for any of the games you listed above.
Not sure exactly how you are testing it, but I tested the plugin you provided here without any modification except renaming the file to .pm, and it worked as expected on my system. The game was tagged as using cheats in SLDB and wasn't rated.
Post Reply

Return to “SPADS AutoHost”