Imrpoved ranking system - Page 3

Imrpoved ranking system

Requests for features in the spring code.

Moderator: Moderators

User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Imrpoved ranking system

Post by hoijui »

hehe :D cool idea.
when hosting a game, the host can specify an SQL statement which checks each players validity (pseudo):

Code: Select all

numRows = SELECT id WHERE player.name=={player that wants to join} AND averageKDratio AND mod=={hosted mod}
if (numRows == 1) {
   player can join
}
for sure, a very flexible approach.
what about complexity? maintainability?
User avatar
bibim
Lobby Developer
Posts: 959
Joined: 06 Dec 2007, 11:12

Re: Improved ranking system

Post by bibim »

Error323 wrote:Within the Spring engine, after a game has ended the statistics (units killed, damage dealt etc.) would be posted to a webserver via e.g. libcurl by the Spring server and each Spring client. These statistics will be matched against eachother to prevent tampering with the source by a single player.
And what happens for unsynced/timeouted players ? And what do you do for 1v1 if data don't match ? And for team games, if all the players of one team autosend the same fakedata after a game if they lost it, what happens ?
Error323 wrote:Even each game could be tracked, the demofile could be posted aswell
Everyone doesn't have enough bandwidth to upload demo of each played game...

If one really wants to implement "secure" automatic ranking without requiring players to register new accounts, I think the way to go is to use a list of trusted hosts (or autohosts) that can submit data.
Error323 wrote:The play against self issue, which can be `solved' by also storing the ipaddress (gained by webserver, not in the Spring source) and using the unitactivity stats.
And what about nated players who all share the same IP ?
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Improved ranking system

Post by Error323 »

bibim wrote:And what happens for unsynced/timeouted players ? And what do you do for 1v1 if data don't match ? And for team games, if all the players of one team autosend the same fakedata after a game if they lost it, what happens ?
When a battle has players that timeout I think the best choice is to not post the game at all, this is just a problem that occurs to everyones loss. If in a 1v1 data doesn't match the game/ranking will also not be posted. Players will guard eachother when they take ranking seriously.
bibim wrote:Everyone doesn't have enough bandwidth to upload demo of each played game...
Well this is secundary, but a solution would be to first compress the demofile. And also not every player should post this blob in a game, only the host posting it would suffice.
bibim wrote: If one really wants to implement "secure" automatic ranking without requiring players to register new accounts, I think the way to go is to use a list of trusted hosts (or autohosts) that can submit data.
This might be a solution but I don't think it's necessary. When players take their ranking seriously they will guard eachother. And when an entire clan tries to tamper with the system, eventually this will be noticed, drastic measures could be taken (nick ban etc).
bibim wrote:And what about nated players who all share the same IP?
This is a good point, but then the unit-activity would kick in. If both sides have `valid' unit activity the game would be posted and ranking would be specified. But perhaps there are better solutions for this problem.
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

hoijui wrote:hehe :D cool idea.
when hosting a game, the host can specify an SQL statement which checks each players validity (pseudo):

Code: Select all

numRows = SELECT id WHERE player.name=={player that wants to join} AND averageKDratio AND mod=={hosted mod}
if (numRows == 1) {
   player can join
}
for sure, a very flexible approach.
what about complexity? maintainability?
Complexity will be taken care of by the webserver, the springhost / clients will just send raw statistical data about the game. The order of checking data would be O(n^2) or something with a max of 32 players? Perfectly duable imho.

Maintainability on the Spring engine would be nearly zero since all it does is send raw data. On the webserver this data would just be stored in a database and the ranking/deductions can be altered lateron as wanted, I think it's a very flexible approach.
User avatar
aegis
Posts: 2456
Joined: 11 Jul 2007, 17:47

Re: Imrpoved ranking system

Post by aegis »

imho it should be done from trusted dedicated servers.
User avatar
bibim
Lobby Developer
Posts: 959
Joined: 06 Dec 2007, 11:12

Re: Improved ranking system

Post by bibim »

Error323 wrote:When a battle has players that timeout I think the best choice is to not post the game at all, this is just a problem that occurs to everyones loss. If in a 1v1 data doesn't match the game/ranking will also not be posted. Players will guard eachother when they take ranking seriously.
So if I see I'm loosing I just have to disconnect or send fake stats at the end so the game won't be taken into account ?
Error323 wrote:Well this is secundary, but a solution would be to first compress the demofile. And also not every player should post this blob in a game, only the host posting it would suffice.
Ok then in this case it's a "trusted" host.
Error323 wrote:This might be a solution but I don't think it's necessary. When players take their ranking seriously they will guard eachother. And when an entire clan tries to tamper with the system, eventually this will be noticed, drastic measures could be taken (nick ban etc).
Yeah, but how would you know who is lying ? I don't see the point of implementing data reconciliation for all players involved in each battle if it doesnt prevent such problems...
Error323 wrote:
bibim wrote:And what about nated players who all share the same IP?
This is a good point, but then the unit-activity would kick in. If both sides have `valid' unit activity the game would be posted and ranking would be specified. But perhaps there are better solutions for this problem.
What do you mean by unit activity ? It's very easy to make units move by themselves with AI, Lua ...
User avatar
TheMightyOne
Posts: 492
Joined: 26 Feb 2007, 14:32

Re: Improved ranking system

Post by TheMightyOne »

bibim wrote:So if I see I'm loosing I just have to disconnect or send fake stats at the end so the game won't be taken into account ?
imo the rakning shouldnt depend just on the win/lost count. thats not necessary shows your true skill. it should be more like a mix of all the other statistics, like dmg dealt/ressorces shared/etc. so somebody who deals a lot of damage might have a worse rank then a person who is dealing less dmg but is the perfect ally who always helps out by sharing ressources. it means even if youve lost the game your ranking can still improve depending of how well you actually did. if ive dealt more dmg then my opponent but lost the game i might still get more points. besides i think that we should think of a ranking system which wouldnt decrease your rank if youre losing coz this way losing wouldnt "hurt" so much, since thats the main reason why ppl would unplug their internet cable.
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

I've been going back and forth on designing a well defined formula/function, but everything I came up with didn't quite cut it. And then I thought what about this: Given all the features (unit kills, metal used, etc.) one can design a neural network that learns the rating function. This ofcourse requires trainingdata, and what data is better than objective human-defined ratings (notice how objective is the keyword in that sentence).

Certain people can assign ratings from e.g. [-5, 5] to players of a spectated game. Or if the community is 'mature' enough, every player itself can assign ratings to all other players, knowing that these will have nothing todo with the actuall player being rated.

Now the benefit of this is that no-one will ever know the actual function. Though this could also be regarded as the downside. However (and this really is of great value) if players/trustees are willing to tag players on played games in an objective manner we can really find out if the current statistics are enough to define a good rating function.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Imrpoved ranking system

Post by hoijui »

Error323 wrote:Now the benefit of this is that no-one will ever know the actual function. Though this could also be regarded as the downside.
That is no problem. if you wanted to know the function, you could do a function approximation with sample values. This can be done with mathematically "perfect" approximations, or with AI algorithms.

an other idea:
let us make the algorithm for finding out a formula modular:
have a well defined API, so we can plugin the algorithms. we could then make a web interface to upload algorithms and open a new project "Spring-Rank-Evaluation" with contests and all...

;-)
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

hoijui wrote:
Error323 wrote:Now the benefit of this is that no-one will ever know the actual function. Though this could also be regarded as the downside.
That is no problem. if you wanted to know the function, you could do a function approximation with sample values. This can be done with mathematically "perfect" approximations, or with AI algorithms.

an other idea:
let us make the algorithm for finding out a formula modular:
have a well defined API, so we can plugin the algorithms. we could then make a web interface to upload algorithms and open a new project "Spring-Rank-Evaluation" with contests and all...

;-)
Oh I like that too! Still, it would be a good idea to use a neural network and see if the statistics spring provides can represent a ranking function at all. I"m now secretively working on the sql schema =).
ZellSF
Posts: 1187
Joined: 08 Jul 2006, 19:07

Re: Imrpoved ranking system

Post by ZellSF »

there is no automated way of making accurate rankings and allowing the players to have any influence on rankings has even more serious flaws..
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

ZellSF wrote:there is no automated way of making accurate rankings and allowing the players to have any influence on rankings has even more serious flaws..
I'm not really sure if you understand the concept here. Players that are tagging are merely providing trainingdata for the supervised learning algorithm. The algorithm will then use this data to construct a function which can give players a ranking.

On another note: If you have doubt in the system and think it will not work, please provide constructive feedback. Just posting that it won't work has nothing of value to anyone (and is just a waste of ur time).
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Imrpoved ranking system

Post by hoijui »

yeah.. if everyone would be able to tag (read: specify which player is bad/middle/good), that could be a bad idea. but of course this would not be allowed to be done by everyone.
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

Yes, perhaps I'm a bit too naive there but I can also take the current ingame time of players into account when they are tagging. Or build up a trustnetwork per mod etc.
ZellSF
Posts: 1187
Joined: 08 Jul 2006, 19:07

Re: Imrpoved ranking system

Post by ZellSF »

Either there's many players capable of tagging (which opens for serious abuse) or there'll be a selected few and there's very little chance they'll even bother.

Also, as mentioned, there is no way a computer algorithm can be used to determine which players are good, no amount of data collected from humans will change this.

What exactly is said algorithm supposed to learn that makes it capable of telling a good player from a bad one?

I did understand the concept, and it's very very flawed and sounds like it would require even more work to implement than all the other flawed "solutions".
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

ZellSF wrote:Either there's many players capable of tagging (which opens for serious abuse) or there'll be a selected few and there's very little chance they'll even bother.
This is simply not true, I've spoken to some people already and they would be willing to help tagging.
ZellSF wrote:Also, as mentioned, there is no way a computer algorithm can be used to determine which players are good, no amount of data collected from humans will change this.
I'm extremely curious how you know this? Is this an hypothesis or can you prove this statement. Also I have not yet been exact on the precise data/features/attributes I will be using for the learning.
ZellSF wrote:What exactly is said algorithm supposed to learn that makes it capable of telling a good player from a bad one?
The relation between the statistical data and the tagged ranking.
ZellSF wrote:I did understand the concept, and it's very very flawed and sounds like it would require even more work to implement than all the other flawed "solutions".
Well if you can prove this, I would like to see it.
ZellSF
Posts: 1187
Joined: 08 Jul 2006, 19:07

Re: Imrpoved ranking system

Post by ZellSF »

This is simply not true, I've spoken to some people already and they would be willing to help tagging.
You need either lots of people or few people that are willing to watch replays and for the learned rating system to be decent, those people also needs to be very skilled players.

I have no doubt there are people willing, I do have my doubts they both know how to play well enough and are numerous enough.
the relation between the statistical data and the tagged ranking.
Which means absolutely nothing because there isn't any statistical data that can seperate a good player from a bad one. No amount of training data can change that.

The correct answer to my question would be telling me WHAT statistical data is supposed to determine skill.
I'm extremely curious how you know this? Is this an hypothesis or can you prove this statement. Also I have not yet been exact on the precise data/features/attributes I will be using for the learning.
Well if you can prove this, I would like to see it.
You can't prove your system works either.
User avatar
TheMightyOne
Posts: 492
Joined: 26 Feb 2007, 14:32

Re: Imrpoved ranking system

Post by TheMightyOne »

dude, youre not helping...
User avatar
Error323
AI Developer
Posts: 237
Joined: 28 Nov 2006, 16:46

Re: Imrpoved ranking system

Post by Error323 »

The correct answer to my question would be telling me WHAT statistical data is supposed to determine skill.
I will, but until then you simply cannot make such bold statements without some evidence. It leads to nothing, please don't get me wrong I'm very happy with critisism, but only with proper arguments. Saying that there isn't any statistical data that can seperate a good player from a bad one is only worth something when you can give me an explanation as to why.
You can't prove your system works either.
I never stated it would work, only if there is a correlation, the learning algorithm will find it. And if it's good, we can use it.
ZellSF
Posts: 1187
Joined: 08 Jul 2006, 19:07

Re: Imrpoved ranking system

Post by ZellSF »

I don't need evidence to see that wasting lots of effort searching for something that most likely never can be found is not a good idea.
Saying that there isn't any statistical data that can seperate a good player from a bad one is only worth something when you can give me an explanation as to why.
Because there isn't a single stat that is related to skill in your average random game. How about you start naming the statisitics which could determine skill?
Locked

Return to “Feature Requests”