A std data format from demo to generate stats/ranking

[albator]

Hi all,

I am working on a prgm to read data extracted from replay and to use it in order to make stats and other stuffs (cf. end)
For that I need to read the data from replays. The question I am asking right now is:
Q1: what the best way to recover this data:

I know there are some widget idea sfrom jool and knorke but I would like to know if there is somehting better or already available. I also know Dansan might work on that but that is far away i guess


The plan is to get all the stats that are usually available at the end the game plus some more:
- damage death [Array]
- damage received [Array]
- metal used
- metal produced
- metal excess
- metal send [Array]
- metal given [Array]
- energy used
- energy produced
- energy excess
- energy send [Array]
- energy given [Array]
- unit active
- unit produced
- unit killed [Array]
- unit recieved [Array]
- unit send [Array]
- unit captured : I dont plan to use this one (
- unit stolen : I dont plan to use this one

- buildpower (jools did it)
- firepower (jools did it)


with a time sample: less than 10 sec

For the values noted as [Array] I would like to get these data as an array (dim2 matrix) between players:
Q2: for which one is it possible (and easy to implement) ?


I also need for each game some additional data, it is great if that is already in the data file, otherwise I will just get it from the replay:
- name of the map
- name of the host (if possible
- team and number of players
- player info
....

I started to create a template file with the data I need: http://pastebin.com/NuMhc4sp


I would like to get some input on:
- the data format used for this file in case you find something supid ....
- do you think I forget anything relevant I might need later


Of course I have not provided anything yet blablabla.... People will judge only when something will be done whatever..


Just so you know, my idea consist, but are not limited to:
- smurf detection based on least square dmg curve (but not limited to) regression over maps.
- finding player of similar type using the same idea of least square dmg (but not limited to) curve regression over maps.
- map DB statistic in order to compare playing style of new player (or unknown smurf)
- ranking based on all these parameters: I am not going to tell more now but it wont be based on dmg only and will involved winning a game as one input
- automation of account merging using CPU/flag/IP and the method explain above.

Q3: I have not see the ip in the replay file, is it possible to recover it ?

Also, if other person are planning to do a similar job, that would be just great we all agree on a similar file format. What do you think ?

[jK]

Q1: what the best way to recover this data:

a lot of the data is directly saved in the demofile, some advanced stuff needs to be created via lua by a headless server.

Also, if other person are planning to do a similar job, that would be just great we all agree on a similar file format. What do you think ?

use XML?

[Jools]

I think it would be helpful to agree to some common definitions. I defined:

buildpower = sum of buildpower of all units
firepower = sum of hitpoints of all units that have a weapon.

You could define them differently.

[dansan]

what the best way to recover this data

IMO as the simulation must run to get that data, the best way would be a widget.

Now if you want all matches to be analyzed, then it will be to CPU-intensive to do it on a central server... although... atm I think a dedicated host with a quad-core would manage it easily. As it can queue the work, and in the CEST-mornings there is a lot of time...

If a central server is not possible, it will work however if either the load is distributed among several servers or (imo better) the widget is run automatically on all/most clients, and continuously transmits (luasocket) to a central server. The client won't have disk-io and storage-problems. The server needs just 1 times the data, but it can compare the data sent by all clients and thus circumvent cheating.

I also need for each game some additional data, it is great if that is already in the data file, otherwise I will just get it from the replay

Teams, Players, Map etc. are all in the replays file and quickly readable. For a start you can use parse_demo_file.py <demo.sdf> - works with plain Python-2.7, and output when scanning this replay this data: http://pastebin.com/LnbwLeyV (which is used on the replay page).

file format

for statistics: RRD :)

[Jools]

I made the thing as a widget, and it works if you are a spectator. LAter I added a player mode too, but I must say there had to be many nil-checks because of that.

Maybe it would be even better to make it as a gadget. Then you could even more data, for example projectiles shot at target and how many of them hit. The question is if that's anything at all meaningful :)

[albator]

Actaually, the only thing I would like to have in addition of what I explain is if that is possible to recover information about who killed a type of building:
like E-Prod, mexes , defence, etc ?

But that is really ultimate nice-to-have data.

Any idea concerning the Array related question ? :

For the values noted as [Array] I would like to get these data as an array (dim2 matrix) between players:
Q2: for which one is it possible (and easy to implement) ?

[Jools]

It is possible for most of your items. Note that spring is quite technical (read: stupid) when it comes to these things. For example, when you build a DT it technically 'dies' instanly and that also counts as a death. But in that case you won't have any information about who 'killed' it.