idea: fps logging for benchmark

[knorke]

I made a widget to log the FPS every 10th frame and write it into a textfile. Then, to the office software!

(2v2 xta on eye of horus)
about 1/3 into the replay I got bored of watching and started to alt tab, that is why it goes really wild/low from there.
Guess for meaningfull results one should not do that.
Also attached widget to log springsettings.cfg (made for some other thread) and a widget to circle the camera around the map. (so that camera movement is always the same, like in benchmark demos. guess the zeroK widget with the waypoints would work too)
Together with a replay one replay one could make a benchmark thing?
For epenis, comparing graphic settings, drivers,window vs fullscreen, mods etc.
Not sure if this is interessting enough to make into a real benchmark thing? Or maybe someone else wants to try.

[jK]

You shouldn't use GetFPS(). It's too imprecise.

Instead you should use timers in widget:Update and update the timer only like each 10th frame. So you can compute the fps like this:

Code: Select all

local time_spend_per_frame = (time_now - time_last) / 10
local FPS = 1 / time_spend_per_frame

[knorke]

yes, i noticed getFPS differs alot from what ie fraps shows.

[jK]

Also nice work.
I got a similar idea, but displaying the graph ingame + min/avg/max FPS ranges.

Btw, I wrote a benchmark mod, but its lua is just too .. complicated. So I never developed it further. Also it just benchmarks the engine (esp. specific parts of it, like terrain rendering, texture loading, feature/unit rendering).

[knorke]

I thought about an ingame graph though but was not sure how much it would influence fps. (probally not much if it is done well)
Since typical replays are quite long, I want to use /cheat etc to create a few-minutes-long replay that recreate "interessting" situations.
ie
-many units start moving
-fighting
-lots of fighters on patrol
-need more ideas?
Since one knows from creating the replay that ie 30sec=100 commanders exploding one could see how hard which situation is.

I think spring.exe can be passed a springsettings.cfg?
Then one could also make a simple .bat file that runs the same replay multiple times, with different settings.

edit:
Spring.Echo (Spring.GetTimer ()) prints "<lightuserdata>"

[hoijui]

abma introduced validation test (see test/validation/ in engine source code). this could be an analogous thing. eg.. could be executed once a way for the current build on the buildbot, and graphs could be put online (would require a buildslave with GFX). even if you would not actually run it on a buildslave by the buildbot, using at least some sub folder under test/ with a somewhat similar structure might not hurt.

https://secure.wikimedia.org/wikipedia/ ... re_testing
https://secure.wikimedia.org/wikipedia/ ... ss_testing

[MidKnight]

This is cool!

All of your ideas reminded me of something I thought of earlier this week, when carpenter(?) posted about low framerates on his new GTX 560.

What if we write a benchmarking mod and then bundle it with a script that launches Spring with different settings disabled until it stops crashing/running slowly? A kind of self-diagnostic!

[dansan]

If this turns out to be a robust benchmarking feature, we could try to get this into Phoronix. It could be interesting for them if it could benchmark different things like
* lots-of-pathing -> CPU-intense (MEM-intense?)
* insane-water-effects -> GPU-intense
* complex AI(s) -> CPU-intense
* etc (you know better than me)

Can "headless" run a game? -> Could it be used to benchmark a game without graphics?

The idea would be to promote SpringRTS through this. Target audience is Linux enthusiast, interested in graphics, video, gaming (and CUDA and stuff). It could attract players and devs. They often promote and benchmark different (OSS) game engines... I guess everyone knows Phoronix...

The demos would need to be scriptable, reproducible and stable. They should run representable games... Zero-K and Evo for example. BA and Gundam have IP issues... don't know if KPs geekyness is a plus or not in this case :)

It's unfortunately that spring is so picky about compilers. If it were possible to build it with diff. gcc-versions, llvm-clang and icc they'd love it. In any case it can be used to compare performance on Windows, Linux and MacOSX. If this benchmarking project gets there, IMO it'd be worth a try to send some results to Phoronix and try to get their attention.

[AF]

Ai can already log times, and graphing for AIs is already built into the engine and implemented yet again internally in most AIs for finer grained control

[hoijui]

spring is only picky about compilers and stuff, if you want to sync with others, which surely should not be the case with a benchmark.
there might be compilation errors with some compilers, but they are usually few, and very easy to fix. spring has already been compiled on at least: GCC, MinGW-GCC, VS, Intel CC, llvm-clang
llvm and especially intel cc may need some fixing, but the others should all compile develop fine as is.

[knorke]

The widget writes to file when you type into chat "WRITE FPS" and every 5 minutes, I was not sure if continous writing would slow down things. (probally not much but anyway)
The jK's idea is not used, I had forgotten?

plotfps.txt is a script for use with http://www.gnuplot.info
Copy it next to the fps log file was written to. (next to spring.exe for me)
start the gnuplot program, there will be a console thing.
Type:

cd "C:/Spiele/spring-95.0"
load "fpslog.txt"

It will make a .png graphic in same folder.

If you want to add more log stuff, in plotfps.txt adjust the number where it says
"using 1", "using 2" etc
that number = row in log file. Rest should be obvious from copy&paste

[Beherith]

Heres mine. Makes these: