Spring Benchmark

[manolo_]

i know its MRT, but is there any yet? we could make a list with different configs, so player will see if its reasonable to just upgrade there cpu for their existing mainboard and stuff. i mean it could be a mini-mod with some 100-1000 units + explosions,...

[Beherith]

Ive been thinking of the same thing lately.
And to make one i thought of launching a game, then pausing, then /giving proper bases and shit, and starting the battle, and unpausing. So all one would have to do was launch the replay, and it would instantly start at a nice scene. Then look at the cpu use (maybe even graph cpu use and FPS).

Then i realized it would break with next release, so I put it on the back burner, but even that is getting very crowded.


Edit: also, can a lua script do a /give command? I think it would be longer lived than a replay if a lua script spawned benchmark state.

[Masure]

Why a specific replay couldn't do the job ?

edit : ok replay is also spring version specific

[zwzsg]

Well, you could you use my pseudo-save mission system, with two teams of NullAI having lots of queued and repeat command.

Tiny change in the engine could potentially result in different outcome in the battle snowballing into major differences, but I believe it could be avoided with a little care in the layout: For exemple, make sure attack unit never come in range of factories and fusions/MMM, so that that base can be static and neither grow nor shrink. And place strong defense just a little behind the frontline, so that any side winning a fight on the frontline still still lose his units right after it breaks the frontline. Of course for better result, you should use my "Regenerative AI" aioverride, or else you the first side getting a temporary advantage would end up stacking more and more units.

As an exemple, see the background of the single player frontend of KP 4.0, when launched through the desktop shorcut installed by the full installer. It is a staged battle, that will replay almost exactly the same, no matter what is your version of Spring. My aim was to have a pretty looping animation, not to do benchmarking, so I aimed at a light load instead of a heavy stress, so of course you'd have to redo one with more action going on.

Edit: also, can a lua script do a /give command? I think it would be longer lived than a replay if a lua script spawned benchmark state.

Currently, I do it with gadget. In theory, it could be done in widget that /cheat /give /godmode GiveOrder etc...

Hmm, porting my save/load/menu/RegenAI to a single external widget sounds like a plan, if I fail to convince modders to integrate it in its gadgets form.


Personnaly, to measure performance I would just spec AI battle. But then that's because KP has a LuaAI that plays in very regular way, expanding at a regular rate, and playing without any big surprise. The only thing that seem to matter is how choky is the map. C++ AI are instable, and get broken way too often when the Spring engine update, when there's something they don't understand in the mod, or simply can grow at very different rate depending on random change in build order and initial raids. But either way, AI matches still have a random element in them, and while it would be a better perfomance indicator of how actual games fare than artificial staged battle, they wouldn't give as a precise figure you can directly compare between Spring version unless you ran lots and averaged them.


I guess I could include a "Staged Battle for Benchmarking" in my next B.A. save mutator. Then I'm not a BA player so wouldn't know what a representative BA battle should look like. Behe, would you make one for me? Tips1: Use latest BA mutator from down there. Tips2: Open .sav, delete the whole heightmap line, and add aioverride=Regenerative AI; into the [TEAMX] sections. Hopefully you'd the kind of person that don't need to be explained how to give units and orders to a pair of NullAIs.

[knorke]

Tiny change in the engine could potentially result in different outcome in the battle snowballing into major differences, but I believe it could be avoided with a little care in the layout

I think replays would be more usefull as they play the same every time.
Yes, they get broken every version but so do most mods. (ie xta hovers etc)
And with replay you could test various extreme situations, like /give 100 commander, pathing of lots of units or have 20 nukes fire at same spot to test performance with many particles.


There already is Spring.GetFPS so why not just make a widget that reads FPS every few seconds and calculates average FPS while running a replay.
And then just have an benchmark.exe that runs spring.exe with a replay and different graphic settings. Does not seem so hard? I think i might try.

Question:
How do you control what lua widgets are active at gamestart, for me it seems totally random.
Like you would want to have the same lua widgets running on all computers, not matter what the user settings. But they should not be messed up of course.
Oh and is there a way to see from lua if shadows, bump water, blur effect etc. worked without parsing infolog.txt?

[jK]

http://trac.caspring.org/browser/trunk/mods/benchmark

[knorke]

lol so its already been done.
uhm, how do i dl from that site, like the .sdz for the complete mod?

[SirMaverick]

lol so its already been done.
uhm, how do i dl from that site, like the .sdz for the complete mod?

checkout svn://svn.caspring.org/trunk/mods/benchmark in spring/mods as .sdd directory

[knorke]

thanks.
so i need some svn client? ok.

[Beherith]

Yeah, tortoiseSVN is pretty good.

[manolo_]

is there a complete manual for it yet? so we could make an easy-to-use-noob-friendly benchmarker, also a webpage where u could add ur system-specifics and the result.

[zerver]

Then look at the cpu use (maybe even graph cpu use and FPS).

Indeed, both are equally important.

[Pxtl]

I just use the air-combat script to do a quick benchmark, personally. Why not just standardize map + mod + air-combat-script and a camera widget that zooms in and out?

[Argh]

What I'd really like is a graph output for every Gadget and Widget, synced and unsynced, so that we can take snapshots of performance of that code really easily and at a glance, and make changes live and then watch what happens when we reload. It's too hard to pinpoint load issues atm.

[manolo_]

also it should check different settings like water, shadows,...

[jK]

What I'd really like is a graph output for every Gadget and Widget, synced and unsynced, so that we can take snapshots of performance of that code really easily and at a glance, and make changes live and then watch what happens when we reload. It's too hard to pinpoint load issues atm.

There are profilers for lua (LuaUI, LuaGadgets Synced&Unsynced) for a very long time ...

[Argh]

Where can I get these tools? And do they give us a visual graph, so that we can see patterns of behavior during high and low load?

[very_bad_soldier]

Where can I get these tools? And do they give us a visual graph, so that we can see patterns of behavior during high and low load?

http://widgets.springrts.de/index.php#67
AFAIK it shows current processing time per frame and total processing time or something.

[jK]

Where can I get these tools? And do they give us a visual graph, so that we can see patterns of behavior during high and low load?

As always check CA.