Where to get older engine dev builds (for bisecting)

[Google_Frog]

Differences in settings are very important. There could be things like LogFlush which change default between versions, if you don't have the ZKL settings you could be affected.

The exact settings are here http://code.google.com/p/zero-k/source/ ... L_default/

Benchmarker has a field for overriding the mod version used by the mutator. You'd have to change the dependance manually.

As for the benchmark itself I have experienced reduced performance with 94.1 as compared to 91.0 but I think a bit too much stock is put in this single benchmarking test. Pathfinding changes between versions should have quite a large effect on the performance on this test because of the way it is set up.

[Licho]

w_update is not what is interesitng, use the values from the gadget not widget (there is also measuring gadget).

Regarding benchmarker just follow wiki instructions here:
http://zero-k.info/Wiki/Benchmarker

after you do that, check "fps_war.dds" in bencharks and add a test case using "default ZKL" config, "null_Vs_null" script and engine name you want to test.

[Beherith]

I'm not completely retarded, so yes, I have disabled logflush and used sensible settings, similar to those in your linked config. By the way, your config does not have logflush explicitly disabled, so you might be affected by that.

The measuring gadget is not present in your linked .sdd. Please link to it.

Edit:
Licho, you also seem to be referring to the widget based dt in your graphs. Please explain why it only counts when you do it.

[Licho]

If you install benchrmarker using instructions linked you will also get latest version of that mutator.

Regarding w_update_dt, yes benchmarker graphs also include it but what is most interesting is g_update which shows biggest difference.

[jK]

There is no technical difference between gadget:Update & widget:Update both get called in the same interval and same code position!
So you are benchmarking random stuff w/o logical connection.
It has a reason why I added --benchmark, and why it contains all the data.
This is esp. important when benchmarking non-syncing stuff.

E.g. you would need to calculate real simfram time & real drawframe time and then compare it with unit count, particle count, feature count, ...

[Beherith]

Licho, if you can come up with a consistent test case, I compiled both 91.0 and 94.1 so that I can run the Very Sleepy profiler on it.

[Licho]

jK there is in the test, one is in gametime and the other in realtime.

Beherith im getting consistent results all the time i run this test tens of times already, just need to finish bisect.

[jK]

jK there is in the test, one is in gametime and the other in realtime.

That's not real simframe/drawtime.
Real simframe time is the time needed to process exact 1 simframe w/o any drawframe or other stuff (the one that is displayed in /debug of newer engine builds).
You just compute the time difference between successive calls to GameFrame, but between those non-sim stuff is processed. So you need to detect that and subtract it from your time.
That way you get 2 theoretical FPS: one drawFPS that you get when you pause the game, and a simFPS that would be achieved when all rendering would be disabled.

[Licho]

I know it's not perfect jK.. What I know is that the test runs at 2x less FPS and takes 3x longer (of realtime) to conclude in 92 than in 91.

So for now im ok with imperfect approximations.

Abma is there any chance of getting the next build? I would like to finish this bisect

[Beherith]

Hey man, the level of your approximation is way off. I modified your test to only spawn max 200 (spawnCount) units at once, over half the line distance, still sufficient load imo.

With 94.1: 4765 units spawned over test
With 91.0: 1629 units spawned over test

I don't think its "dragon" also because decals woudlnt slow down sim that way and because even damage is lower in 94.x
(In 91 it manages to do ~= 120k damage and in 94.x only ~= 60k with 3x more sim load and 2-3x less FPS)

[Licho]

But it is set to only spawn fixed number of units? What did you change can you commit it?

Also in my test 94.1 does LESS damage than 91

[Beherith]

Why the hell are you counting damage? How is that representative of anything?

Diff this with yours. Pay close attention to 'total'.

Code: Select all

function gadget:GetInfo()
  return {
    name      = "War Spawner",
    desc      = "Spawns units for war test. Use with NullAI",
    author    = "Google Frog",
    date      = "14 May 2013",
    license   = "GNU GPL, v2 or later",
    layer     = 0,
    enabled   = true  --  loaded by default?
  }
end

-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------

if (not gadgetHandler:IsSyncedCode()) then
    return
end

-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------

local MAP_SIZE_X = Game.mapSizeX
local MAP_SIZE_Z = Game.mapSizeZ
local CMD_FIGHT = CMD.FIGHT

local spawnDef = UnitDefNames["corfav"].id
local SPACING = 50
local LineLength = 50
local spawnCount = 200
local total=0

local END_FRAME = 30*20*1

local spawn = {
	[0] = {x = MAP_SIZE_X*0.5 - LineLength*SPACING*0.5, z = MAP_SIZE_Z*0.75-200, face = 0, pos = 0},
	[1] = {x = MAP_SIZE_X*0.5 - LineLength*SPACING*0.5, z = MAP_SIZE_Z*0.75+200, face = 2, pos = 0}
	}

function gadget:GameStart() -- GameStart

	Spring.Echo("War Spawner Active")

	for i = 1, spawnCount do
		for t = 0, 1 do
			spawn[t].pos = spawn[t].pos + 1
			if spawn[t].pos > LineLength then
				spawn[t].pos = 0
			end
			
			local unitID = Spring.CreateUnit(spawnDef, spawn[t].x + spawn[t].pos*SPACING, 0, spawn[t].z, spawn[t].face, t)
			total=total+1
			Spring.GiveOrderToUnit(unitID, CMD_FIGHT, {spawn[1 - t].x  + spawn[t].pos*SPACING,0,spawn[1 - t].z}, {})
			--units[t][#units[t] + 1] = unitID
		end
	end
	

end

function gadget:UnitDestroyed(unitID, unitDefID, unitTeam)
	if unitDefID == spawnDef and spawn[unitTeam] then
		spawn[unitTeam].pos = spawn[unitTeam].pos + 1
		if spawn[unitTeam].pos > LineLength then
			spawn[unitTeam].pos = 0
		end
		local unitID = Spring.CreateUnit(spawnDef, spawn[unitTeam].x + spawn[unitTeam].pos*SPACING, 0, spawn[unitTeam].z, spawn[unitTeam].face, unitTeam)
		total=total+1
		Spring.GiveOrderToUnit(unitID, CMD_FIGHT, {spawn[1 - unitTeam].x  + spawn[unitTeam].pos*SPACING,0,spawn[1 - unitTeam].z}, {})
	end
end

function gadget:GameFrame(f)
	if f == END_FRAME then
		Spring.Echo('total units spawned='..total)
		Spring.GameOver({})
		
	end
end

[Licho]

Thank you, we adressed issues you found, new test gadget is not spawning any units.

Test is now very consistent in damage done and several runs make similar results.


Yet, it still shows huge drop in 94.1 against 91.0 (2x less FPS)
Avg FPS on typical run:

91.0: 33 FPS
94.1: 16 FPS
94.643: 22 FPS

Also post 91.0 tests show jitter in various timings:
http://i.imgur.com/Z8PSvKJ.png

[jK]

told you, you need to improve your data mining, here are my results:

system = 91.0
results are clear ...

And no, the results aren't stable at all. The grapher just fails to visualize more than 4 graphs at once (such lines should really use alpha-transparency). But clear is that 91.0 gives __much__ less stable fps compared to 94.1.1 and like upto 4/5 less fps in average.

So the results are contrary to your's -> the data you fetch is not enough to extract informations from it

[Licho]

Ok, thanks, that is very interesting.

So it might be rendering related thing .. hmm what to do next? Ask more people to test, do automated data aggregation?

Regardingl lines yeah i will add alpha and make them thinner i think.