Benchmarking tool for Spring (included in ZKL)

[Licho]

Recent problems show, that we need an objective way to measure performance changes over time and across various versions of game, engine and configurations. Benchmarker is a tool to make this easier.

Wiki page: https://code.google.com/p/zero-k/wiki/Benchmarker

Benchmarker is so far intended for developers and needs people to write more tests (mutators).

Please help if you can :)


EDIT:
Benchmarker now works on both Windows and Linux;

[Silentwings]

How about having a little widget that once per minute records (number of units, gamespeed, (cpu % per player)) and saves data in csv format. It's pretty easy with recent excel/calc to turn multiple csv's into graphs.

I think this would give more realistic indicator of performance than benchmarks which are run on single machine; the limiting factor is always the speed of the slowest player, not how fast my super-duper i7 can pedal!

If there is interest I could code it.

[Licho]

Erm and how do you think it gets its data from? Some mutators have a widget that erm outputs FPS and whatever interests you..
This is a thing that runs benchmarks (mutators), not benchmark set..

People have to write benchmark, including outputting values of interest..

This thing simplifies running various test cases, gathering of data and presenting them.

It is meant to compared various test cases on the same machine!

[Silentwings]

Ofc anything that is in luaui can only get accurate data if the game honesty reports fps/cpu etc - since benchmarks are mostly for devs I don't see this as a problem. The data I listed is all accessible from luaui.

I agree that what you've implemented is, once people write benchmarks, 100% of what you need to compare two spring versions on the same machine and I'm not saying that isn't useful - but I think its not the full picture. Suppose I write some stuff that only gives a speed increase to those with fast systems but is no help to others. I won't make real games run much faster but I could look great on a single machine benchmark.

[Licho]

Then you would ask people to run same batch of tests and submit results.

Once there are some reasonable benchmarks I can add website to collect tests from various people..

You could then check for how many people feature X is actual increase for example.

[Licho]

Some performance results using old jK's CA benchmark mutator.

Notice the massive difference between 94 and 91 in unit creation, destruction and "war"

https://docs.google.com/spreadsheet/ccc ... sp=sharing

[abma]

doesn't work on linux -> not really useful for engine devs :-/

but good that it exists, others can report issues then.

[Anarchid]

doesn't work on linux -> not really useful for engine devs :-/

Supposedly it should work with mono.

[Licho]

Yeah you can test this version with mono

http://zero-k.info/temp/bench.zip

[abma]

doesn't work :-/

$ mono RunBenchmarker.exe

Unhandled Exception: System.TypeLoadException: A type load exception has occurred.
at (wrapper remoting-invoke-with-check) Benchmarker.MainForm:.ctor (PlasmaShared.SpringPaths,PlasmaShared.SpringScanner,PlasmaDownloader.PlasmaDownloader)
at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: A type load exception has occurred.
at (wrapper remoting-invoke-with-check) Benchmarker.MainForm:.ctor (PlasmaShared.SpringPaths,PlasmaShared.SpringScanner,PlasmaDownloader.PlasmaDownloader)
at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0

[Licho]

libmono-system-runtime you need

[abma]

its "apt-get install libmono-system-runtime4.0-cil"

seems to run at least, but all controls are disabled / empty?

edit:

Code: Select all

svn checkout https://zero-k.googlecode.com/svn/trunk/other/Benchmarker/Benchmarks

made it non-emtpy. how to "add" an engine?

[Licho]

Yeah content is not packed with it, its in svn http://zero-k.googlecode.com/svn/trunk/ ... enchmarks/

if you checkout this to .spring/Benchmarks it should show something in lists (after "refresh")

[abma]

ok, when typing "94.1" to Engine and click a add it starts downloading "94.1" but got stuck at ~90%. (changes from 92 to 93% but goes back to 92% then).

~/.spring/engine/94.1/spring

exists and works...

when i hit start (ignoring the ongoing download) i get:

Code: Select all

Unhandled Exception: System.ComponentModel.Win32Exception: ApplicationName='/home/abma/.spring/engine/91.0/spring', CommandLine='--config "/tmp/Benchmarks/Configs/ZKL default/springsettings.cfg" "/tmp/tmp49128b30.tmp"', CurrentDirectory='/home/abma/.spring/engine/91.0'
  at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Start ()
  at Benchmarker.SpringRun.Start (PlasmaShared.SpringPaths paths, Benchmarker.TestCase test, Benchmarker.Benchmark benchmark) [0x00000] in <filename unknown>:0 
  at Benchmarker.Batch.RunTests (PlasmaShared.SpringPaths paths) [0x00000] in <filename unknown>:0 
  at Benchmarker.MainForm.<btnStart_Click>b__a () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.ComponentModel.Win32Exception: ApplicationName='/home/abma/.spring/engine/91.0/spring', CommandLine='--config "/tmp/Benchmarks/Configs/ZKL default/springsettings.cfg" "/tmp/tmp49128b30.tmp"', CurrentDirectory='/home/abma/.spring/engine/91.0'
  at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Start ()
  at Benchmarker.SpringRun.Start (PlasmaShared.SpringPaths paths, Benchmarker.TestCase test, Benchmarker.Benchmark benchmark) [0x00000] in <filename unknown>:0 
  at Benchmarker.Batch.RunTests (PlasmaShared.SpringPaths paths) [0x00000] in <filename unknown>:0 
  at Benchmarker.MainForm.<btnStart_Click>b__a () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
_wapi_handle_unref_full: Attempting to unref unused handle 0x12

Unhandled Exception: System.Net.Sockets.SocketException: The descriptor is not a socket
  at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue, Boolean allowIdle) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr handle, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUI.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr hWnd, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000] in <filename unknown>:0 
  at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.Net.Sockets.SocketException: The descriptor is not a socket
  at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue, Boolean allowIdle) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUIX11.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr handle, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.XplatUI.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr hWnd, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0 
  at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000] in <filename unknown>:0 
  at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0 
abma@sonne:/tmp$ mono RunBenchmarker.exe 
System.SystemException: Error running dmcs: Cannot find the specified file
  at Mono.CSharp.CSharpCodeCompiler.CompileFromFileBatch (System.CodeDom.Compiler.CompilerParameters options, System.String[] fileNames) [0x00000] in <filename unknown>:0 
  at Mono.CSharp.CSharpCodeCompiler.CompileAssemblyFromFileBatch (System.CodeDom.Compiler.CompilerParameters options, System.String[] fileNames) [0x00000] in <filename unknown>:0 
  at Mono.CSharp.CSharpCodeCompiler.CompileAssemblyFromFile (System.CodeDom.Compiler.CompilerParameters options, System.String fileName) [0x00000] in <filename unknown>:0 
  at System.Xml.Serialization.XmlSerializer.GenerateSerializers (System.Xml.Serialization.GenerationBatch batch, System.CodeDom.Compiler.CompilerParameters cp) [0x00000] in <filename unknown>:0 
  at System.Xml.Serialization.XmlSerializer.RunSerializerGeneration (System.Object obj) [0x00000] in <filename unknown>:0 

note: running
/home/abma/.spring/engine/94.1/spring --config "/tmp/Benchmarks/Configs/ZKL default/springsettings.cfg" "/tmp/tmp49128b30.tmp"

worked! :) so pretty close from working / running...

[Licho]

Well regarding 94.1 - 90-100% it is decompressing 7z..
unfortunatelly it takes shitton of time with the lib im using (why oh why cant you use standard zip, decompressing that is instant....)

It seems you had test case with 91.0 because it tried to start that.

[abma]

ok, letting it run ~20 minutes it seems to finish.

but still why does it flip back? it prints something like:

90% 91% 90% 91% 90%... 91% 92% 91% 92% 91%...

imo SharpCompress itself can't be so slow, it looks more like some implementation mistake. for example something like the status is polled with 100% cpu load and there is no cpu left for extracting... but no clue.

after it finished i still get an exception when clicking start:

Unhandled Exception: System.ComponentModel.Win32Exception: ApplicationName='/home/abma/.spring/engine/94.1/spring', CommandLine='--config "/tmp/Benchmarks/Configs/ZKL default/springsettings.cfg" "/tmp/tmp23c7b524.tmp"', CurrentDirectory='/home/abma/.spring/engine/94.1'
at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0
at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0
at System.Diagnostics.Process.Start () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Start ()
at Benchmarker.SpringRun.Start (PlasmaShared.SpringPaths paths, Benchmarker.TestCase test, Benchmarker.Benchmark benchmark) [0x00000] in <filename unknown>:0
at Benchmarker.Batch.RunTests (PlasmaShared.SpringPaths paths) [0x00000] in <filename unknown>:0
at Benchmarker.MainForm.<btnStart_Click>b__a () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ComponentModel.Win32Exception: ApplicationName='/home/abma/.spring/engine/94.1/spring', CommandLine='--config "/tmp/Benchmarks/Configs/ZKL default/springsettings.cfg" "/tmp/tmp23c7b524.tmp"', CurrentDirectory='/home/abma/.spring/engine/94.1'
at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0
at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0
at System.Diagnostics.Process.Start () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Start ()
at Benchmarker.SpringRun.Start (PlasmaShared.SpringPaths paths, Benchmarker.TestCase test, Benchmarker.Benchmark benchmark) [0x00000] in <filename unknown>:0
at Benchmarker.Batch.RunTests (PlasmaShared.SpringPaths paths) [0x00000] in <filename unknown>:0
at Benchmarker.MainForm.<btnStart_Click>b__a () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0

Unhandled Exception: System.Net.Sockets.SocketException: The descriptor is not a socket
at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue, Boolean allowIdle) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr handle, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUI.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr hWnd, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000] in <filename unknown>:0
at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Net.Sockets.SocketException: The descriptor is not a socket
at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue, Boolean allowIdle) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.UpdateMessageQueue (System.Windows.Forms.XEventQueue queue) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUIX11.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr handle, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0
at System.Windows.Forms.XplatUI.GetMessage (System.Object queue_id, System.Windows.Forms.MSG& msg, IntPtr hWnd, Int32 wFilterMin, Int32 wFilterMax) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.RunLoop (Boolean Modal, System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.Run (System.Windows.Forms.ApplicationContext context) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Application.Run (System.Windows.Forms.Form mainForm) [0x00000] in <filename unknown>:0
at BenchmarkerRun.Program.Main () [0x00000] in <filename unknown>:0

also why does it redownload the spring engine when the folder and executable already exists?

[Licho]

Is it possible that it was missing x file flag?

And regarding 7z it really is that slow.. I can make it a bit faster by streaming from memory but thats it


I also noticed that sharpcompress throws exception when near end and after exception it deletes entire engine folder...

so meh it just cannot work with 7z ..

[Licho]

I added support for --benchmark values too...

Does anybody why effFps differes from measured FPS (by a widget) sometimes by a lot (50% in some tests).



EDIT:answered to me by jK

[abma]

Is it possible that it was missing x file flag?

not sure, initially it was set, don't know if it was set after it was extracted by the benchmark program. i can't believe that it takes 20 minutes to extract a 12MB file, where in comparison it took like 10 seconds with native 7z. thats 1200% slower.

still the application doesn't check if the engine already exists, it seems to always extract it, that could be at least changed to make it useable.

[Licho]

Fixed that, it was looking for spring.exe

please redowload.. if it fails again i will try to get my virtualbox running again to test myself.