Spring is running in SMP

[zerver]

issues left to fix which I have no idea how to:

Code: Select all

rts\Rendering\UnitModels\s3oParser.cpp: In member function 'void CS3OParser::DrawSub(SS3O*)':
rts\Rendering\UnitModels\s3oParser.cpp:302: error: 'GML_FUNCTION_NOT_IMPLEMENTED' was not declared in this scope
rts\Rendering\GL\VertexArray.cpp: In member function 'void CVertexArray::DrawArrays(int, int)':
rts\Rendering\GL\VertexArray.cpp:64: error: 'GML_FUNCTION_NOT_IMPLEMENTED' was not declared in this scope

I fixed these unimplemented functions and changed the enable flag from _GML_ to USE_GML.

However, I cannot get it to run at the moment.
It says something about "Explosion info for XXXXXX not found".
Edit: Never mind, solved itself somehow.

[imbaczek]

http://www.tdragon.net/ contains installer for gcc 4.3.2, which looks like it compiles GML out of the box. Not sure yet if it works (compiling boost now), will keep you updated so you can try debugging deadlocks.

edit: I'm afraid your dirty hacks with atomic_count don't work with boost 1.36. Suggest fixing that by importing whole atomic lock from boost and putting it into e.g. gml namespace, with appropriate stuff changed.

[zerver]

Great, it compiles now.

TLS appears not to be working. This causes the deadlock.

Boost 1.36... well, I'm not surprised they changed the atomic_count.
Some essential features were definitely missing

[imbaczek]

got a small test program to find out whether TLS works?

[zerver]

Have you thought about (imho) most obvious design of running two main loops (in 2 threads), one for rendering and one for sim.

The biggest obstacle to making this reality seems to be LUA. I do not know much about LUA, but it seems to be very much stack based. Both the rendering and sim use the same LUA stack.
My tests with running the sim in parallel with the rendering seem to confirm this. LUA instantly starts to churn out errors on the console and then it crashes after some seconds.

Would it be possible to use two stacks?

Since this design change would mean HUGE performance gains, suggestions are welcome...

[Tobi]

AFAIK every Lua script has it's own lua_handle, associated with it's own stack. (Where from engine POV all gadgets = one script, all widgets = one script, etc., IIRC.)

Of course it could be problematic that Lua widgets can access a lot of simulation data, while they probably wouldn't be executed on sim thread. This should be solvable however by making LUA calls to sim thread safe.

[Kloot]

Yes, you can maintain a separate lua_State per thread (AFAIK, every script
has its own handle / state / stack). But remember that unsynced code can
read synced data too, so all Spring functions sitting behind the synced API
will need guards against concurrent access at least (and synced code can
write unsynced data, same caveat).

[zerver]

Could someone on Linux help me debug the compile error on buildbot?

It seems to think that _syscall3 is a function, but it should be a macro defined in <asm/unistd.h>

Thanks

Code: Select all

#include <linux/types.h>
#include <linux/fs.h>
#include <sys/syscall.h>
#include <asm/ldt.h>
#include <asm/unistd.h>
#include <sys/types.h>

//Define the modify_ldt function that will call this particular syscall.
_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount);

Code: Select all

rts/lib/gml/speedy-tls.cpp:70: error: 'modify_ldt' has not been declared
rts/lib/gml/speedy-tls.cpp:70: error: 'func' has not been declared
rts/lib/gml/speedy-tls.cpp:70: error: 'ptr' has not been declared
rts/lib/gml/speedy-tls.cpp:70: error: 'bytecount' has not been declared
rts/lib/gml/speedy-tls.cpp:70: error: expected constructor, destructor, or type conversion before ';' token
rts/lib/gml/speedy-tls.cpp: In function 'int speedy_tls_get_next_avail_ldt()':
rts/lib/gml/speedy-tls.cpp:161: error: 'modify_ldt' was not declared in this scope
rts/lib/gml/speedy-tls.cpp: In function 'int speedy_tls_get_number_ldt_entries()':
rts/lib/gml/speedy-tls.cpp:179: error: 'modify_ldt' was not declared in this scope
rts/lib/gml/speedy-tls.cpp: In function 'void speedy_tls_onthread_ending(void*)':
rts/lib/gml/speedy-tls.cpp:230: error: aggregate 'modify_ldt_ldt_s freeLDT' has incomplete type and cannot be defined
rts/lib/gml/speedy-tls.cpp:268: error: 'modify_ldt' was not declared in this scope
rts/lib/gml/speedy-tls.cpp: In function 'int speedy_tls_init_foraddr(void*, int)':
rts/lib/gml/speedy-tls.cpp:327: error: aggregate 'modify_ldt_ldt_s newLDT' has incomplete type and cannot be defined
rts/lib/gml/speedy-tls.cpp:363: error: 'modify_ldt' was not declared in this scope

[Kloot]

The _syscall* macros have been obsoleted in newer kernels (they
were removed from the headers in 2.6.18? for those architectures
that provided them in the first place, you are supposed to use the
syscall() wrapper now which goes through libc).

[zerver]

Great, it seems to compile now!

It would be good if someone could test if it runs on linux compiled with gml=yes.

At least it seems to run with GCC/win32 now. The performance gain is not big (5-10%) but still better than nothing.

[clericvash]

Great, it seems to compile now!

It would be good if someone could test if it runs on linux compiled with gml=yes.

At least it seems to run with GCC/win32 now. The performance gain is not big (5-10%) but still better than nothing.

I will gladly take 5-10% over nothing for the next spring version :D

[zerver]

I now have the parallel Sim/Draw *almost working*. If you wanna try it, compile with gml=yes gmlsim=yes.

It currently only works with mods that do not make GL calls from Sim (such as BA).

It can be enabled/disabled ingame with
/multithread (multithreaded DrawUnit etc)
/multithreadsim (parallel Sim/Draw)

The performance gain with both enabled appears to be something like 25% on a quad core which is not bad at all.
Try /give all a couple of times and u will see...

And by the way, the new Dynamic and Bump waters are absolutely gorgeous, thanks a bunch.

[Masure]

Could you provide a windows binary with SMP for 77b5 ?

[imbaczek]

go to #buildserv, do !rebuild tag=spring_0.77b5 profile=gml

[zerver]

go to #buildserv, do !rebuild tag=spring_0.77b5 profile=gml

That profile does not use gmlsim=yes, so it will not have parallel Sim/Draw. Bibim, plz make a new profile...

[Masure]

Once done, we could plan an SMP test event :)

[zerver]

Yes, I'm assuming there will be a 77b6 soon since 77b5 seems to desync. I tried to play Tropical several times yesterday but couple of players would always instadesync.

With 77b6 we would have a good SMP testing opportunity.

[tizbac]

If needed i could help testing on linux

[Masure]

Yes, I'm assuming there will be a 77b6 soon since 77b5 seems to desync. I tried to play Tropical several times yesterday but couple of players would always instadesync.

With 77b6 we would have a good SMP testing opportunity.

Sorry if i'm disappointing but I don't think including SMP in .77 is a good idea.

Spring development suffers from its messy release logic organization. Supposed "bugfix" releases often include new features which doesn't help to make a stable version. Next supposed "feature" release often combines bugfix from the last one and the new features that cause new issues.

I know I'm not in the dev team but I hope the release cycle to be propoer one day.

IMHO, feature freeze approach in a version should be a priority in the dev team mind. I understand it's hard for a dev to wait for his feature to be included in the next version.

[Tobi]

Only works when using separate branch.

Don't know why this wasn't done for b3 b4 and b5, since it seemed to work reasonably well for b2? Or was in the end just trunk tagged?

Either way, on topic, default build won't get SMP enabled by default in either case. If you want to take a b6 as opportunity however to organise some testing with SMP build, then that's fine.