Looking to start a C# AI - Approaches? - Page 2

Looking to start a C# AI - Approaches?

Here is where ideas can be collected for the skirmish AI in development

Moderators: hoijui, Moderators

ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

gajop wrote: Those files are rather large to read completaly - do you have an xsd perhaps? Anyhow, from what I understand, you seem to be parsing C interfaces and you're creating simple intermediate XML files, your goal being to later use those XML files in all languages, as a point of reference, without having to parse it again specifically.
Exactly, I needed to have full spec of structures, enumerations, functions, parameters, etc. in order to figure out how to marshal those from managed environment. As you will see in my code, special access modifiers are used to indicate whether the variable is intended to go in/out or both directions.

Besides that, the approach you will find in the attached archive cannot be used with Mono according to information I have.
gajop wrote: I guess I can see the reason for it, but it won't make new interfaces trivial though, much of the awk scripts is the actual production of new interfaces/bridges rather than the C parsing, thus I don't think that the couple of lines saved would be worth it.
Ok, you may be right. It's probably better to create two different scripts for Mono and for Native case
gajop wrote: Btw, can you show your code that you used for the CSAI - the actual C#<->C bridge (both the C# and C/C++ files, even if they're generated or incompatible with the current version), I'm curious to see if I could create some new awk scripts to generate a functional interface, and would like to have a point of reference.
Sure go ahead. The most interesting/challenging part is in archive. I will upload complete sources soon.
Attachments
CSProxy.zip
(53.07 KiB) Downloaded 58 times
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Looking to start a C# AI - Approaches?

Post by gajop »

ivand wrote:
gajop wrote: Those files are rather large to read completaly - do you have an xsd perhaps? Anyhow, from what I understand, you seem to be parsing C interfaces and you're creating simple intermediate XML files, your goal being to later use those XML files in all languages, as a point of reference, without having to parse it again specifically.
Exactly, I needed to have full spec of structures, enumerations, functions, parameters, etc. in order to figure out how to marshal those from managed environment. As you will see in my code, special access modifiers are used to indicate whether the variable is intended to go in/out or both directions.

Besides that, the approach you will find in the attached archive cannot be used with Mono according to information I have.
gajop wrote: I guess I can see the reason for it, but it won't make new interfaces trivial though, much of the awk scripts is the actual production of new interfaces/bridges rather than the C parsing, thus I don't think that the couple of lines saved would be worth it.
Ok, you may be right. It's probably better to create two different scripts for Mono and for Native case
gajop wrote: Btw, can you show your code that you used for the CSAI - the actual C#<->C bridge (both the C# and C/C++ files, even if they're generated or incompatible with the current version), I'm curious to see if I could create some new awk scripts to generate a functional interface, and would like to have a point of reference.
Sure go ahead. The most interesting/challenging part is in archive. I will upload complete sources soon.
OK, great, while I can't promise any results, I will look into creating a C# interface/wrapper similar to the Java one that we currently have. While I can't promise any results as I'm still busy with the studies, I do look forward to work on it during the weekends at least.
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

Thanks a lot, gajop! Sounds promising!
Attachments
NestedLexRules_.txt
Here you can find RegEx schemes I used to dissect the AI c-headers. Today, I checked the results, and they seem to be consistent, although I clearly remember that something was wrong a year ago. Anyway you might find them useful. It also contains XML scheme in some weird format.
(1.29 KiB) Downloaded 136 times
CSLoaderFull.zip
This file is a standard implementation of AI interface lib
(6.52 KiB) Downloaded 49 times
CSProxyFull.zip
This file is a proxy between c-based spring API and corresponding c#-based implementation. Once compiled with MSVC in "C++ interop" mode (consisting both managed and unmanaged code) gives a c# interface assembly. Once this assembly is referred by c#-only AI Skirmish library it exposes all necessary data types, classes, functions and interfaces automatically.
(71.84 KiB) Downloaded 46 times
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Looking to start a C# AI - Approaches?

Post by gajop »

Ok, so I did some research/work on this thing and I think the work is a 3-part thing:
1) Generate a CSharp interface from a set of .h files (done)
2) Create the way of loading a CSharp AI (you have already created a proxy and i'll be looking into using it from tomorrow on)
3) Generate a CSharp OO wrapper using the generated interface

Now, I'm currently thinking it would be best if I used whatever is available in mono (compiles, method of bridging between C#<->C, etc), as this is an open source, cross platform project.
This means that I will also be using msc (mono compiler) to compile whatever C# code I end up generating.
However, from what I managed to find out, it seems that there should be no problem running "regular" Microsoft compiled C# programs, as well as the ones compiled with mono, as long as no system-specific code is written. It should also run on both linux and windows (I'm currently developing on linux but I will test it on windows when I'm done).

If there's anything you feel should be done differently feel free to suggest.
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

Hey Gajop!

Excellent results. Please find my stub implementation of CSharpAI. It's based on native implementation of AICallback: ICSAICallback (the one you did I guess). And therefore it's not very convenient to use when a query to engine in required. Anyway attached file might help.

Regarding the Mono. I've been checking and prototyping some stuff from mono arsenal. Still I haven't found the easy and safe way to call Skirmish AI methods from assembly. Of course I can call these methods simply by specifying their names, however this won't give any indication if ICSAI or ICSAICallback interfaces have changed. In MSVC case the interface-to-implementation integrity is checked in run-time.

Another problem with Mono is calling the native c callback by itself. Mono documentation (http://www.mono-project.com/Embedding_Mono) recommends to solve this issue with two approaches (see "Exposing C code to the CIL universe" section). As you see both approaches consist of c# and c parts of the code. I think it was too obvious for documentation maintainer, that he/she didn't mention how to link these two very different object files together (after compilation of respectively c# and c files) in one .dll/.so file. For now I've failed to found the way which works for MSVC environment, although I think I saw one for gcc. Finding the way for MSVC is crucial as their c# debugger is really a wonder.

Anyway I'm continuing digging Mono docs, mail-lists, etc, to find the optimal way.
Attachments
Class1.zip
This is the almost stub implementation of ICSAI interface.
(1014 Bytes) Downloaded 50 times
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Looking to start a C# AI - Approaches?

Post by hoijui »

I made some CMake macros/functions that generate hashes for the C AI Interface headers. You might write them to files which you cna include, then call a method with for-ever-fixed name and no params, which returns the C# parts compiled-in version fo that hash, and compare it with the hash that is compiled into the C part.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Looking to start a C# AI - Approaches?

Post by gajop »

I admit I haven't read the entire thing, but this is one of the articles that may be useful in my endevour. http://www.mono-project.com/Interop_wit ... _Libraries

I'll post further information when I start working on it.
User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Looking to start a C# AI - Approaches?

Post by hughperkins »

I wrote CSAI. Now I use the Java interface.

Personally I think C# and Java are pretty similar, though C# does have a couple of advantages, but I can live without them. The advantages I found in C# are:

- operator overloading. I hate having to write vectorone.add(vectortwo).multiply(3).
- generics. yes, java has generics. but type erasure really sucks, though they sort of work for 80% of things, and you can use annotations to work around another 18% of cases

I used to think that not having events/delegates would be a limitation, but you can just use adapter classes, and personally I found adapter classes to be cleaner than delegates.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Looking to start a C# AI - Approaches?

Post by hoijui »

hughperkins wrote:operator overloading. I hate having to write vectorone.add(vectortwo).multiply(3)
Maybe some of the JVM supported languages allow to do that too? i do not know, and even if it is the case, it would of course still not be the same, cause you have to use an other language to achieve it.
hughperkins wrote:I used to think that not having events/delegates would be a limitation, but you can just use adapter classes, and personally I found adapter classes to be cleaner than delegates.
There seem to be very few people that see it that way, but i totally agree with you.
i also agree in general, that a C# interface is not worth the trouble, but as logn as i don't have to do it, i don't care of course. :D
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Looking to start a C# AI - Approaches?

Post by gajop »

I understand that there's a lot of similarity between the languages, but I think it wouldn't be bad to have Python and C# support just to cover the major languages.
I also think C# will have a slightly better interface due to having default parameters (no need to pass the "options" and "timeout" parameter to unit commands).
User avatar
hughperkins
AI Developer
Posts: 836
Joined: 17 Oct 2006, 04:14

Re: Looking to start a C# AI - Approaches?

Post by hughperkins »

Default parameters... you can create wrapper functions to achieve the same affect.
operator overloading. Maybe some of the JVM supported languages allow to do that too?
Scala has operator-overloading, and looks quite interesting.

It doesn't solve the generics issue, since the generics type erasure is a jvm-issue.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Looking to start a C# AI - Approaches?

Post by Licho »

Imo best recent advantages of C# are LINQ/PLINQ and similar .. it just makes coding complex stuff much easier and makes paralelization and db access simple and secure.
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

BTW, during this weekend I've started creating the Mono Interface. Currently I have come to passing all the messages from engine to SkirmishLib, though over way around (i.e. callback) is not done yet.

Unfortunately it turned out that when Mono is used to activate CLR assembly, it does it in the way which does not allow MSVC to perform debug inside the assembly. In other words, when MonoINterface is used, I cannot set MSVC debugger to trace the code inside the skirmish lib. This is absolutely disappointing, as my primary motivation to use C# was impressive MSVC debug abilities. Now I'm thinking how to make the interface conditional: to introduce choosing whether CLR or Mono mechanism should be invoked. It'd be difficult to accomplish of course without writing two separate Interfaces.

If anyone has good experience with topics like this http://msdn.microsoft.com/en-US/library ... 80%29.aspx please let me know.


Thanks!
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

Hi!

I've dropped the idea to write an interface towards Mono or native dotNet. Instead I was feeling that RPC approach should be utilized in order to decouple Spring and Skirmish binary, allowing to write Skirmish in any language which supports the RPC.

The RPC mechanism chosen is Facebook-born Thrift protocol: http://en.wikipedia.org/wiki/Apache_Thrift
It's very similar to another one from Google namely Protocol Buffers, however it does implement RPC mechanism.

The interface definition language looks as following

Code: Select all

namespace * springrts.callback

service CallbackFrom
{
	i32 Engine_handleCommand(1: i32 skirmishAIId, 2: i32 toId, 3:i32 commandId, 4:i32 commandTopic, 5:binary commandData);
	string Engine_Version_getMajor(1: i32 skirmishAIId);
}

Code: Select all

namespace * springrts.direct

service DirectTo
{
	i32 init(1:i32 skirmishAIId, 2:string callback_url);
	i32 release(1:i32 skirmishAIId);
	i32 handleEvent(1:i32 skirmishAIId, 2:i32 topicId, 3:binary data);
}

By compiling the IDL files, one gets code in his/her favorite language (Delphi, Sharp, c++, OCaml, HTML, Flash, etc...). It generates portable code for Win/*nix.

I have a working prototype running c# null ai, while interface on the spring side is written in c++.

Up to now I encountered only one major problem, it's a philosophy question of how to start a skirmishAI. Current approach is to threat AI as combination of launcher and arguments. For instance for EXE files only the launcher is required, and for perl for instance both are very useful (<path_to_perl>/perl <path_to_ai>/skirmishAI.pl). Spring or better say interface_lib launches that as a separate process.

Unfortunately, this method doesn't give precise control of the launched process as it could be terminated or throw exception without spring's awareness. Another problem with this one is that again I cannot debug the skirmish code (the debugger doesn't hook on). Now I'm thinking to change it a bit to avoid starting/terminating the skirmish process by spring and just leave it as a separate process to be started by user independently.

Another, I hope small issue, is that every data which goes across the channel has to be copied. TCP/IP although done thru loopback takes its performance toll too.

Please let me know if someone is interested in getting more information.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: Looking to start a C# AI - Approaches?

Post by hoijui »

wow, NICE! :D
good luck wiht that one, i like this approach. i am ready to answer questions if i might be of help, though it seems you do not need that. i can't dedicate a mentionable ammount of time to the project though.
ride on!
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Looking to start a C# AI - Approaches?

Post by gajop »

I have to ask for some more time before I can get back to this one. Currently working on a spring in-game mission editor and that's taking pretty much all of my free time.

I don't know much about Apache Thrift, but RPC sounds interesting. The data seems to be able to be sent both ways and it should be synced, i.e those RPC calls are blocking, right?

Btw, you seem to mention that spring wouldn't be affected that skirmish AI got terminated. That's actually not a bad thing at all, AI crashes/slowdowns shouldn't interfering with the engine. This may be a way to prevent those (sure, java/c# with proper error handling does that as well...)

This architecture you're proposing (separated AI/game) reminds me of Michael Buro's ORTS http://skatgame.net/mburo/orts/.

Regarding skirmish AI crash detection, wouldn't it be possible to do that with checks on the method return values? Or simply request each AI to send an "i'm alive" package each frame/couple of frames.
User avatar
danil_kalina
Posts: 505
Joined: 08 Feb 2010, 22:21

Re: Looking to start a C# AI - Approaches?

Post by danil_kalina »

We are using Protocol Buffer for transmitting data through TCP-IP in our job projects.
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

hoijui wrote:wow, NICE! :D
good luck wiht that one, i like this approach. i am ready to answer questions if i might be of help, though it seems you do not need that. i can't dedicate a mentionable ammount of time to the project though.
ride on!
As always, I'd be very grateful if someone could take awk part as it is required in this case is well (although in significantly less occasions).

For instance class like following should be created as actual implementation for callback (this is the only part of code that should be created by a coder)

Code: Select all

class CallbackFromHandler : virtual public CallbackFromIf
{
  int32_t Engine_handleCommand(const int32_t skirmishAIId, const int32_t toId, const int32_t commandId, const int32_t commandTopic, const std::string& commandData) {
    // Your implementation goes here
    printf("Engine_handleCommand\n");
  }

  void Engine_Version_getMajor(std::string& _return, const int32_t skirmishAIId) {
    // Your implementation goes here
    printf("Engine_Version_getMajor\n");
  }

};
So instead of "// Your implementation goes here" there should be a corresponding callback-> call.
gajop wrote:I don't know much about Apache Thrift, but RPC sounds interesting. The data seems to be able to be sent both ways and it should be synced, i.e those RPC calls are blocking, right?
Exactly call to external lib and vise versa blocks execution until code returns.
gajop wrote:Btw, you seem to mention that spring wouldn't be affected that skirmish AI got terminated. That's actually not a bad thing at all, AI crashes/slowdowns shouldn't interfering with the engine. This may be a way to prevent those (sure, java/c# with proper error handling does that as well...)
Good point. I'm working with Thrift only few days, so I'm kind of noob to this lib. Anyway, the way it works now is that if Skirmish gets crashed it raises lots of Exceptions from the lib. I don't know what degree of control Thrift's coder has to handle those Exceptions, how well it's designed to reestablish the connection and restore the AI status...
gajop wrote:This architecture you're proposing (separated AI/game) reminds me of Michael Buro's ORTS http://skatgame.net/mburo/orts/.
Yes, the approach is probably the same, however few years back it was able to send out only the basic commands, no pathfinding, etc...
gajop wrote: Regarding skirmish AI crash detection, wouldn't it be possible to do that with checks on the method return values? Or simply request each AI to send an "i'm alive" package each frame/couple of frames.
I was thinking about ping command, however now I don't see it bringing a big value: either remote call returns or Exception is thrown. But the idea to do some actions every N frames is actually very useful as in case of crash, when restart is required it's quite slow to check if connection has again become available every frame, rather this should be done once a second or smth.
danil_kalina wrote:We are using Protocol Buffer for transmitting data through TCP-IP in our job projects.
ProtoBufs are actually execellent lib, very mature and very-very well documented. Thrift dramatically loses in these criteria. The reason I still choose Thrift is that, it provided in-house RPC for many languages, while in ProtoBuf world RPC implementations are made by 3rd parties and no single solution covers majority of languages.
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

Yea, another idea is to cover not ordinal SAIInterfaceCallback, rather try to group objects as it's done by c++ wrapper. The rationale behind is that one call which fetches big structure is less time-consuming than hundred of calls to fetch every single field in the structure (imagine Unitdef example). And yes, Thrift supports marshaling structures thru IDL!
ivand
Posts: 310
Joined: 27 Jun 2007, 17:05

Re: Looking to start a C# AI - Approaches?

Post by ivand »

Gentlemen, I need your advice.

The question is about multithreaded environment which I tried to avoid for Interface Library, but didn't succeed in the end.

First what I have done so far. Roughly:

Code: Select all

<InterfaceLib->handleEvent> --> TCPClient1 ---> TCPServer1--> <AILib-->handleEvent_Handler>
The result of handleEvent_Handle execution goes other way around.

But I have problem with callback. Callback is implemented as a server on InterfaceLib side and it strictly blocks the thread which it runs at.

Code: Select all

<AILib-->some_callback> --> TCPClient2 --> TCPServer2 --> <InterfaceLib_CallbackServer>
In other words, "default" thread, which is involved in AI activity, can either be TCPClient or TCPServer, but not both.

Not a big deal. I've created a separate thread for CallbackServer and run it in the background. But one thing I cannot now decide is that if synchronization between "default" and "Callback_server" thread is required or they are synchronized "by design". I think they are in sync, please prove me I'm wrong. If I'm wrong, please suggest what part of code should be synchronized!

See the generalized flow diagram (I put "D" in parenthesis for default thread and "S" for server)

Code: Select all

1: (S) Callback_server(TCPServer2) serving all callback functionality started
2: (D) <Springrts> -->AILib -->TCPClient1--> TCPServer1 --> AILib->handleEvent_Handler
3: inside AILib-->handleEvent_Handler the callback is requested
3a: AILib --> TCPClient1 --> TCPServer2(S) --> <TCPServer2 calls callback>(S) --> <Springrts>.
Another priority2 idea is to move TCPClient1 to separate thread as well as when the AILib Server (TCPServer1) is not started each TCPClient1-->connect() try stops the "default" thread for couple of seconds, even if the call is done once in 10 seconds, still game will be seen as "lagging". I appreciate if someone could draw a high-level picture of how to organize it in effective way.

Thanks in advance!
Post Reply

Return to “AI”