Interface Redesign

[AF]

So I give you the current itnerface, BEHOLD!


(click to enlarge)

And here is what I propose:



This should outline a basic C++ Interface, highly generic, and easy to create bindings for. The matter of creating a C API is purely negotiating the communication between CAIObject instances across the library boundary. This can be done by changing the IAIObject class that the AIs inherit from into a basic wrapper class around which a C API can be built.

For backwards compatibility, the existing interface can be hoisted up as a wrapper using the old code to implement its functionality. As the new interface is built we can change the implementation of the compatibility layer and eventually remove the old interface completely. This way the process can be done gradually over a long period of time should there be periods where no developers are able to contribute to the effort.

While Ive started this thread I must stress that I do not have the time or means to do this single handedly, nor do I have the resources to spearhead as a leader by example. I will help out but I cant push this forward implementation-wise as I have my lobby project and other projects to think about.

Having said that, I've kept in mind that what I've proposed could be done piece by piece slowly over a very long period, possibly multiple spring releases. The advantages are that we could use a lot of new languages that compile into native code, and adding new language bindings would be far easier, and we could use any compiler we wanted, and there would only be one interface in the end to maintain and a set standard method of updating and adding to it.

[Acidd_UK]

It seems like a good idea, but what about legacy AI code? It all dies, right?

[jcnossen]

It seems like a good idea, but what about legacy AI code? It all dies, right?

The problem right now is that legacy AI code is already dying/bitrotting because the engine moves ahead but the AIs can't.
Also, the principles stay the same (IE: reading properties from objects/engine and giving commands), its just a different interface.

The compatibility layer from AF sounds like a nice idea, however it will be a LOT of work since current AI's read about every struct that is used by UnitDef/WeaponDef/FeatureDef. Exposing all that through a layer is not fun. I think a better way is just to invest that time into upgrading the major active AI's (AAI, NTAI, KAIK, ??), it might be even less work.

AF: About the interface, I guess you have more right to speak than me, but why go with another C++ interface + a C layer? It's only more work, and in the end you will use the C layer anyway since you don't want to build on mingw.
Also, a clean C api can be just as nice as a C++ API.

[AF]

As I said the existing AI Interface would remain, however its implementation would shift from the current implementation towards simply wrapping around the new interface. Eventually it would not be an AI interface at all but a legacy compatibility layer around the new interface.

That way AIs will still work, and AI developers can incrementally shift from one interface to the other by undoing parts of the compatibility layer until they can do away with it all together.

But indeed as jelmer says it would be less work in some cases to upgrade the existing AIs, thus side stepping parts of the compatibility effort entirely.

One such step I believe would be best is to port the event system from the old interface to the new interface. This should be one of the easiest bits to do and it could be done without bothering with the compatibility layer as it would only affect a the IGlobalAI class on the AI end, whereas stuff like UnitDef structures could be spread across hundreds of lines in 20-30 files.

[Tobi]

AF: About the interface, I guess you have more right to speak than me, but why go with another C++ interface + a C layer? It's only more work, and in the end you will use the C layer anyway since you don't want to build on mingw.
Also, a clean C api can be just as nice as a C++ API.

If you have C++ on top of C interface you have best of both worlds. C's ABI compatibility, ie. AIs can be compiled with MSVC/borland/gcc/you name it, and you still have easy to ready IGlobalAI IGlobalAICallback etc, with the only cost of compiling some proxy classes into the AI.

I don't think a C interface can be as nice as a good C++ interface at all, it'll always feel clumsy and hackish to work with IMO. Of course an AI wouldn't have to use C++ interface though, if someone thinks C>C++ then he could even make AI entirely in C, or write his own proxy classes the way he likes it.

EDIT: a same trick is actually used in KDE desktop environment, there are some objects that seem like real objects in code but if you look at source they are just proxy classes that do a sort of remote procedure calls to e.g. kwin window system. Similar thing is also used with webservices in .NET/Java.

visually, it's C++ (Spring side) <-> C (Spring side) <-> (dll boundary) <-> C (AI side) <-> C++ (AI side)

Note also similarity with the typical addin design, where you have:
host app <-> host app-interface adapter <-> strict interface <-> addin-interface adapter <-> addin

(Implying you can in principle maintain back- and forward compatibility as much as you want, while still being able to totally refactor either side, because you can just write new adapters. Note that I'm not saying actually trying to maintain back- and forward compatibility is a necessarily a good thing.)

[hoijui]

..ahhh before this thread, i did not really get the pro of the C interface... but now...
so when the AI interface is purely C, spring can be compiled with mingw e.g., and the AI with gcc or msvc, and they still could be used together?
yeah thats good :D
hmm...
what about the this, as first step:
introducing interface classes, eg. IUnitDef, for all structs and classes that get used by AIs. And those interfaces only present data that is useful for AIs.
then make the current interface only use those, and then start the real work on the new interface. these interface classes/structs could then be used for automatically generating some of the code for the C interface, and they could also be used on the new C++ interface, with C++ implementations wrapping the C interface of the AI, which wrappes the C interface (in the AI side). so far.. i still have lots of free time, and i currently have quite a good idea about what parts are used by the AI and what should be wrapped with interface classes and such.
when changing something in UnitDef eg, it ould only have to be added to IUnitDef if its usefull for AIs, and if this is done, it could easyly be checked if the C AI api supports all things from UnitDef that are relevant for AIs.
As JAI is kind of finnished, i could do some work (i also have some practice with regex replaces now, to easy up automatic conversion of interfaces.

it would like this:
C++ (Spring side) <-> C++ I<...> classes only! (Spring side) <-> C (Spring side) <-> (dll boundary) <-> C (AI side) <-> C++ wrappers (AI side) <-> C++ I<...> classes only! (AI side)

now thinking about it....
it has the pro that we could compile a C++ AI, or even a wrapper interface, lets say JAI, wrapping C++, against the spring side C++ or the AI side one, which would be equal. compiling against the spring side woul dpossibly make it faster, while compiling against the AI side C++ interface would make it more compatible. (though the speed gain is small, hm?)

[AF]

Could you break up your paragraphs a bit more and put new lines in-between them so its clearer? And I recommend in future that we use diagrams rather than a<->b<->c<->d as its quickly escalating into spaghetti hoops.

Regardless of IUnitDef etc I still think the entire interface should be passed through overloads of HandleMessage() in the IAIObject class. Sure you could build IUnitDef etc as an implementation detail on the spring side or the AI side, but lets not keep it as a part of the interface itself.

I think the basic prototype of the HandleMessage method should go along the following lines:

Code: Select all

int HandleMessage(int myteamID, int messageID, const char* messageTopic, ...);

myteamID identifies the AI sending the message aka where the message is coming from, messageID is there should we add asynchronous messages to the interface so we can identify a reply at a later date. messageTopic is a string identifier as suggested by jelmer for a command or attribute, lets say "unitdef:canmove", and the ... is any arbitrary parameters needed in the overloads.

Obviously the C++ wrapper CAIObject class would not have the myteamID parameter when sending messages to the engine so that spoofing of commands is not directly possible. I suggest the spring engine itself have a team ID of -1, this way it is possible that at some point in the future we could allow inter-AI communication using this method.

I would recommend that the asynchronous methods be used for unitdefinitions, feature definitions and the various map arrays, with the intention that these values are loaded at the beginning all at once, or perhaps that the CAIObject implements itself AI side as a seperate thread, this way we can continue with lazy loading, and the AI thread can remain blocked while it waits for the interface/engine thread to update the necessary data.

We could even change it so that it only loads parameters as they're needed, so say I needed a unitdef for a peewee, I could get one and it wouldn't load until I asked for the canmove, at which point the rest of the unitdef could then be loaded while the AI thread plods along. The engine could then send an event to the AI signifying that a unitdef changed, which tbh should have been added ages ago..

This should also give rise to a nice AI sdk that would compile without needing the engine source.

[hoijui]

.. i'll try if i remember, to structure my posts a bit more

When would my IUnitDef be part of the AI interface, and when not?
You mean you do not want an extra file/class for everythign that gets touched by the AI on the engine side? if so... well i think it would make it more simple, and nothign else. When a new value gets added to UnitDef eg, one has to decide wheter its interesting for the AI (field or method gets into IUnitDef) or not (gets into UnitDef). when we use the same IUnitDef on the C++ AI side, we instantly can use it (though it may not work, but we at least see that then).

If we will not use this concept, and we add a field to UnitDef, then the one adding it has to add it to the C++ AI wrapper interface aswell, and tlel it to the other wrap maintainers (eg to me, for JAI) that there is a new field. and if he forgets to change the C AI api, the field will be forgotten and not available to AIs.

I thingk this concept comes at practically no costs (in dev time), has some pros in practice, and nothing negative. Maybe it would be better to call it AIUnitDef, instead of IUnitDef.

Isn't there someone separating graphical from functional code? This is kind of the same, no?

To your ideas now...

Code: Select all

int HandleMessage(int myteamID, int messageID, const char* messageTopic, ...);

I suggest using an int for messageTopic, and additionally have a method:

Code: Select all

int GetMessageTopicHash(const char* messageTopic);

otherwise, hashes have to be generated, or strings compared on every call of your HandleMessage method. And when using an hash directly, we can (if we want) store those hashes on the AI side, and only had to generate them once.

i did not get the idea of the other paragraph. why would we want to load def data lazyly? you mean, when a mod changed a def with LUA eg. after the game started, the AIs would get a message and could reload their def data?

i did not get lots of things you explained.

[AF]

Well it would allow a simpler interface, as well as a more network like message based approach.

Also, when an AI loads a unitdef, the engine does a full load, which causes a slowdown and more memory IIRC. Perhaps this has changed with trepans lua stuff, but the spring engine itself uses lazy loading.

Also I would like to keep the interface as only IAIObject and the C structures and overloaded methods needed to transfer data between those objects. I think your idea on changing it from a string to an int and using a lookup method is a good idea too.

But, I would like it if we kept things as generic as possible and that we maintained a disconnect between the engine implementation and the AI side.

Your point on adding things to unitdef and forgetting to add them to the interface already applies and isn't really avoidable in a C interface, not without invoking a C++ class or structure. And I'm really not keen on having to go through swig and everything for the interface, I'd rather keep the C interface itself extremely small and simple so that there's no need for automated binding tools.

[hoijui]

Hmm...
In JAI, I load all UnitDefs at startup, and save all values into my Java wrapper class JUnitDef. it does not take long and so far i had no moemory problems. Its maybe 1MB or something? i though it wont be a problem.. is that your concern, that it could be too much? I only tested with XTA so far.

It sounds good too, keeping it simple, using HandleMessage() extensively, and keeping the engine and AI side disconnected is good. And.. i am not a real fan of SWIG did not think about using it for that :D

Sorry to talk about that again, but I may explained it wrong before:
With an AIUnitDef, and the C interface, we could have a small shell script that compares the variables in AIUnitDef with the constants in the C AI interface for example. We could copy the AIUnitDef.h file to the folder of the C++ wrapper project and add a AIUnitDef.ccp implementing it using the C AI interface. In my eyes, it is totally disconnected then, but eases up work for the one maintaining the C++ AI interface wrapper, and this guy would also see wehn/if the C AI interface is missing the possibility/the messageID constant for a specific value of UnitDef eg, and could tell it to the guy maintaining the C AI interface.

I Would do the work.. did it for UnitDef already.. its simply sorting all the values into 2 files. And it are nto so many files...
definatly:
* UnitDef
* WeaponDef(Handler)
* FeatureDef

possibly:
* MoveInfo
* DamageArray
* CollisionVolume


Something else:
Would Krogothes Metal Spot Finder be integrated into the interface? (available through HandleMessage())?
I don't care so much, just asking.

And...
Who would do this new interface?
I think i could do it, but if someone else has time...
anyone?

[AF]

krogothes metal finder algorithm is used in KAI/K AAI and NTai, I don't know about RAI, however they are not identical. AAI uses the core algorithm hacked in place of an older algorithm, KAIK integrates the algorithm into a set of spotfinder helper classes, and NTai uses wrapper classes as well as a few internal modifications of my own.

So no, it should be treated as a library for now not an interface component.

And regarding this unitdef stuff, it appears to be an implementation detail not an interface detail.

[hoijui]

was thinking about the AIUnitDef thing again..
and yes.. you are right.. it doesnt have much to do with the interface, and its not important.

[Agon]

Is it possible to extend the interface later?
What about Lua<->Ai?

Who will implement it?

About the algorithms. Maybe we can provide an extra library with a set of important or often used algorithms.

[AF]

I have my own ideas regarding lua<->AI, namely 2 grand theories, however one of them is tailored to NTai, shows most promise but uses the same structures as NTai, and the other is highly generic and would require an extraordinary amount of effort from AI developers to support it.

[hoijui]

..describe them please

[AF]

I'd rather make pretty blog posts first.

[Agon]

I'd rather make pretty blog posts first.

I´m waiting for you blog post. I´m very interested in it. And I hope this ball will roll soon. I know it takes time...

[hoijui]

... am not sure about this, but what about sync with this C interface?
am i right that we still had to use streflop on the AI side, as otherwise we can not guarantee eaual calculations?
i though on this:
compiling NTai once with VC++, and once with g++ with the C AI interface, we could use both builds with the mingw build of spring then. but would they sync?

[aegis]

an AI is like an automatic player only seen on your computer... it isn't globally synchronized... it seems like a normal player to all of the other players...

[hoijui]

hmm.. yeah right
still...
i had the problem with JAI in the beginning, that the JVM changed the FPU settings, and then spring would desync if i did not change the FPU back to spring settings with streflop when returning from a call to Java, as the engine internal stuff, the game itsself, would have been using the wronf settings then. so the AI can casue desyncs. i guess it could be solved through calling streflop (resetting the FPU to spring settings) whenever a functioncall to the AI returns. would this be an acceptable solution? i am doing it with JAI already, it would definitly solve problems and not be such a slow down i think.