Interface Redesign

[hoijui]

sorry, still no update. i had some practical C++ problems, but i solved them now. aehm.. i am still on it! working.

[hughperkins]

Kindof getting an Osborne effect here: no-one is working on anything AI-related until the soon-to-be-released new interface comes out.

[AF]

I assume once the C API is in a workable condition and in the engine we'll all start working on transferring what code we have to use it as well as building wrappers and refactoring structures to better suit the new paradigm.

What is the progress at the moment on the API itself? It would be nice if we could see progress itself maybe via svn commits?

Also it would be useful if someone could create a prototype AI and lua gadget that exchanged primitive messages so that we can all see exactly how it works. Right now all we have is a handful of obscure methods and a few cryptic posts by trepan that have been lost as new posts are piled on top.

[zenzike]

Unfortunately, real life issues have meant that I've put my side of development on hold for a little while -- I've been AFK for a few weeks, and have to catch up with work commitments for another fortnight. Once that period is over, I'll be able to spend some time in earnest to get this sorted out.

In terms of development of existing AI systems, I suggest you carry on with business as usual -- the interface we've been describing so far hasn't described anything that will permanently break compatibility with the current AIs, and once the new interface is up and running, wrapper code will be released to make sure that the transition is relatively smooth.

[hoijui]

aehm.. we have the wrapper code already, no? in the SVN branch already.
what i am currently working on is modularization of language support, support for static AI properties and AI options and i am slowly porting the groupAI interface as well.
the code is kind of finnished, but it is not running yet (btw, i hate C++. always a pain to work with it). i will try switching to g++ now, as i have an error which i can not get with mingw and gdb; so i just hope i will have more luck with g++ and gdb.
there was no submit since a long while cause.. i started all these changes at ones. and casue it does nto yet compile to a working version, i did not want to submit.
and.. yes i know it was a stupid idea to do all of it at once
if one of you has time and will to work on it, and wants to do that now.. contact me, and i will commit my stuff anyway (it is at least compiling, and its only a branch, so that would be ok i guess).

[hoijui]

after some more days of swearing because of C++ i got a runing version again! debugging under linux is really much better then debugging the mingw build under windows.
i will now clean up the code a bit, and then, the following things have to be adressed:
* finnish the group AI interface port (from C++ to C)
* implement a good way of logging and error handling, which works well with dynamically loaded shared objects over a C interface
* test some
* remove the last bits of Global AI stuff (currently unused stuff, as i renamed globalAI to SkirmishAI, group ai stays the same)
* centralise the AI side C++ interface wrapper code (and separate the legacy one from the event one)
* make all AIs in SVN work with the legacy C++ wrapper (should be VERY little work)

so an svn commit should come soon. i will also explain the changes i did then.

[hoijui]

i forgot these points:
* LUA stuff (ainfo thing)
* making info about AIs easyly available, e.g. for lobbies

[Agon]

So this means the new interface is soon finished (ready for testing)?

[hoijui]

i would hope so
most of the points i have listed are not very urgent, or say: testing can begin before they are done.
i see future events in this order:
1. centralise the AI side C++ interface wrapper code (and separate the legacy one from the event one)
2. here we could already start testing
3. LUA stuff (ainfo thing)
4. making info about AIs easyly available, e.g. for lobbies

some words about what i did:
(language-) AI interface interfaces are all modularised.
currently i have only the C one (which will be used for C AIs, new and legacy C++ AIs), and i will do the Java one later.

current layout is as follows:
<spring_root>/AI/Skirmish/impls/KAIK-0.13.dll
<spring_root>/AI/Skirmish/impls/RAI-0.600.dll
<spring_root>/AI/Skirmish/impls/RAI-0.700.jar
<spring_root>/AI/Skirmish/data/RAI
<spring_root>/AI/Skirmish/data/KAIK
<spring_root>/AI/Interface/impls/C-0.1.dll
<spring_root>/AI/Interface/impls/Java-0.2.dll

the lobbies will have to put something like this into script.txt:

Code: Select all

aikey=C#0.1_KAIK#0.13

but it could also be just:

Code: Select all

aikey=RAI

in which case th engine will figure out which interface and version to use. in this example, RAI would be expanded by the engine to:

Code: Select all

aikey=Java#0.2_RAI#0.700

all valid keys would be:

Code: Select all

aikey=RAI
aikey=RAI#0.700
aikey=Java_RAI
aikey=Java_RAI#0.700
aikey=Java#0.2_RAI
aikey=Java#0.2_RAI#0.700

both. interfaces and AIs export static properties (map<string, string>), eg shortName, version, description, infoURL, ...
in addition to that, they export a set of possible options (comparable to modOptions).
these two things can be querried from an Interface or AI lib. as pointed out earlyer in this topic, we should not always load and querry all libraries, when we want that info, cause then each library had the possiblity to crash the whole application. thats why we wanted to use the LUA files, to store this info. the idea is, to primarly get properties and options from LUA files, but beeing able to actualize or generate the LUa files by requesting that info directly from the libs. so usually, we will rely on the LUA files.
why i did it like this:
this way, as an AI dev:
* i do not have to write anything in LUA -> the AI interface are cleaner (a single language, which one ever)
* i can, but do not have to ship any LUA info files with my AI
* we could have a command line argument to actualize all AI info, eg: spring.exe --actualize-all-ai-info
this would try to load all AI Interface libs and AIs and generate the LUA info files for them. if something goes wrong with one of them, we could output an error for that AI or interface, and go on with the others.
* we coul have an other option: spring.exe --generate-missing-ai-info, which would only generate the missing info

the info files could be stored like this:
<spring_root>/AI/Skirmish/info/C_KAIK-0.13.lua
<spring_root>/AI/Skirmish/info/Java_RAI-0.600.lua
<spring_root>/AI/Interface/info/C-0.1.lua

an interface library has to support skirmish and group AIs. In other words, the Java.dll will support using skirmish and gorup AIs written in Java. that is what the interface looks like at least.

if i have to explain better, tell me.
(more details will come later)

[AF]

map<string,string> ? Isnt that C++ meaning compiler differences etc?

[hoijui]

hehe :D yeah you are right; it actually is a char*[][2].
i just though, saying it is a map<string, string> is more straight forward, and i did not think about that detail at this moment.

[hoijui]

an other thing...
about the AI interface libraries (eg. C-0.1.dll):
most likely, each such library will have to be compiled very specially, for example, Java-xxx.dll will have to be linked against streflop and some jvm stuff, for C# we will need special things as well i guess...
so we should probably integrate only the C interface library in the global built process, as otherwise, everyone that wants to compile spring will need to have all the Java, C# and probably other languages stuff installed as well. if possible, we could optionally allow to compile these libraries when building spring eg with:
-DCOMPILE_CSHARP_AI_INTERFACE=TRUE

[AF]

As far as I can tell the compiler will export a char* and not a char[][2] across the boundary. While this could be engineered to work, I'm sure there's a much better solution than using multidimensional arrays.

[Kloot]

the lobbies will have to put something like this into script.txt:

Code: Select all

aikey=C#0.1_KAIK#0.13

but it could also be just:

Code: Select all

aikey=RAI

in which case th engine will figure out which interface and version to use. in this example, RAI would be expanded by the engine to:

Code: Select all

aikey=Java#0.2_RAI#0.700

all valid keys would be:

Code: Select all

aikey=RAI
aikey=RAI#0.700
aikey=Java_RAI
aikey=Java_RAI#0.700
aikey=Java#0.2_RAI
aikey=Java#0.2_RAI#0.700

Don't. Do. This., it is hackish and error-prone (I might like to use
some hash characters in my AI name one day ).

both. interfaces and AIs export static properties (map<string, string>), eg shortName, version, description, infoURL, ...
in addition to that, they export a set of possible options (comparable to modOptions).
these two things can be querried from an Interface or AI lib.

That makes the Lua files redundant. The idea was to have all the
information pertinent to how an AI should be loaded external to
its implementation so lobbies can access it (through the unitsync
Lua parser) without having to jump through hoops, a lobby having
to query an AI first (whether directly or via the engine) would work
against that.

as pointed out earlyer in this topic, we should not always load and querry all libraries, when we want that info, cause then each library had the possiblity to crash the whole application. thats why we wanted to use the LUA files, to store this info. the idea is, to primarly get properties and options from LUA files, but beeing able to actualize or generate the LUa files by requesting that info directly from the libs. so usually, we will rely on the LUA files.

Define "usually", this just causes more confusion than it prevents. Ex.:
how do you determine when to actualize? When the file doesn't exist?
When it is older than 30 days? In practice you'd always just stick to the
first rule to keep things simple anyway, and from there it's a small step
to assume that if the file is missing it isn't required at all (see below).

this way, as an AI dev:
* i do not have to write anything in LUA -> the AI interface are cleaner (a single language, which one ever)
* i can, but do not have to ship any LUA info files with my AI
* we could have a command line argument to actualize all AI info, eg: spring.exe --actualize-all-ai-info
this would try to load all AI Interface libs and AIs and generate the LUA info files for them. if something goes wrong with one of them, we could output an error for that AI or interface, and go on with the others.
* we coul have an other option: spring.exe --generate-missing-ai-info, which would only generate the missing info

You have already introduced a "<spring_root>/AI/Skirmish/data/" dir
where the Lua info files can be stored, the lobbies / engine can just
assume they're dealing with an ordinary C++ library if the matching
Lua file is not present and no user / lobby should ever have to walk
through a (re-)generation procedure like that. Don't create all sorts
of additional complexity where it is not needed.

Moreover, "if something goes wrong" can again mean "segfault" and
you'd be left with inconsistent (or corrupt) data in the best scenario.

i can, but do not have to ship any LUA info files with my AI

If the AI dev is lazy in that sense, allow the engine to be as well.

[AF]

All AIs should bundle their lua files. I would say this is important and we should not bother at all with auto-generation. This will only lead to AIs exporting what they'd normally export via lua in C API land defeating the whole point and introducing new problems for native bindings in some languages that did not exist before.

+1 to the comment on the lobby itnerface

Code: Select all

ai=name;

If 2 AIs share the same name but are in different languages then this should not be handled and the AI developer should receive an ambiguous name error from the engine and the conflict will be fixed and if the AI developer has any sense it would never occur on the end users machine.

The implementation dll business should be hidden inside the engine as it is an implementation detail. Don't drag it all the way up into API land.

[hoijui]

@Kloot:
about this:

Code: Select all

aikey=Java#0.2_RAI#0.700

what i use here is the Interface and AI shortName and version properties. there is also a name property, where you could use your #
i added comments for that in the code.

AF, if you do not agree with something, please bring alternate solutions.
saying: "i think this is bad and i am sure there is something better" does not help much. do you think i will trust your feeling that it is bad and search for an alternate solution myself, not even knowing why it should be bad? not to say i have as much to do as you, but for this, i have (always) to much to do.

when to actualize(/generate) the LUA files:
when lobby wants info about the available AIs, it will look only at the available LUA info files. if there is an AI lib without a corresponding info file, the info file will obviously NOT be generated at that time, as the lobby would not know that, as it only looks at the LUA files. as said, in my scenario, the AI dev can ship a LUA file with his AI. in this case, the LUA file will NEVER have to be generated, and everything works just as if the AI lib did not export that info at all.

about the generation process:
the generation process would only be needed in special cases, namely:
* the AI shipped without a LUA file
* the user running the generation is the AI dev himself, who wants to easy up life and generate the file, to ship it with the AI
* regeneration of an old info file, replacing one form an old version of an AI
as we will have the LUA file stored separate form the AI: what do you guess, how often it will happen that a user downloads AAI.zip. containing the dll and the lua, and will only extract the dll?
-> very often
in my scenario, we could allow the lobbies to let the engine generate all missing info, when clicking a button. not the best solution, but better than nothing.

or if you want:
assume all AI devs will ship their lua files with the AIs.
then my scenario is the exactly same as what you want, and none of the code you don't like will ever be used.
and then look at that code, as help, when something goes wrong. eg:
a user comes to the forum, and sais, an AI does not show up in the lobby, even doh he downloaded nad installed it. so you can tell him to download again and extract the lua file to this and this directory, or tell him to run spring.exe --regenerate-ai-stuff. you can still tel him the first thing, and not mention the later one, if you so fear it.

i though about the <spring_root>/AI/Skirmish/data/ dir as container for the cache and config files and such only, but yeah, we could also have the info files there, I do not care. though, then the subdirs in data would have to contain the version as well, eg ../data/RAI-0.600/, instead of only ../data/RAI/ , though, that should not be a problem either.

during the configuration process, we could output something like this:

Code: Select all

generating missing AI info files
================================

generating info for AAI ...
../AAIinfo.lua successfully generated
generating info for RAI ...
../AAIinfo.lua successfully generated
generating info for hoijui ...
SEGFAULT

i was thinking of the generation process as run and then exit. so whenever you start spring with one of the generation parameters, it will never try to load a game at all. just generate the info and exit, or crash trying.

@AF
i think, no lobby should ever have to know about the interfaces either, they should never specify it. that is how i intended it.
but as AI dev, i could be porting my AI from C++ to Java eg, and then i could specify it manually in script.txt (which interface library to use, if the AIs would also have the same version).
the only difference between what you want, and what I did is:
when we have:

Code: Select all

C#0.1_RAI#0.600
Java#0.2_RAI#0.600

and a lobby specifies one of these:

Code: Select all

aikey=RAI
aikey=RAI#0.600

what to do then?
in my scenario, the engine would choose one, and possibly issue a warning that the AI was specified ambiguously. in yours, the game would halt and issue a warning. i can see that your solution is a bit better, as one could oversee my warning, but this is really a small issue, as it most likely will not happen that a user will end up with two AIs with same shortName and version, written in different languages. though, if you really want, we could also let the game crash in my scenario.

so.. lets say, we never specify shortName or version of the interface in script.txt, we would still end up with at least these two:

Code: Select all

aikey=RAI
aikey=RAI#0.600

if a lobby does not want to, it would never have to use a version, and the engine would simply always take the latest version of that AI available. but i think we have to support specifying the version somehow, as we may want to use an old version, cause to works better with a specific mod, then the latest one, eg. are you ok with RAI#0.600? or would yo use two tags like this:

Code: Select all

aishortname=RAI
aiversion=0.600 # optional

or how?

[BrainDamage]

how do you intend to inform the lobby about AIs and their available versions?

are you gonna add unitsync callbacks to list available AIs and their properties? (version, ecc)

EDIT: please don't say to load the AI from the lobby because in my experience many AIs are really crash prone and having the lobby crash due to a bugged AI would be quite silly (not to mention i won't know how to load a java based AI, ecc)

[hoijui]

no worry brain damage, lobbies will not load AIs.
i do not know what ylobby defs prefer, possible scenarios are:
* lobby reads info directly from LUA info files
* lobby asks unitsync, which gets the info from the LUA info files
* lobby asks unitsync, which executes spring.exe with --list-ais, and parses the output
or if there
i would have though the second one is the best option, but it may not. are there other ways i did not think of?

btw.. why is it called unitsync? was it initially only used to let lobbies acess unit info? is it ok if i think of unitsync as a common interface for lobbies (an possibly other programms) to access game data/content?

[AF]

What will C# do since your using the # seperator?

Also I would have NTaiJava if I were doing a java port. Add an AIImpl= tag if you wish for debug purposes but the engine should be able to figure this out for itself. Perhaps instead of crashing ti can simply load the first valid matching AI it finds.

Code: Select all

char[][2]

or...

Code: Select all

char** keys, char** values, int keycount

I thought that was somewhat obvious

[Agon]

[quote="AF"]What will C# do since your using the # seperator?/quote]
C# will not have problems with that. It is then impossible to use #.
But I do not like it.
Doing it like this is better and used in database structures, too:
ainame="xy";
aiversion="0.1";
aiinterface="C";
....

Easier to understand for AI developers and using # is possible.
If a AI has the " char in his name you could do it like this: "yasd""a22ys".