Spring RTS Engine

AF wrote:
Currently native AIs can do what players do, they cant send commands to enemy units, etc

No, they can not. They can not grasp when lua changes something. And its about more then just not grasping some auras and shields. If you dont define were you want to meet the Gamedevs half - way the resulting matches and experiences will always be flawed.

I don't see why you are calling this thing a mutator. It's an AI packaged in a spring archive that runs widget code in it's own Lua state (which is a good idea as it could have a different API than regular widgets).

Silentwings wrote:Games that ban userland widgets would not matter - since each such AI would need to have its own unsynced widget lua state, running only on its host, and would not be part of anyones userland widgets.

Google_Frog wrote:The predominate LuaAI suggestion seems to be to run widgets in their own lua state. These widgets would be like player widgets but without the player. Widgets can see a lot of what goes on in a game and as far as I am aware they can do any player action.

So I'm having a trouble understanding if you actually plan to allow only one AI per AI host? Or would you allow multiple AIs like it's currently being done? Can I still be a player and play against the AIs I host? (note that there's a difference between game host and AI host, or AI owner).

We need multiple AIs in order to have them compete, which is essential when developing a new one, as you have a "base AI" you want to be able to best.

I see no problem in a single player hosting multiple such AIs.

I did not call it a mutator - I called it an unsynced mutator and later qualified with

the point of using the words unsynced mutator is to define a new object that would not need to be regarded by lobbies as a different game, only as a package of widget-style code that ran entirely inside of its own unsynced lua state.

If you dislike the term feel free to invent another, I don't care much for semantics.

FLOZi wrote:You are asking for two things here:

1. Ability to write lua AIs that are purely unsynced (if you had bothered to look, CRAIG runs as much in unsynced as is currently possible)

2. Ability to distribute all (native + lua) AIs via rapid etc.

Also, this. I'd really like to see 2. discussed for native AIs as well. I've stated my reasons why I think non-Lua AI interfaces are important to have in the other thread.
At the very least you'll want to specify the arch for native AIs (even those in Java sometimes). This shouldn't be too hard to do, you can just have separate repositories for different arch's.

Silentwings wrote: If you dislike the term feel free to invent another, I don't care much for semantics.

OK, how's "Lua AI"? How is this different to other AIs other than not using the C AI Interface?

I also don't see the point of exposing the entire Spring Unsynced Ctrl/Read + Synced Read interface here. What's lacking in the C AI interface and why can't we have it added?
What I understood this proposal was desired in order to access some data available to the game only, but if you isolate it in a separate Lua state (which you should), how will this data still be available?

The name "Lua AI" is already taken by the AIs as gadgets included in games.

Maybe we can call those independent AI written in Lua : "Lua Bot"?

# 2 is irrelevant at the moment since all native AIs are bundled with the engine currently, and there's little interest in developing for an undocumented API, short of pestering myself and engine devs or trying to poach ex AI devs out of retirement

I would hope that the lua AI would be as close as possible to the current lua AIs, can I assume when talking about the widget API you're referring to everything that isn't synced/doesn't require a gadget? If so then that, attached to a headless player instance in place of the native AI interface would be appropriate.

Regarding separate lua environments, is that how widgets are currently implemented?

Synced vs Unsynced Lua AIs should suffice for naming

AF wrote:# 2 is irrelevant at the moment since all native AIs are bundled with the engine currently, and there's little interest in developing for an undocumented API, short of pestering myself and engine devs or trying to poach ex AI devs out of retirement

a) Not all are bundled with the engine. You could get my AI without it, as well as any other Java AI I guess.
b) I'm only interested in native AIs, as are the two current AI devs we have . Popular game AI competitions allow native AIs: https://code.google.com/p/bwapi/ , http://robocode.sourceforge.net/ , https://skatgame.net/mburo/orts/orts.html
c) There is existing documentation for the Java interface at least: http://spring.abma.de/doc-Interface-JavaOO/ , http://springrts.com/wiki/AI:Development:Lang:Java
.

AF wrote: I would hope that the lua AI would be as close as possible to the current lua AIs, can I assume when talking about the widget API you're referring to everything that isn't synced/doesn't require a gadget? If so then that, attached to a headless player instance in place of the native AI interface would be appropriate.

Regarding separate lua environments, is that how widgets are currently implemented?

You have separate lua "states", such as LuaRules, LuaUI, and LuaGaia, with the idea that you can't access things from one state in another directly. LuaAI would be a state for AIs, which runs unsynced much like the LuaUI.

AF wrote:Synced vs Unsynced Lua AIs should suffice for naming

Yep

gajop wrote:

Silentwings wrote: If you dislike the term feel free to invent another, I don't care much for semantics.

OK, how's "Lua AI"? How is this different to other AIs other than not using the C AI Interface?

I also don't see the point of exposing the entire Spring Unsynced Ctrl/Read + Synced Read interface here. What's lacking in the C AI interface and why can't we have it added?
What I understood this proposal was desired in order to access some data available to the game only, but if you isolate it in a separate Lua state (which you should), how will this data still be available?

As far as I can tell the proposal aims to combine the advantages of both Lua AI and native AI.

The desired advantages of native AI are that they can be updated and run independent on the game and its release cycle. Performance is improved because only one player runs the AI and the AI developer can rapidly update and test their AI without cooperation from the game developer.

I am less well aware of the advantages of Lua AI because I don't know how native AIs fail. Apparently the native AI interface is broken/undocumented because it is not used by many people. Additionally luarules has been used to add game mechanics which native AI currently has no way of accessing. The lua interface is actively used by many people which results in it being powerful and well documented. So the advantage from the lua side is a well documented interface that is capable of interacting with almost all of the game.

The problems with the native AI:

- Only Hoijui really understands how it works, and he's no longer active
- The interface is unfinished, e.g. the OO C++ VFS apis are incomplete, you have to use the low level C version of the API
- Parts of the interface don't work, nobody is entirely sure why
- A substantial portion of the interface is auto-generated
- Those generation scripts are poorly understood
- There's no documentation of the generator scripts
- There's no documentation of the output
- Nobody is actively maintaining the functionality beyond making sure it compiles with the engine
- There's no reference listing what objects and methods are available
- Getting a copy of the interface headers involves grabbing a copy of the full engine and running cmake + awk to generate headers from scratch, which is terribly for new users wanting to look at the API before they start writing code

So the native interface is unstable, undocumented, poorly maintained, and nobody knows how to fix/add to it. Correcting this situation would take a substantial amount of work.



In the meantime, people have expressed an interest in lua AIs. My proposal should make it a lot easier for these people experiment and play, so that their work can then be distributed and bundled into games as official AIs.

AF wrote:The problems with the native AI:
- Only Hoijui really understands how it works, and he's no longer active

Incorrect. There are still people who understand parts of it (enough to develop and maintain it).

AF wrote: - Parts of the interface don't work, nobody is entirely sure why

Which?

AF wrote: - A substantial portion of the interface is auto-generated
- Those generation scripts are poorly understood
- There's no documentation of the generator scripts

It's understood, and I don't see the need for docs of the scripts.

AF wrote: - There's no documentation of the output

There certainly is for java (check the AI-Java dev page).

AF wrote: - Nobody is actively maintaining the functionality beyond making sure it compiles with the engine
- There's no reference listing what objects and methods are available

Wrong and wrong.

AF wrote: - Getting a copy of the interface headers involves grabbing a copy of the full engine and running cmake + awk to generate headers from scratch, which is terribly for new users wanting to look at the API before they start writing code

That's a distribution problem, not an API one.

AF wrote: So the native interface is unstable, undocumented, poorly maintained, and nobody knows how to fix/add to it. Correcting this situation would take a substantial amount of work.

Not true. It seems to me the issue is people who complain on it don't really know what's wrong with it.
You should first do some research like Anarchid did to get a clear picture on the status of the interface. You would be surprised to see that some of the things you find lacking are actually present, and while other things might require work, it doesn't mean we should ignore doing any work just because of laziness.

I have nothing against the addition of Lua AIs, but as I said, they should use the same AI interface as the native AIs do. You also shouldn't necessarily give Lua AIs direct access to game (gadget) data, or the ability to modify the game using the existing mod Lua_Scripting API.
AIs need their own API (AI interface), and only using that can we improve the state of AIs: such as having flags for different degrees cheating/non-cheating AIs, interface to retrieve AI-specific data from games, and so on.

- That doc set is for the Java API, not the legacy C or C++ OO APIs

Which?

Lua <-> Native AI comms, specifically the ability for an AI to send a message to lua is broken. Kloot says it's only working in half duplex mode, the full duplex code causes crashes because the API generators can't handle C pointer pointers. The VFS API for OO C++ AIs is also incomplete, the C API is there, but the OO layers are incomplete with abstractions

http://springrts.com/phpbb/viewtopic.ph ... 92#p546792

kloot wrote:The data return paths from fudget:RecvSkirmishAIMessage (invoked by AI:CallLua*) and from AI:RecvLuaMessage (invoked by Spring.SendSkirmishAIMessage) do not work because the interface generator scripts are too primitive to handle basic C pointer-pointers. I was initially unaware of that "limitation" (not being their author) so full bidirectional AI<-->Lua communications briefly functioned as advertised (for legacy C++ AI's) until this happened and they had to be disabled. As a result only half-duplex mode is available which means you can send data from Lua to AI's (but not hear anything back) and send data from AI's to Lua (but not hear anything back), but nothing else. You must write your own syn/ack system to get around this.

Anarchid implemented Lua Message support recently ( I was unaware of such an event ), so a work around should be possible

This would be pretty cool, people could theoretically add AIs to games they made themselfs.
On the other hand ... "yeah right".

People can already do that - the point of this thread is offering AI developers a measure of independence from game developers (which they used to have, but which was lost for various reasons). Maybe I just haven't parsed your sentence as you meant it

I meant to say that players can let 3d party AIs play from their PCs without them being included in the game.

I have a very good understanding of the current state of the Lua AI interface, thought I'd clarify some information because I think some people might be looking for easy answers that don't actually exist. For brevity AI means "Lua AI" unless I specifically say "native AI" (meaning the C++/Java interfaces)

Synced Lua AI
* The current AI interface is synced and gadget-based.
* It runs in the same environment as other gadgets.
* It cheats automatically (can access enemy data), but
* Can fake "unsynced" by filtering out data it "shouldn't know" (eg, ignore unseen units for team X). I think CRAIG does this and I also have some wrapper code based on (trepans?) old work encapsulating each team AI in a closure.
* Can play on a team/alliance
* Runs simultaneous on cpus of all players (like all synced code)
* Can send serialised data to unsynced widgets (insecurely)
* You can select them from the lobby if the game includes a LuaAI.lua file
* Therefore, Lua AI must be known/supported by game and shipped in the game package, unless..
* You write a mutator "game" with your AI and give it to all players, eg. create BA6-MyAI.sd7 that either wholly includes or better yet requires/depends on the main game.
* Cannot read game-specific data (eg, custom resources or rules) unless the game deliberately exposes them.

Unsynced Lua AI
* Provided the game allows user widgets you can have an AI help your units
* AI can only see detailed data for your team. Enemies have limited data based on LOS/radar
* Only runs on your computer and can be installed in system spring path
* Can communicate with gadgets or synced AI when the gadget allows for it.
* Can be blocked if widgets are blocked/disabled.
* Can cheat if cheats are enabled, but all players get cheat
* Can't be selected in lobby but can be enabled/disable at any time (assuming AI widget is written to support "resuming" or mid-game starts)
* Can cheat via a synced gadget and synced<->unsynced communication (insecure)
* Cannot read game-specific data (eg, custom resources or rules) unless the game deliberately exposes them.

AI over TCP/UDP Unsynced
* Can be done in a widget (Unsynced Lua AI) using socket networking library (LuaSocket) which is already available to communicate with an external AI.
* Good knowledge of socket-based networking recommended
* Language of AI would be irrelevant as long as it supported sockets.
* Can be blocked if widgets are blocked/disabled.
* May require some user settings changes (LuaSockets allow/deny list)?
* Not possible or advisable for Synced AI because network issues will desync game.
* Cannot read game-specific data (eg, custom resources or rules) unless the game deliberately exposes them.

Native AI (C++/Java,...)
* Currently required to run a few popular BA/XTA playing AI, but
* Is in a poor state in practically all respects
* Has limited access to Lua state, therefore is problematic with non-*A games. Spring 1944 comes to mind as it uses Lua for much of it's game logic.
* Known to cause crashes and desyncs.
* Known to have old unpatched bugs.
* Can break engine compiling under some setups.
* Cannot read game-specific lua data (eg, custom resources or rules) unless the game deliberately exposes them.


I highlight the note about "game-specific" because it's wholly relevant to the debate on whether an AI should require the support or permission of the game/mod developer. When the debate gets emotional or moralistic it's easy to overlook how many factors relevant to the "state" of the game your AI will NEED to be an effective player and how much work and planning is required to expose and manipulate those factors. A simple case can illustrate this:
The logic seems correct but the question is how does the AI logic get the state of enemy_has_flag or the current location of the flag to begin with? How does the AI activate the teleport feature if it was implemented in Lua and not the engine? Do we make them global (add to _G), do we receive a callback, perform a callin, send an "event" message, ...?

All of these approaches work in theory so the questions might be asked "which method(s) are best, and for who?" and "should the game and/or engine endorse or enforce particular method(s)?"

Neither question seems to have a simple answer. I'm not even taking into account here what the players and engine devs might want and how much work is involved in each option. What happens when the game interfaces are added and removed over time? What about the broader AI research / bot battle communities who might might to test their "general purpose" RTS AI on your Spring game?

Even a very simple game is going to present these challenges so what it boils down to in reality is that an AI author is going to have a much easier time of things by working directly and cooperatively with the game author(s). This will remain true regardless of the interface used. There isn't much the Spring engine developers can do about it "generally" other than either restrict/remove options OR provide as many options as can be practically maintained. There's unlikely to be general agreement on that because the needs of all developers - engine, game and AI, will be different in nearly all individual cases.

Nice summing up of the current state of things. There are just two points I'd like to complete:

Unsynced Lua AI are "Widgets so advanced that instead of merely helping the player a bit, they can play and win an entire game for him".
It's possible, I have one. But people around here generally call them widgets and not Unsynced Lua AIs.

SpliFF wrote:* Cannot read game-specific data (eg, custom resources or rules) unless the game deliberately exposes them.

Yes, a custom resources or rules could be implemented in a self contained gadget that doesn't expose anything. However, in practice, custom resources system gadgets almost always already have gadget interface, and are even often implemented as a set of gadgets instead of a single one. What it means is that:

Synced Lua AI can easily interface custom resources and rules, either because the interface is already there, or because it's a couple lines of Lua to add in the custom resources/rules gadget.

Unsynced Lua AI can access custom resources and rules with little more difficulty: It must goes through the synced-unsynced interface, which confuse beginners Lua coders. Still no real issue.

Native AI (C++/Java,...) have a very hard times accessing custom resources/rules. I've heard rumor of a Native AI - game gadget interface, but nobody ever used it, and AF and Kloot were discussing some bug or limitation of it earlier this thread.

Also, having each side of that interface written in different language by different people for different projects doesn't help. I mean, most of the time Lua AI are written by modders game-makers, familiar with Lua gadgets, if not by the game's author himself. While Native AI are often written by people only interested by the AI side of things, and who aren't familiar with Spring Lua gadgetry.

zwzsg wrote: modders game-makers

Using the M-word are we. Discrimating Noobians, are we? Just good enough for file modification, but certainly not allowed to create new files, are we? So nice, to donate some of your gamedesign privileg to us. Thank you Master, allow me to kiss your feet, oh shit, somebody did bite a toe off- how that could happen... we shall never know..

. I've heard rumor of a Native AI - game gadget interface, but nobody ever used it, and AF and Kloot were discussing some bug or limitation of it earlier this thread.

This is no longer correct.

Here is an example of an AI sending a command (not unit command, a direct message) to a gadget.

Here is an example of a gadget sending a data structure to AI world, then to be parsed here, here and here, occasionally resulting in this.

The idea that LuaRules world and Native AI world are wholly separate can now be considered thoroughly smashed.

The actual issues with native interfaces seem to all be related to multilayered metaprogramming. Who would have guessed? Writing programs that write programs which are then used as source for programs that write other programs using those former programs sounds quite robust...

The C interface seems rather wholesome if unconvenient, but the most actually documented one - JavaOO - is riddled with horrible lasting damage, for example:
- Querying customparams causes an immediate segfault.
- It is not possible to send messages to LuaRules world from Java (while possible from C and CPP).
- reading GameRulesParams is impossible in all native AI's
- Reading UnitRulesParams likely also causes a segfault since 95.0 which added string values to URP -- and AI wrappers were never told.