We are developing an AI with RL (Reinforcement Learning). When using RL we need to train the AI throughout several thousand games.
For this purpose we use the headless spring, which works very well. However, it is a problem that you can't directly reset a game from the AI. This means that you need to do a complete restart & reload of the game for every match. This loading takes ~13 seconds, which is alot compared to the actual gametime of 5-10 seconds at speed 120. Waiting the ~13 seconds for every single match is a huge waste of time.
As an attempt to solve this, we self-destruct all our units and add a new commander (using cheat) at the start position. This way we can somewhat start a new game, but it is an ugly approach. It still has the problem that a self-destruct explosion destroys the ground underneath, slowly revealing water and making the area unusable.
So now to our actual feature request Why not either add a remove-unit cheat (without explosion and countdown), or even better add a possibility to reset/restart the game, without having to reload all the spring resrouces again. Such a reset/restart would simply reset the map and reload the AI's.
Feature request: Remove unit cheat and/or Reset/restart game
Moderators: hoijui, Moderators
Re: Feature request: Remove unit cheat and/or Reset/restart game
add a luarules gadget to the mod you're doing your training on - this will be the simplest solution for now.
Re: Feature request: Remove unit cheat and/or Reset/restart game
Also, with regards to the map getting damaged, you can use the engine option NoMapDamage to disable map damage altogether.
Re: Feature request: Remove unit cheat and/or Reset/restart game
Thanks for the tips!
I am not familiar with the posibilities of LuaRules in Spring, so what exactly would be possible with such a script?
I am not familiar with the posibilities of LuaRules in Spring, so what exactly would be possible with such a script?
Re: Feature request: Remove unit cheat and/or Reset/restart game
You have to edit the mod file itself. (Unzip it with 7z, and zip it back with 7z using normal compression mode). Look at luarules\gadgets inside the mod, Those are lua scripts that are, how do I put this, unlimited in their power to change game state.
Re: Feature request: Remove unit cheat and/or Reset/restart game
Hi allanmc,
As I am a student A.I. I would really like to know some of the details of the RL approach like:
Very curious!
As I am a student A.I. I would really like to know some of the details of the RL approach like:
- How did you define the states?
- What actions have you got?
- What is the reward function?
- What RL algorithm are you using specifically? (Q-Learning, Sarsa,...)
Very curious!
Re: Feature request: Remove unit cheat and/or Reset/restart game
Hi Error,Error323 wrote:
- How did you define the states?
- What actions have you got?
- What is the reward function?
- What RL algorithm are you using specifically? (Q-Learning, Sarsa,...)
This AI is part of a Master Thesis in Software Engineering, which i am doing along with three other people (on this forum: allanmc, initram, jepperc and shredguitar).
Currently we are trying to use BN to classify the opponent, and RL only to build the base (reach many labs as quickly as possible). In this simple RL we just have 3 build actions: Lab, Solar, Mex. The counts of these are used as state variables - Lab and solar can be in the range 0-19, and Lab in the range 0-4. The time used to build any of these buildings is used as negative reward, which ensures that the AI eventually figures out that it needs resources to build quickly. The goal state is reached when 4 labs has been build, and this results in a high positive reward. This has been implemented with standard Q-Learning, and works very well.
Right now we are in the middle of expanding the use of RL to more usefull cases, such as complete base building, attacking, scouting, etc.. You are right that with these bigger problems, the state space, among other things, can become a problem. Therefore we are currently looking into the possibilities with hierarchical RL: http://www.ijcai.org/papers/1552.pdf
Re: Feature request: Remove unit cheat and/or Reset/restart game
You also should probably be culling the wrecks between restarts. All the unreclaimed stuff that existed before the big Self-D is still going to be sitting there.
Got a screenshot of the 100th game? I want to see what that map looks like after that much mayhem.
Got a screenshot of the 100th game? I want to see what that map looks like after that much mayhem.
Re: Feature request: Remove unit cheat and/or Reset/restart game
features too, you want a map with little in the way of trees and metal rocks