Support for multiple engine versions

[Licho]

Idea:

2 support libraries for lobbies (only springsync being used directly)

1) "springsync" - used for common things - it could provide archive checksum and iterate engines installed

2) multiple "unitsync" (require no major change, can be current unitsync) - used to do engine specific things, like calculating full mod/map hash, iterating mod options etc. Springsync will have "SetEngine" command to pick correct unitsync and relay requests through correct unitsync dll

Lobby protocol changes:
- for maps and mods archive checksum will be used instead of mod/map checksum which depend on spring content.

- each battle advertises its engine version, mod and map archive checksums

- when player joins battle room full mod checksum (not archive) can be used to check for sync

[Licho]

Also for this to work:

* spring engines stored in extra folder along with their springcontent and unitsync

* spring.exe will be optionally replaced with loader that picks correct engine based on version number or script tag

[imbaczek]

need uberserver deployed ASAP IMHO.

[AF]

I like your spring loader idea, a minimal wrapper written in C++ should do the trick requiring only the necessary data to determine the preferred engine.

Ill write up a design document detailing this on the way home.

[Licho]

The existing protocol requirements already exist in uberserver, there's even support for different game engines nm their versions.

Yes protocol is fine, just semantic is not correct. Current lobbies send MapHash instead of Map Archive hash..
Map hash depends on springcontent.sdz and that should be allowed to change from version to version.

Map/mod hash used by lobbies should depend just on map file and not anything else, so that it can stay same across multiple engine versions

[hoijui]

i recommend using an other name then unitsync.

[Licho]

Springsync should contain all current unitsync functions + extra function to list engine versions installed.

All functions that depend on engine/springcontent will take extra "engine version" parameter and relay call to actual unitsync.

Lobbies would only need to use springsync and the idea is, that they would not need to use engine specific functions until you join battleroom, because listing map and mod archives wont need engine version number.

[Licho]

Distribution of engine files can be handled using current downloader infrastructure.
You need corrent mod and engine before you are allowed to join battleroom.

I also propose using the "STABLE" concept from CA. Simply marking some releases as stable and latest stable being the official "default" version to use.

[imbaczek]

spring uses svn tags for stable versions.

[Licho]

This however needs to be accessible through "springsync" system somehow..

You could store subversion revision number too (and this being the primary ID of the engine version).

It could be compiled to unitsync for example..

[AF]

I disagree.

Unitsync should remain separate from the engine version library, engineSync and unitSync not springSync. An example being sdd archives. Proxying between springSync and the old unitsync is also an unnecessary complexity, especially when your exposing APIs that dont exist in the target unitsync version.

For example what if the lobby builds a list fo parameters for spring based upon what unitsync/springsync says, but they differ, thus the parameters the lobby gives differ, thus desync? We must retain maximum flexibility.

In my proposal a lua file would control which is the 'preferred' engine, and would be modified when a major installation or upgrade occurs anyway.

[hoijui]

when doing this, we should not address it under svnRevision, but under revision. Make it a string (not int) or something, so we dont have to change it once we go over to git, as it uses hashes which are to big for int as revision ids.
..if i am not equivocated.

[Licho]

According to imbaczek, unitsync is 90% based on common code from engine. Therefore its engine dependent and subject to change.

Interface that lobbies use to iterate basic "static" information like installed engines and archive hashes should be independent from engine.

But you cannot make current unitsync independent because hashes contain engine version specific information - springcontent is being hashed.

Also you seem to not understand how its supposed to work.
Parameters WONT differ. Because you will call springsync "GetModOption" with engine version as parameter.
And engine-version-independent functions will be safe and stable (like providing archivehashes) .


My proposal has simple design goal - to make that work in universal way, without major changes anywhere. Changes that have to be done:

1) create wrapper for existing unitsync function that calls proper dll and contains getarchivecache function. (I dont call that complexity)

2) relatively simple change of lobby interaction with springsync by adding engine version for certain calls

[Licho]

when doing this, we should not address it under svnRevision, but under revision. Make it a string (not int) or something, so we dont have to change it once we go over to git, as it uses hashes which are to big for int as revision ids.
..if i am not equivocated.

Numbers provide easy to decode information about what is newer.

[AF]

The problem is the APIs differ from each unitsync to the next with additions being made. What will you do if something is added in 0.78 that isn't there in 0.77 and the lobby tries to use it while running a .77 game?

Also what about language bindings? Numerous times I've simplified and changed around my own Java bindings as I've learnt more about JNI.

I support the idea of using strings not integers. We should see 0.77b5 not 0174843 in the lobby, and making the lobby auto-replace is not a solution if you don't have that version installed already.

I'd also seperate the engine dependant code and the engine management code if possible for easier maintenance.

It may seem simple to you but its opening up a can of worms in the future.

[Licho]

Obviously if there is major change to unitsync interface involving addition of new functions/paramters, it would mean full new release and new springsync.

But since such changes are rare (in the terms of commits involving unitsync interface change vs commits total) its not a big problem.

Regarding engine names- obviously engine can have string identifier but we should also have something hardcompiled into engine/libraries to prevent easy tinkering by renaming files.

Springsync cannot rely on user to keep version 0.74b8 in folder "0.74b8" it should doublecheck by querying unitsync in that folder for actual version.

And since all commits will be availiable for use, subversion revision is logical choise for internal identifier (it does not have to be displayed that way, it does not have to be displayed anywhere at all! - just your "engines" or "bin" folder will contain numbered subfolders end unitsync source will contain hardcoded revision number)

[AF]

Not exactly user friendly. I also don't like the idea of polling unitsync for the version string, we shouldn't load and unload libraries unnecesarily. Id like it if we could keep data like that purely as data without requiring binary execution to occur. If only to reduce complexity in the library chain

[Argh]

I don't have anything intelligent to say about the technical side of this. I'd just like to say that the minute that we can have multiple engine versions can't come soon enough- it'd make life a lot easier for players and for game designers wanting to give gamers the true "it's all in one box" experience.

[Licho]

AF - but there is no other way..
If some data depend both on engine and on content (as is the case of unitsync produced data) you need something to generate them on the fly..

Also you dont have to load unecessary libraries.. springsync will be smaller than unitsync.
And lobby witll request engine specific data only when you join battleroom - springsync will simply load dll on demand and forward call (and it can keep library in memory)..

As a lobby developer this is not your concern - you will only use springsync .. there is nothing else from your perspective and implementation might change in the future.

[AF]

AF - but there is no other way..
If some data depend both on engine and on content (as is the case of unitsync produced data) you need something to generate them on the fly..

Also you dont have to load unecessary libraries.. springsync will be smaller than unitsync.
And lobby witll request engine specific data only when you join battleroom - springsync will simply load dll on demand and forward call (and it can keep library in memory)..

As a lobby developer this is not your concern - you will only use springsync .. there is nothing else from your perspective and implementation might change in the future.

You cannot say this, especially where tool based management of content is involved, and you already know your use of the registry makes spring downloader a huge no-go for me. You also forget, I'm not just a lobby developer! Either way dismissing me out of hand because I'm a lobby developer is disrespectful.

Your proposal for springsync seems hackish. I would suggest even if we did follow your proposal that we instead use lua files in each directory to specify the version. This would be both safer from a security perspective and easier to manage in the tool chain both for buildbot and the lobbies and the libraries and spring itself.

It seems pointless that we load springsync to access an API in a library via proxy when the proxied library exposes an identical API to that which we expect. You still haven't solved the problem of what happens when a springsync method is called that does not exist in the target unitsync.dll. Missing functionality.

Most of all the naming scheme for folders is counter intuitive. Nor should we rely on folder names to tell us which version of the engine is the newest, and thats botha technical and a user issue.