No Saves makes it impossible to play the greatest games.

[FLOZi]

Engine assisted save/load has existed for ages but noone has updated the lua solutions to utilise it.

End of story.

[Tim Blokdijk]

People here and especially the engine developers understand what serialization is.

I didn’t look into the source code yet, because I don’t think I will have the time to contribute in a useful way at all in the next few weeks anyway.
I just wanted to suggest an approach to fix saving (and to show that there is at least one person more who would like to save his progress) and hear what you think about this proposal.

We can wait a few weeks.
With a few years of C++ development you should be able to contribute. Just read up on the source code and ask questions where necessary.

The mayor disadvantage of serialization is that you can't save with version x of the engine, update to version y and load your game again.
Also implementing serialization code can introduce all sorts of new bugs.
Serialization of AI state? Lua?

So developers decided that loading and saving the game via lua is a better solution.

[Fast2]

People here and especially the engine developers understand what serialization is.

Oh, I didn’t want to put this in question.
I just wanted to emphasize that – in theory – it should be easy and that I don’t really know what is hindering the integration of some means of saving.
(Actually not meant as a reproach (it would be one only if it was really easy and not much work (but I don’t say that it clearly is!))
The Lua-code should be able to use the same model. Since it can interact with the engine, saving shouldn’t be much different (or maybe even easier).
It should even be possible to equip the interpreter with means of saving the complete state of execution, this would make it transparent to the scripts, even simplifying the development of them. :) (Yet if this was impossible duo to inconsistencies with handles or something like that, using the approach the actual core of the engine uses should also be possible.)

With kind regards,
Fast2

[Forboding Angel]

Edit:
I didn’t look into the source code yet, because I don’t think I will have the time to contribute in a useful way at all in the next few weeks anyway.

Not to be a jerk, but everyone reading this just rolled his eyes, no matter how pure your intentions.

[PicassoCT]

I just wanted to emphasize that – in theory – it should be easy and that I don’t really know what is hindering the integration of some means of saving.

With kind regards,
Fast2

Well, well, if it is so easy in theory, maybee you could do explanations for me. About Floating Point determinism, about synchronized deterministic simulations and the whole theory, that doesent come as easily to me.

Still, this is some might good, old fashioned trollin. You dont get this quality anymore today. Just some fast Offensive re-post. The whole psychology thing is completely left out.

Here have some fish for your efforts
>(((((°>

[Tobi]

Problem with the serialization solutions (we had one of those in the past, which worked occasionally -- we got our own reflection framework for C++) is twofold:

First, it likely won't get updated before decent tests for it are in place, so effectively it breaks every release (if just one field of one class isn't serialized properly it is end of story, in particular if it is a pointer).

And second, the classes need to actually support being re-instated out of thin air. This is likely the most work initially, as there are too much spaghetti-dependencies (one class needs another, which might not have been loaded yet, to re-initialize itself, etc.), and some classes may need external resources that are different each run (e.g., texture IDs).

[jK]

First, it likely won't get updated before decent tests for it are in place, so effectively it breaks every release (if just one field of one class isn't serialized properly it is end of story, in particular if it is a pointer).

This could be partly fixed by adding a validation test that errors when the CREG registered members are unequal to sizeof(class). And it could warn about registered pointer members.

[hoijui]

i dont think that this is a solution, jk.
we do not want to serialize all class members, as some might point to memory allocated by a 3rd party lib, or some might contain.. something like textures, that we do not want to put into a save-file, but reload at startup. in general, it is a lot of work to maintain an engine/C++ based save-system. it is very ugly, and it only has a slight chance of being usable with a lot of tests, as was already stated. even these tests though, would not reduce the work required, and this work is required for practically every 2nd commit, which means workload for commits rises by x%. not to mention that many errors done in commits are only surfacing with the tests (which will never exist), and thus we will see a Load-system/CREG related fix commit after every 5 or so other commits.

all the work that had to be put into this, including the tests and the constant additional workload, could be put into making so much more other stuff... it is simply ridiculous to go that road, as with that amount of work, we could make the engine clean, after which point all sorts of other changes would be much simpler, and doable in a nicer way... even a save/load system (though that would still be one of the things way low on a sane mans todo list).

no, a class does no know at all what it should save, and how, and how to reload it.

[jK]

i dont think that this is a solution, jk.
we do not want to serialize all class members, as some might point to memory allocated by a 3rd party lib, or some might contain..

You don't understand.
CREG should know about _all_ members, but it shouldn't serialize all (runtime data). There already is CR_RESERVED(x) that marks such members in a global way (it would be better if you do it on a per-member level). So at the end CREG should be able to check if there are missing/unregistered members in a class.
This would fix like 90% of all bugs/issues/problems with current save/load system.

[Tobi]

Hmm, good point yeah, though 90% is very optimistic

(Generating CREG definitions from header files might also work, but would complicate build system quite a bit.)

I agree with hoijui though that time would be better spent currently in cleaning up the code. Quite decent save-load is already possible using the Lua and AI call-ins anyway.

Note: CR_RESERVED was actually intended for forward compatibility, i.e., to reserve some space in the file so more members can be added later without changing the size of the structure. The class still needs to figure out by itself whether it is loading an old version of itself or not though.

[Fast2]

Hm, firstly, I have to complain that seemingly everybody (of course it’s just a few people, the exaggeration should emphasize it’s annoying nature) accuses me of being a troll. What am I doing wrong?
The only reason I can think of is that I’m really that stupid because* I didn’t want to troll …
*To the native english speakers: Why do you (at least according to my English-teacher and English-schoolbook) never use a colon comma before because? It irritates me every time.

PicassoCT:
The serialization-framework should take care of using a portable floating-point representation

Tobi:
Using my method, I don’t think this would be an issue.
Every class knows about what to save from itself and about its members.
The framework and each class should give back to each depending class what it got from it.
So, for example, suppose the map, here called Map, has a member, Heightmap, which is a subclass, another one, Texture, also a subclass, and one called Length, a long.
It should have two methods, commonly named Serialize and Deserialize, I assume.
That’s what you should have already had.
So, I think of Map::Serialize’s job as to call Heightmap::Serialize and Texture::Serialize, to collect what they return, and to add Length’s data.
So, in Map::Serialize(Savegame) (Savegame being the required object):
Heightmap.Serialize(Savegame); Texture.Serialize(Savegame); Savegame.Serialize(Length);
Texture and Heightmap will do the trick by theirselves, maybe they would just use something like Savegame.Serialize(Filename);
Deserialization would be the other way round:
Heightmap.Deserialize(Savegame); Texture.Deserialize(Savegame); Savegame.Deserialize(Length);
With Texture just reading the name of the texture file and reloading it like it does when the engine starts a game from scratch. If it shouldn’t touch the files directly, then it would redo whatever it does when starting freshly.

What I was trying to express is that I didn’t know what hinders the implementation, as I don’t think that I, myself, with my extraordinary ingeniusness (;)), am really telling something completely new.
Now, you have given a few reasons, which I tried to counter.

It seems at least like I’m proposing another way of doing it, which would hopefully work better than this reflection-based approach you already had.

So, I hope I have answered all of your questions.
There is just one thing bugging me at this moment:

no, a class does no know at all what it should save, and how, and how to reload it.

What? Why doesn’t it? I really have to look into the code, this seems almost impossible to me.
Or did you just want to say that sometimes, the caller only has some plain reference to a baseclass, so without the use of virtual methods, it can’t properly save? There is probably a way to overcome this hurdle, but I believe that this will also need a glance at the code.

So, the only issue left is the arising depedency on different versions.
Maybe the seemingly scamped version which just came to my mind is already the best for it: Having some flags set by the engines loading function based upon the version number.
The necessary changes in processing the data from older version (initializing members with default-values, or just ignoring one) would then be added (with some special marker for easily finding them again) and removed every major version (which is then save-incompatible)
Does it really sound that bad? The assured cleaning with every major version should ensure that there are not that many if-s flying around.

As an example, suppose you would have dropped Length in Map and used the dimensions of Texture instead. In Map::Deserialize, there would be:
if(Map_Length_saved) { long Discard; Savegame.Deserialize(Discard);}
Map_Length_saved would be set by the engine based on version number.
Therefore, you would have changes in two places, in Deserialize and in Engine::Load. Not that bad.
If it really bugs you because you wouldn’t like to assume that it is possible to do these changes to both functions in a coherent way, then using boost::preprocessor to do the trick could be possible as well.

Already awaiting your answer,
Fast2

PS, having just pressed preview:
Hm, I really like writing long texts


Addition: Rejected try, left here so you see that I really think about it:
I have no idea how serialization happens with boost::serialization, but for demonstration purposes, I will assume that it behaves like a filestream.
So, in Map::Serialize(), it would be:
return Savegame << Heightmap.Serialize() << Texture.Serialize() << Length;
(With Savegame being some instance of what is actually needed, and with the necessary extra code.)
Map::Deserialize() would be:
Savegame >> Heightmap.Deserialize …
Oh, well, this won’t work, left it in this post as a reference of my thoughts.

Edit: Changed some member selection operators to scope resolution operators (that’s what I promised), corrected the question asked after the asterisk.

[scifi]

Someones been learning java

Anyways Fast2 i dont want to discourage you, its positive thinking, sometimes its harder to get prespective without looking at the engines code, and knowing how things are done.

From what youve posted that looks prety mutch like a standard serialization process done with java aplications, or any other language you migth know, java just popped up to mind when i read your post.
Only having "."s just remenbered the work i did with java these last 2 months.

<< remenber C++ but thats a diferent matter ;D

My 2 cents on the issue, we already have a somewhat unstable save game system, or replays that somewhat represent this Save system. So id say this, debug the save system, example: when you reload a save or replay the file is corrupted somehow.

And then its up to us Modders to include a button to save a game automatacly. Making it user friendly.

I dont see anny need for more functionality. But maybe its just me i dunno.. Anyways this is just a post to cheer you up.

[Tim Blokdijk]

Hm, firstly, I have to complain that seemingly everybody (of course it’s just a few people, the exaggeration should emphasize it’s annoying nature) accuses me of being a troll. What am I doing wrong?

I think your complaint is valid.

Try to understand that the guy that started this topic (not you) made comments like "Spring is a failure as a engine" and "ALL of this effort a complete Waste". While only talking about Total Annihillation.
As 90% of the Spring project is about developing all sorts of exciting new stuff..
Anyway, not the best topic to have a technical discussion about a feature. If you are interested in contributing to the engine you should not hesitate to start a topic in de development section of the forum.

[PicassoCT]

My apologies, Fire, looked at this thread for a second time, and found you to be actually a nice guy, and even willing to contribute. Welcome, dont take the welcome salute as the final word. Actually lots of nice people here, working on exciting stuff.

PM me, if you ever need a scarecrow to ward of teenagers who want to be CEO of a Modteam, and awarded you with the honor to do there work for free.

Hope that Serialisation stuff works out.

[smoth]

the honor to do there work for free.

so i have this idea.... BWA HA HA

[Tobi]

Tobi:
Using my method, I don’t think this would be an issue.
Every class knows about what to save from itself and about its members.
The framework and each class should give back to each depending class what it got from it.
So, for example, suppose the map, here called Map, has a member, Heightmap, which is a subclass, another one, Texture, also a subclass, and one called Length, a long.
It should have two methods, commonly named Serialize and Deserialize, I assume.
That’s what you should have already had.
So, I think of Map.Serialize’s job as to call Heightmap.Serialize and Texture.Serialize, to collect what they return, and to add Length’s data.
So, in Map.Serialize(Savegame) (Savegame being the required object):
Heightmap.Serialize(Savegame); Texture.Serialize(Savegame); Savegame.Serialize(Length);
Texture and Heightmap will do the trick by theirselves, maybe they would just use something like Savegame.Serialize(Filename);
Deserialization would be the other way round:
Heightmap.Deserialize(Savegame); Texture.Deserialize(Savegame); Savegame.Deserialize(Length);
With Texture just reading the name of the texture file and reloading it like it does when the engine starts a game from scratch. If it shouldn’t touch the files directly, then it would redo whatever it does when starting freshly.

What I was trying to express is that I didn’t know what hinders the implementation, as I don’t think that I, myself, with my extraordinary ingeniusness (;)), am really telling something completely new.
Now, you have given a few reasons, which I tried to counter.

It seems at least like I’m proposing another way of doing it, which would hopefully work better than this reflection-based approach you already had.

This still has the main problem that the reflection-based approach also has: there will be a duplicate list of members (one in the class definition, one in the Serialize method, and one in the Deserialize method), which are bound to get out of sync unless tests are in place.

Actually the reflection part of CREG can be seen as thin layer on top of the rest of the serialization support: the difference between a loop though declaratively specified members and imperative code that goes through specified members isn't that big.

Problems such as cyclic dependencies and saving each object only once need to be handled by the framework. (and are handled by e.g. CREG)

PS: we had such a non-reflection based system before the reflection based system, it was less maintained and more broken...

no, a class does no know at all what it should save, and how, and how to reload it.

What? Why doesn’t it? I really have to look into the code, this seems almost impossible to me.
Or did you just want to say that sometimes, the caller only has some plain reference to a baseclass, so without the use of virtual methods, it can’t properly save? There is probably a way to overcome this hurdle, but I believe that this will also need a glance at the code.

I think the point is that although technically all this information is implicitly encoded into the definition and the semantics of the class, a developer needs to have this information in an understandable form to be able to implement serialization/deserialization.

In other words: Classes don't "know" things - developers know things about classes, and developers will have to specify exactly how a class should serialize/deserialize itself. (which is work, takes time, is prone to errors, etc. etc.)

[hoijui]

Classes don't "know" things - developers know things about classes, and developers will have to specify exactly how a class should serialize/deserialize itself. (which is work, takes time, is prone to errors, etc. etc.)

this!

i was hinting at the same, but i guess it was not obvious enough... pros for tobi to describe it well!

[Fast2]

So, I had some further thoughts, but no time to write a post.

scifi: Yeah, I’m learning Java for school, but it’s not fun!
I developed this naming-convention way before, maybe because I came from AutoIt to C++. But I must admit I didn’t notice that I had written points instead of the namespace-operator.
And thanks for trying to cheer me up :)

Tim Blokdijk: I think you’re right, maybe I should really have started a new topic in the development section.

PicassoCT: :)

Thank you, guys, you are really nice :) It’s ok, I was just a bit frustrated because it seemingly keeps happening and I didn’t know why.
I guess it was just a bad idea to write into this thread, instead of opening a new one …


Tobi: If one watched out, they would probably not get desynchronized, but it’s right that this is another trap for new developers.
I don’t know how cyclic depedencies should arise, is this really allowed to happen? (Or is handling them just meant as a failsafe?)
I guess I will download the code after writing this post, maybe it becomes clear when I look into it.
Bad to hear about the other system, I planned to write something about it, which I later on forgot

In other words: Classes don't "know" things - developers know things about classes

I look upon this like writing destructors, just some additional code.
You are correct, this was just the attempt to express that each class itself should be in charge for saving its state and the state of its members,
the reason being that the necessary informating is exactly there and not in some central save-function (without additional frameworks).
It’s not that classes have a brain, but the only changes required when changing a class should normally solely to be made to the class itself.


Well, if my other idea works out, it would – from the developer’s view – supposedly work similar to CREG.
Did you ever think of writing some additional program creating the necessary code? It could possibly even work without external programs (which you would probably dislike because of the additional hurdles when building the code), but just as a meta-program within the preprocessor and/or TMP, perhaps with the help of boost::preprocessor and/or boost::mpl.

So, finally, what I thought of:
I suggest creating some tool which can generate some boilerplate code required for the proposed system.
It would watch for three key words: NOSAVE, STDSAVE and CTMSAVE(CUSTOMSAVE), reacting in the following ways to them:
For each of these keywords, it would read a list of the members belonging to it (could be in curly brackets, looking like a block).
If it is in a NOSAVE block, it would do exactly nothing, except maybe registering the names for debugging purposes.
The members listed in the STDSAVE section would be written to the Serialize and Deserialize functions also solely generated by this tool.
For members with CTMSAVE, it would generate two function calls of the form Serialize_Member(Savegame) and Deserialize_Member(Savegame), which would be implemented by the developer, allowing for special cases where just calling Member::Serialize or Member::Deserialize isn’t suffficient.

Some example:

Code: Select all

class Map
{
  STDSAVE
  {
    long Length;
  }
  CTMSAVE
  {
    Image Texture;
    Image Heightmap; 
  }
  NOSAVE
  {
    bool Had_to_make_sth_up;
  }
  void Serialize_Texture(Save_t& Savegame)
  { /* Just needed some usecase*/ }
  void Serialize_Heightmap(Save_t& Savegame)
  {/*...*/}
  /*and the deserialization methods*/
}
//Would result in
class Map
{
  long Length;
  Image Texture;
  Image Heightmap;
  bool Had_to_make_sth_up;
  void Serialize(Save_t& Savegame)
  {
    Serialize<long>(Length);
     /*Serialize doing some stuff with boost::mpl to determine whether
        Serialize should be called or Savegame.Serialize(...) is the right choice */
    Serialize_Texture(Savegame);
    Serialize_Heightmap(Savegame);
  }
  void Deserialize(Save_t& Savegame)
  { /*...*/ }
}

Maybe the tool could also create the special method’s definitions when code is appended in curly braces after each member.

It looks uglier than I thought, maybe I will have to rethink this solution.
My worst concern is that it will be pretty hard to get the member’s name from the definition, when attributes (such as __attribute__((...))) and forms of definition like templates or inner classes are considered.

Also, I think I had imagined something better in terms of readability, but this could be an illusion as well.

So, I will try to rely on TMP and maybe the preprocessor, this will make things probably easier, considering that the compiler knows how to parse C++ without additional work.
Hopefully, it won’t tenfold compiling times …


So, now I will have to install git, a compiler and all the necessary depedencies the code has, then I will write a few test cases (also in order to get back to C++, seems like I got a bit out of practice).
Frankly, I just don’t feel like I’m already good enough (again) to contribute directly, not having programmed in C++ for approximately half a year, I think.

With kind regards,
Fast2

[hoijui]

as your understanding of the problem is converging towards reality, your idea for the solution is converging (somewhat slower) towards the solution we already have. it is fail, due to multiple reasons, all (or at least most/the important ones) of which have already been mentioned here. .. it makes no sense for you to try doing this. your solution will have the same flaws, and due to the nature of the flaws, it makes no sense to include it in the main repo (it needs constant maintenance, which is not worth it).

as already said, if you want to contribute, help making spring code nicer, or implement other stuff.. that would be very welcome!

[Fast2]

hoijui: Maybe later on, but for now, it is some inner force pushing me to this solution.
I already felt that it is converging, but this is probably due to the temporal distance between my idea and me posting it.
I will try to reread my first posts and also rethink this issue in order to get back to what I originally thought.
So far, there is one difference which remained: I say that saving and loading should recurse through the whole hierarchy, whereas you have a centralized function using some custom reflection solution, at least according to what I inferred from your posts. (I still didn’t look at the code, it took me three hours to get Git and MinGW (TDM) nearly working, I guess I will finish tomorrow.)

The next days, I plan to get back into C++ (syntax should be ok, but there are going be a lot of things to remember).
Concurrently, I will read about boost::mpl, boost::serialize and maybe boost::preprocessor, then write a few example programs, looking for pitfalls.
I planned to learn how to use these boost-libs anyway somewhen in the future, so why not now? :)
My rough estimate is that I will be able to begin rewriting the save-system next year. (At first, this was meant to be a joke, but now, it seems to be a sane guess …)

Just wanted to inform you (and to reply to hoijui).

With kind regards,
Fast2