2019-08-20 12:29 CEST

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0005189Spring engineGeneralpublic2018-05-03 12:46
Reporterabma 
Assigned Toabma 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusresolvedResolutionwon't fix 
Product Version 
Target VersionFixed in Version 
Summary0005189: deprecate maps which use tdf and add/find a conversion utility to newer version(s)
Description.tdf should be deprecated and removed


also see comments at https://github.com/spring/spring/pull/242
TagsNo tags attached.
Checked infolog.txt for lua Errors
Attached Files

-Relationships
+Relationships

-Notes

~0016129

abma (administrator)

Last edited: 2016-03-30 02:38

View 2 revisions

as we have a central distribution system (springfiles) this shouldn't be this hard to accomplish, only thing is:

what needs to be done to get rid of .tdf / convert to lua?


with some "how to convert a tdf to lua" warning message in infolog.txt we could step forward here imo.

~0016130

silentwings (reporter)

Last edited: 2016-03-30 10:15

View 5 revisions

The maps would have to *all* be recompiled. In some cases this is not at all easy, and with hundreds of maps even covering the easy cases would involve a lot of work that probably nobody is willing to do.

~0016132

user744

To get an estimate how many maps this affects:
In my maps\ folder 69 out of 73 maps had .smd files.
(The exceptions were: apophis_v2_2.sd7, codar_1.6.sd7, glacies_1.3, lonely_oasis_v1.sd7)

> The maps would have to *all* be recompiled.
Just rezipped.

Two possible ways:
a) convert every .smd file into a mapinfo.lua
b) Have one mapinfo.lua that reads the map's .smd file. Add that mapinfo.lua file to each archive.

For warning message in infolog I see no use, converting would have to be automatic for all maps.
It is simply too many maps.

How to handle existance of new & converted versions? Names, filenames?

Imo it is way too late to get rid of tdf format. Even if all maps get converted to mapinfo.lua, then many maps still contain tdf files for units/features/effects. I thought tdf is already 'gone' from engine anyway and parsed through helper-lua files? What is to gain?

~0016133

jK (developer)

I don't see any win in this

~0016134

silentwings (reporter)

Last edited: 2016-03-30 13:27

View 3 revisions

> Just rezipped.
Checksum changes -> sanity (and likely some infrastructure too) requires a new version/name -> recompile.

~0016135

user744

new version/name does not require recompile.

~0016136

silentwings (reporter)

Last edited: 2016-03-30 13:56

View 2 revisions

"sanity"

(the alternative is encouraging situation where maps have a mapinfo.lua claiming a different version number for the map than the smd/smt filename -> no)

~0016137

user744

>situation where maps have a mapinfo.lua claiming a different version number for the map than the smd/smt filename
I doubt any infrastructure gets mapname from smt. That would be foolish.
To outside world only mapinfo.lua matters. Everything else is mapper's own problem/busines. Maybe in mapinfo.lua it says "Deltasiege v3" but when all all the changes were to something not-smt-related (like features, startpositions, scripts,..) then why recompile?

> I don't see any win in this
me neither. only problems.

~0016138

abma (administrator)

> I don't see any win in this

less code. also maps will be in a more standard format which allows faster parsing of metadata.

> Checksum changes -> sanity (and likely some infrastructure too) requires a new version/name -> recompile.

yes, a new version of the map must be created to handle this.

~0016139

Beherith (reporter)

I highly disagree with deprecating old maps, there are many not even on springfiles. Recompiling them all is not trivial.

~0016140

abma (administrator)

> there are many not even on springfiles

thats an other issue.

> Recompiling them all is not trivial.

i assumed that the .tdf (.smd?) files just needs to be "converted" to a mapinfo.lua. is that wrong / or is sth. missing?

~0016142

Beherith (reporter)

Forcing every client to trash their entire map pool and redownload it (Hey I already have that map! Why do you mean I have to redl DSD?) is a waste.

I agree with the previous sentiment of not adding new mapping features to the tdf's (like detail normal texture splatting) as a way of forcing adoption for new maps.

~0016143

abma (administrator)

> Forcing every client to trash their entire map pool and redownload it (Hey I already have that map! Why do you mean I have to redl DSD?) is a waste.

"every" ... "entire"... calm down please! not all maps are affected / needs to be redownloaded. also it would be a migration, meaning new maps will be created and spring will throw a deprecated warning and then at some point the newer version must be used.

also there are a lot of maps spread arround which are compressed solid which makes loading metadata from it very slow -> this should be changed as well.

also if there is a better upgrade process of maps established, some maps could be fixed. several maps throw errors/warnings on load.

~0016144

silentwings (reporter)

Last edited: 2016-03-31 16:59

View 7 revisions

> i assumed that the .tdf (.smd?) files just needs to be "converted" to a mapinfo.lua. is that wrong / or is sth. missing?
See https://springrts.com/mantis/view.php?id=5189#c16134.

>less code.
Exactly how much code is gained here? Afaics the only saving is 200 lines of fhttps://github.com/spring/spring/blob/5c05652c57a6e2fb5545d2121fcad076698d14e3/cont/base/maphelper/maphelper/parse_tdf_map.lua so I can't see any argument here. There are *much* worse things in basecontent.

> not all maps are affected / needs to be redownloaded.
The vast majority are, I think.

> also it would be a migration, meaning new maps will be created and spring will throw a deprecated warning and then at some point the newer version must be used.
Even via the lazy/hacky option of addign mapinfo.lua and not recompiling its still a vast amount of work. Has anybody actually offered to do it?

> also there are a lot of maps spread arround which are compressed solid which makes loading metadata from it very slow -> this should be changed
A different issue affecting a very small number of maps that is probably worth fixing but somebody has to actually want to do the work...

~0016145

abma (administrator)

> Exactly how much code is gained here? Afaics the only saving is 200 lines of https://github.com/spring/spring/blob/5c05652c57a6e2fb5545d2121fcad076698d14e3/cont/base/maphelper/maphelper/parse_tdf_map.lua [^] so I can't see any argument here. There are *much* worse things in basecontent.

there is several code at springfiles.com / in springlobby / ... to workarround inconsistence.

> See https://springrts.com/mantis/view.php?id=5189#c16134. [^]

whats meant with "recompile"?

~0016146

abma (administrator)

forget it, i'm really demotivated now, thanks.

have fun with:

- maintaining docs of an old format
- having old and broken maps forever
- have no sanity checks after map uploads
- dead infrastructure / no development at all

~0016148

abma (administrator)

https://springrts.com/phpbb/viewtopic.php?f=13&t=34643

~0016154

user744

Last edited: 2016-04-03 12:32

View 4 revisions

(1) "re-zip map:"
Parse the .smd file, turn into into lua-format, add a mapinfo.lua file, delete the .smd file.
Or: Add a mapinfo.lua file that interprets the existing .smd file.

(2) "re-compile map":
Converting the map's smt/smf mapfiles into editable files with tools such mapdeconv.exe as mapconvNG.
Then re-compile those files back into spring-map-format, possible with better settings or after editing the source files.
Decompiling a map into texture heightmap16bit.raw and texture.bmp takes a few seconds.
Compiling image files into springformat takes long, potentially minutes.
Some maps are tricky to compile because it can use lots of RAM or require certain mapconv version.
I do not know if the decompile/recompress process causes loss of quality.
I do not know whether it is possible to directly manipulate mapfiles to speed this up.


The difference between re-zip VS re-compile is huge, hence the discussion about it.
(1) Is basically editing a few textfiles in a ziparchive and relatively simple and error-proof.
(2) Is less trivial, takes longer, more can go wrong, is harder to test. (Basically a human must load the map in spring and inspect it visually)


(2) Only makes sense if:
a) If the mapfiles are in some way broken:
-"scanlines bug"
-bad heightmap terrassing
Silentwings argues re-compiling a map is always nessecary for name/checksum/sanity reasons.
But this is not true: Many new map-versions have been released that were not re-recompiled.
Change of startpositions in mapinfo.lua or fixes to Lua-script does not require to re-compile the mapfiles.
Re-compiling perfectly good mapfiles is NOT nessecary and imo even risks to break them.


> not all maps are affected / needs to be redownloaded.
> The vast majority are, I think.
> "every" ... "entire"... calm down please!

"Many! Few! Many!" - lol. Get some numbers instead?
My estimate is https://springrts.com/mantis/view.php?id=5189#c16132 and I think 95% of maps would be affected and in need of 'fixing.'
springfiles has 0000709:0001500 maps, so that are lots of maps!
Both re-zipping or re-compiling would absolutely have to be done by script, by hand is unfeasible.


Simply stopping to recognize .smd maps in the hope that community fixes the most popular maps is imo unrealistic.
Yes, if a map is worth fixing, it sometimes gets fixed.
However adjusting hundreds of maps because an engine-change (that few players will understand the need of) broke perfectly playable maps seems less likely to work.
Well maybe some volunteer might do all maps by script.


> some maps could be fixed. several maps throw errors/warnings on load.

If whole mappool is to be touched by a script it seems good chance to fix as many things as possible in one go.
However:
Maps can be broken for hundreds of possible reasons. First question is whhich problems are recongizeable by script.
Stuff as bad lightning-settings might be fixable, bad description might be fixable. (Lots of maps have its description-text copied from SmallDivide, because mappers used it as base)
Solid-zipped archive might be fixable. Is scanline-bug recognizeable beside a from human eye?
Not only the .smd of maps is old format, included units/features are often in .fbi or .tdf format too. (Seems inconsequent not to convert those too. And of

course in mods too. But where to stop?)
Bad features, bad metalmap, bad startpositions, bad included scripts, bad heightmap - not fixable by scripts.

For some time the original maps would have be kept together with the new versions, until it is clear all has worked.


> - maintaining docs of an old format
As far I see it is already not maintained anymore, it is all about mapinfo.lua now.
People continue to use .smd because one can take existing maps as base. Lua is obviously better but often the advantages do not matter from mapper's view.

> - having old and broken maps forever
Tons of old&broken maps exist but many are testversions or old versions. They do not matter because autohosts update their maplists.
Potentially lots of piskspace could be free by deleting unused version but imo is a different topic.

> - have no sanity checks after map uploads
What kind of checks?
Mapper tests their map in spring.exe : If that works then it is good enough to upload?

> - dead infrastructure / no development at all
Appearently so far no nobody could understand why parse_tdf_map.lua (or whatever) would be such big blocker, so needs more explaining.

Sorry for tl;dr but if there is different or no understanding of things like "recompile map" and its scale, then discussing it goes nowhere.

~0016155

jK (developer)

>> I don't see any win in this

> less code. also maps will be in a more standard format which allows faster parsing of metadata.

not true, the parsing of the map's tdf happens in lua



when you want to reduce code get rid of script.txt parsing and replace it with some lua
+Notes

-Issue History
Date Modified Username Field Change
2016-03-30 02:34 abma New Issue
2016-03-30 02:37 abma Note Added: 0016129
2016-03-30 02:38 abma Note Edited: 0016129 View Revisions
2016-03-30 02:59 abma Description Updated View Revisions
2016-03-30 09:10 silentwings Note Added: 0016130
2016-03-30 09:11 silentwings Note Edited: 0016130 View Revisions
2016-03-30 09:11 silentwings Note Edited: 0016130 View Revisions
2016-03-30 09:17 silentwings Note Edited: 0016130 View Revisions
2016-03-30 10:15 silentwings Note Edited: 0016130 View Revisions
2016-03-30 12:41 user744 Note Added: 0016132
2016-03-30 12:53 jK Note Added: 0016133
2016-03-30 13:23 silentwings Note Added: 0016134
2016-03-30 13:26 silentwings Note Edited: 0016134 View Revisions
2016-03-30 13:27 silentwings Note Edited: 0016134 View Revisions
2016-03-30 13:42 user744 Note Added: 0016135
2016-03-30 13:55 silentwings Note Added: 0016136
2016-03-30 13:56 silentwings Note Edited: 0016136 View Revisions
2016-03-30 14:36 user744 Note Added: 0016137
2016-03-30 15:45 abma Note Added: 0016138
2016-03-31 10:54 Beherith Note Added: 0016139
2016-03-31 11:06 abma Note Added: 0016140
2016-03-31 13:57 hokomoko Summary deprecate maps which use tdf and add/find a conversation utility to newer version(s) => deprecate maps which use tdf and add/find a convertion utility to newer version(s)
2016-03-31 13:57 hokomoko Summary deprecate maps which use tdf and add/find a convertion utility to newer version(s) => deprecate maps which use tdf and add/find a conversion utility to newer version(s)
2016-03-31 14:01 Beherith Note Added: 0016142
2016-03-31 14:15 abma Note Added: 0016143
2016-03-31 16:51 silentwings Note Added: 0016144
2016-03-31 16:54 silentwings Note Edited: 0016144 View Revisions
2016-03-31 16:55 silentwings Note Edited: 0016144 View Revisions
2016-03-31 16:57 silentwings Note Edited: 0016144 View Revisions
2016-03-31 16:57 silentwings Note Edited: 0016144 View Revisions
2016-03-31 16:58 silentwings Note Edited: 0016144 View Revisions
2016-03-31 16:59 silentwings Note Edited: 0016144 View Revisions
2016-04-01 06:34 abma Note Added: 0016145
2016-04-01 07:39 abma Note Added: 0016146
2016-04-01 07:39 abma Status new => closed
2016-04-01 07:39 abma Assigned To => abma
2016-04-01 07:39 abma Resolution open => not fixable
2016-04-01 11:42 abma Note Added: 0016148
2016-04-01 11:42 abma Status closed => new
2016-04-03 12:25 user744 Note Added: 0016154
2016-04-03 12:25 user744 Note Edited: 0016154 View Revisions
2016-04-03 12:27 user744 Note Edited: 0016154 View Revisions
2016-04-03 12:32 user744 Note Edited: 0016154 View Revisions
2016-04-03 15:03 jK Note Added: 0016155
2018-05-03 12:46 abma Status new => resolved
2018-05-03 12:46 abma Resolution not fixable => won't fix
+Issue History