| View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||
| ID | Project | Category | View Status | Date Submitted | Last Update | ||||
|---|---|---|---|---|---|---|---|---|---|
| 0005189 | Spring engine | General | public | 2016-03-30 02:34 | 2018-05-03 12:46 | ||||
| Reporter | abma | ||||||||
| Assigned To | abma | ||||||||
| Priority | normal | Severity | minor | Reproducibility | have not tried | ||||
| Status | resolved | Resolution | won't fix | ||||||
| Product Version | |||||||||
| Target Version | Fixed in Version | ||||||||
| Summary | 0005189: deprecate maps which use tdf and add/find a conversion utility to newer version(s) | ||||||||
| Description | .tdf should be deprecated and removed also see comments at https://github.com/spring/spring/pull/242 | ||||||||
| Tags | No tags attached. | ||||||||
| Checked infolog.txt for Errors | |||||||||
| Attached Files |
| ||||||||
 Relationships |
|
 Relationships |
| Notes |
|
|
abma (administrator) 2016-03-30 02:37 Last edited: 2016-03-30 02:38 |
as we have a central distribution system (springfiles) this shouldn't be this hard to accomplish, only thing is: what needs to be done to get rid of .tdf / convert to lua? with some "how to convert a tdf to lua" warning message in infolog.txt we could step forward here imo. |
|
silentwings (reporter) 2016-03-30 09:10 Last edited: 2016-03-30 10:15 |
The maps would have to *all* be recompiled. In some cases this is not at all easy, and with hundreds of maps even covering the easy cases would involve a lot of work that probably nobody is willing to do. |
|
2016-03-30 12:41 |
To get an estimate how many maps this affects: In my maps\ folder 69 out of 73 maps had .smd files. (The exceptions were: apophis_v2_2.sd7, codar_1.6.sd7, glacies_1.3, lonely_oasis_v1.sd7) > The maps would have to *all* be recompiled. Just rezipped. Two possible ways: a) convert every .smd file into a mapinfo.lua b) Have one mapinfo.lua that reads the map's .smd file. Add that mapinfo.lua file to each archive. For warning message in infolog I see no use, converting would have to be automatic for all maps. It is simply too many maps. How to handle existance of new & converted versions? Names, filenames? Imo it is way too late to get rid of tdf format. Even if all maps get converted to mapinfo.lua, then many maps still contain tdf files for units/features/effects. I thought tdf is already 'gone' from engine anyway and parsed through helper-lua files? What is to gain? |
|
jK (developer) 2016-03-30 12:53 |
I don't see any win in this |
|
silentwings (reporter) 2016-03-30 13:23 Last edited: 2016-03-30 13:27 |
> Just rezipped. Checksum changes -> sanity (and likely some infrastructure too) requires a new version/name -> recompile. |
|
2016-03-30 13:42 |
new version/name does not require recompile. |
|
silentwings (reporter) 2016-03-30 13:55 Last edited: 2016-03-30 13:56 |
"sanity" (the alternative is encouraging situation where maps have a mapinfo.lua claiming a different version number for the map than the smd/smt filename -> no) |
|
2016-03-30 14:36 |
>situation where maps have a mapinfo.lua claiming a different version number for the map than the smd/smt filename I doubt any infrastructure gets mapname from smt. That would be foolish. To outside world only mapinfo.lua matters. Everything else is mapper's own problem/busines. Maybe in mapinfo.lua it says "Deltasiege v3" but when all all the changes were to something not-smt-related (like features, startpositions, scripts,..) then why recompile? > I don't see any win in this me neither. only problems. |
|
abma (administrator) 2016-03-30 15:45 |
> I don't see any win in this less code. also maps will be in a more standard format which allows faster parsing of metadata. > Checksum changes -> sanity (and likely some infrastructure too) requires a new version/name -> recompile. yes, a new version of the map must be created to handle this. |
|
Beherith (reporter) 2016-03-31 10:54 |
I highly disagree with deprecating old maps, there are many not even on springfiles. Recompiling them all is not trivial. |
|
abma (administrator) 2016-03-31 11:06 |
> there are many not even on springfiles thats an other issue. > Recompiling them all is not trivial. i assumed that the .tdf (.smd?) files just needs to be "converted" to a mapinfo.lua. is that wrong / or is sth. missing? |
|
Beherith (reporter) 2016-03-31 14:01 |
Forcing every client to trash their entire map pool and redownload it (Hey I already have that map! Why do you mean I have to redl DSD?) is a waste. I agree with the previous sentiment of not adding new mapping features to the tdf's (like detail normal texture splatting) as a way of forcing adoption for new maps. |
|
abma (administrator) 2016-03-31 14:15 |
> Forcing every client to trash their entire map pool and redownload it (Hey I already have that map! Why do you mean I have to redl DSD?) is a waste. "every" ... "entire"... calm down please! not all maps are affected / needs to be redownloaded. also it would be a migration, meaning new maps will be created and spring will throw a deprecated warning and then at some point the newer version must be used. also there are a lot of maps spread arround which are compressed solid which makes loading metadata from it very slow -> this should be changed as well. also if there is a better upgrade process of maps established, some maps could be fixed. several maps throw errors/warnings on load. |
|
silentwings (reporter) 2016-03-31 16:51 Last edited: 2016-03-31 16:59 |
> i assumed that the .tdf (.smd?) files just needs to be "converted" to a mapinfo.lua. is that wrong / or is sth. missing? See https://springrts.com/mantis/view.php?id=5189#c16134. >less code. Exactly how much code is gained here? Afaics the only saving is 200 lines of fhttps://github.com/spring/spring/blob/5c05652c57a6e2fb5545d2121fcad076698d14e3/cont/base/maphelper/maphelper/parse_tdf_map.lua so I can't see any argument here. There are *much* worse things in basecontent. > not all maps are affected / needs to be redownloaded. The vast majority are, I think. > also it would be a migration, meaning new maps will be created and spring will throw a deprecated warning and then at some point the newer version must be used. Even via the lazy/hacky option of addign mapinfo.lua and not recompiling its still a vast amount of work. Has anybody actually offered to do it? > also there are a lot of maps spread arround which are compressed solid which makes loading metadata from it very slow -> this should be changed A different issue affecting a very small number of maps that is probably worth fixing but somebody has to actually want to do the work... |
|
abma (administrator) 2016-04-01 06:34 |
> Exactly how much code is gained here? Afaics the only saving is 200 lines of https://github.com/spring/spring/blob/5c05652c57a6e2fb5545d2121fcad076698d14e3/cont/base/maphelper/maphelper/parse_tdf_map.lua [^] so I can't see any argument here. There are *much* worse things in basecontent. there is several code at springfiles.com / in springlobby / ... to workarround inconsistence. > See https://springrts.com/mantis/view.php?id=5189#c16134. [^] whats meant with "recompile"? |
|
abma (administrator) 2016-04-01 07:39 |
forget it, i'm really demotivated now, thanks. have fun with: - maintaining docs of an old format - having old and broken maps forever - have no sanity checks after map uploads - dead infrastructure / no development at all |
|
abma (administrator) 2016-04-01 11:42 |
https://springrts.com/phpbb/viewtopic.php?f=13&t=34643 |
|
2016-04-03 12:25 Last edited: 2016-04-03 12:32 |
(1) "re-zip map:" Parse the .smd file, turn into into lua-format, add a mapinfo.lua file, delete the .smd file. Or: Add a mapinfo.lua file that interprets the existing .smd file. (2) "re-compile map": Converting the map's smt/smf mapfiles into editable files with tools such mapdeconv.exe as mapconvNG. Then re-compile those files back into spring-map-format, possible with better settings or after editing the source files. Decompiling a map into texture heightmap16bit.raw and texture.bmp takes a few seconds. Compiling image files into springformat takes long, potentially minutes. Some maps are tricky to compile because it can use lots of RAM or require certain mapconv version. I do not know if the decompile/recompress process causes loss of quality. I do not know whether it is possible to directly manipulate mapfiles to speed this up. The difference between re-zip VS re-compile is huge, hence the discussion about it. (1) Is basically editing a few textfiles in a ziparchive and relatively simple and error-proof. (2) Is less trivial, takes longer, more can go wrong, is harder to test. (Basically a human must load the map in spring and inspect it visually) (2) Only makes sense if: a) If the mapfiles are in some way broken: -"scanlines bug" -bad heightmap terrassing Silentwings argues re-compiling a map is always nessecary for name/checksum/sanity reasons. But this is not true: Many new map-versions have been released that were not re-recompiled. Change of startpositions in mapinfo.lua or fixes to Lua-script does not require to re-compile the mapfiles. Re-compiling perfectly good mapfiles is NOT nessecary and imo even risks to break them. > not all maps are affected / needs to be redownloaded. > The vast majority are, I think. > "every" ... "entire"... calm down please! "Many! Few! Many!" - lol. Get some numbers instead? My estimate is https://springrts.com/mantis/view.php?id=5189#c16132 and I think 95% of maps would be affected and in need of 'fixing.' springfiles has 0000709:0001500 maps, so that are lots of maps! Both re-zipping or re-compiling would absolutely have to be done by script, by hand is unfeasible. Simply stopping to recognize .smd maps in the hope that community fixes the most popular maps is imo unrealistic. Yes, if a map is worth fixing, it sometimes gets fixed. However adjusting hundreds of maps because an engine-change (that few players will understand the need of) broke perfectly playable maps seems less likely to work. Well maybe some volunteer might do all maps by script. > some maps could be fixed. several maps throw errors/warnings on load. If whole mappool is to be touched by a script it seems good chance to fix as many things as possible in one go. However: Maps can be broken for hundreds of possible reasons. First question is whhich problems are recongizeable by script. Stuff as bad lightning-settings might be fixable, bad description might be fixable. (Lots of maps have its description-text copied from SmallDivide, because mappers used it as base) Solid-zipped archive might be fixable. Is scanline-bug recognizeable beside a from human eye? Not only the .smd of maps is old format, included units/features are often in .fbi or .tdf format too. (Seems inconsequent not to convert those too. And of course in mods too. But where to stop?) Bad features, bad metalmap, bad startpositions, bad included scripts, bad heightmap - not fixable by scripts. For some time the original maps would have be kept together with the new versions, until it is clear all has worked. > - maintaining docs of an old format As far I see it is already not maintained anymore, it is all about mapinfo.lua now. People continue to use .smd because one can take existing maps as base. Lua is obviously better but often the advantages do not matter from mapper's view. > - having old and broken maps forever Tons of old&broken maps exist but many are testversions or old versions. They do not matter because autohosts update their maplists. Potentially lots of piskspace could be free by deleting unused version but imo is a different topic. > - have no sanity checks after map uploads What kind of checks? Mapper tests their map in spring.exe : If that works then it is good enough to upload? > - dead infrastructure / no development at all Appearently so far no nobody could understand why parse_tdf_map.lua (or whatever) would be such big blocker, so needs more explaining. Sorry for tl;dr but if there is different or no understanding of things like "recompile map" and its scale, then discussing it goes nowhere. |
|
jK (developer) 2016-04-03 15:03 |
>> I don't see any win in this > less code. also maps will be in a more standard format which allows faster parsing of metadata. not true, the parsing of the map's tdf happens in lua when you want to reduce code get rid of script.txt parsing and replace it with some lua |
| Notes |
| Issue History |
|||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-03-30 02:34 | abma | New Issue | |
| 2016-03-30 02:37 | abma | Note Added: 0016129 | |
| 2016-03-30 02:38 | abma | Note Edited: 0016129 | View Revisions |
| 2016-03-30 02:59 | abma | Description Updated | View Revisions |
| 2016-03-30 09:10 | silentwings | Note Added: 0016130 | |
| 2016-03-30 09:11 | silentwings | Note Edited: 0016130 | View Revisions |
| 2016-03-30 09:11 | silentwings | Note Edited: 0016130 | View Revisions |
| 2016-03-30 09:17 | silentwings | Note Edited: 0016130 | View Revisions |
| 2016-03-30 10:15 | silentwings | Note Edited: 0016130 | View Revisions |
| 2016-03-30 12:41 |
|
Note Added: 0016132 | |
| 2016-03-30 12:53 | jK | Note Added: 0016133 | |
| 2016-03-30 13:23 | silentwings | Note Added: 0016134 | |
| 2016-03-30 13:26 | silentwings | Note Edited: 0016134 | View Revisions |
| 2016-03-30 13:27 | silentwings | Note Edited: 0016134 | View Revisions |
| 2016-03-30 13:42 |
|
Note Added: 0016135 | |
| 2016-03-30 13:55 | silentwings | Note Added: 0016136 | |
| 2016-03-30 13:56 | silentwings | Note Edited: 0016136 | View Revisions |
| 2016-03-30 14:36 |
|
Note Added: 0016137 | |
| 2016-03-30 15:45 | abma | Note Added: 0016138 | |
| 2016-03-31 10:54 | Beherith | Note Added: 0016139 | |
| 2016-03-31 11:06 | abma | Note Added: 0016140 | |
| 2016-03-31 13:57 | hokomoko | Summary | deprecate maps which use tdf and add/find a conversation utility to newer version(s) => deprecate maps which use tdf and add/find a convertion utility to newer version(s) |
| 2016-03-31 13:57 | hokomoko | Summary | deprecate maps which use tdf and add/find a convertion utility to newer version(s) => deprecate maps which use tdf and add/find a conversion utility to newer version(s) |
| 2016-03-31 14:01 | Beherith | Note Added: 0016142 | |
| 2016-03-31 14:15 | abma | Note Added: 0016143 | |
| 2016-03-31 16:51 | silentwings | Note Added: 0016144 | |
| 2016-03-31 16:54 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-03-31 16:55 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-03-31 16:57 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-03-31 16:57 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-03-31 16:58 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-03-31 16:59 | silentwings | Note Edited: 0016144 | View Revisions |
| 2016-04-01 06:34 | abma | Note Added: 0016145 | |
| 2016-04-01 07:39 | abma | Note Added: 0016146 | |
| 2016-04-01 07:39 | abma | Status | new => closed |
| 2016-04-01 07:39 | abma | Assigned To | => abma |
| 2016-04-01 07:39 | abma | Resolution | open => not fixable |
| 2016-04-01 11:42 | abma | Note Added: 0016148 | |
| 2016-04-01 11:42 | abma | Status | closed => new |
| 2016-04-03 12:25 |
|
Note Added: 0016154 | |
| 2016-04-03 12:25 |
|
Note Edited: 0016154 | View Revisions |
| 2016-04-03 12:27 |
|
Note Edited: 0016154 | View Revisions |
| 2016-04-03 12:32 |
|
Note Edited: 0016154 | View Revisions |
| 2016-04-03 15:03 | jK | Note Added: 0016155 | |
| 2018-05-03 12:46 | abma | Status | new => resolved |
| 2018-05-03 12:46 | abma | Resolution | not fixable => won't fix |
| Issue History |