Encoding of string data in Spring?

[AF]

Just because the King has no intention of going out and ploughing the fields himself, doesn't mean acquiring food to feed his kingdom is not a priority

[hoijui]

with this analogy, and engine devs being the king, it would mean that you as a common citizen should do the food gathering.
that is in the interest of the king, as long as you do it according to his rules (no hacks), or if you are ok with being in war with the king (changes will not get into spring main repo).

[AF]

There's a difference between saying somethign is important, and doing it. Just because you personally won't do the latter, doesn't make the former impossible.

It's a problem we should be mindful of, and at least have a solution for when someone who can be arsed comes along.

[Anarchid]

at least have a solution for when someone who can be arsed comes along.

Having some kind of a plan would definitely help.

[Tobi]

Some plan:

• Add Unicode-aware font renderer, that decodes std::strings as UTF-8 (or other Unicode encoding, but UTF-8 seems best).
• Test using increasingly non-latin1 fonts. Look for: 1) incorrect sorting of unicode characters; 2) equal, but differently normalized strings not considered equal; 3) incorrect string splitting/parsing/tokenizing; 4) incorrect/missing reencoding/decoding at external interfaces; 5) maybe more?
• Evaluate test to see if it is useful/necessary to change std::string to an Unicode-aware string class.
• Fix reencoding/decoding bugs at external interfaces as they pop up.

Alternative plan (better, because it relies on type errors on undesired/incorrect conversions):

• Add Unicode-aware font renderer.
• Change std::string to an Unicode aware string class without implicit conversion from/to std::string or char*, at every place where Unicode is desired.
• Make font renderer take ONLY Unicode aware strings as input, to force Unicode support in all on-screen text.
• Fix the millions of compile errors that result, by introducing proper reencoding/decoding at all interfaces.

In both cases the contract with Lua scripts would be that Lua strings contain UTF-8 encoded strings. That way old scripts containing 7-bit ASCII continue to work, while Unicode chars can be added seamlessly.

Lua string funcs might need to support UTF-8 too, should see how other projects do this.