Hmm.

Well if Allegro isn't going to be able to print those kinds of characters anyway, then I guess there's no hurry for me to support them in my text box. Ultimately I'd like for all this stuff to work properly, but I don't want to spend too long on this given that I don't intend to use non-ascii characters anyway.

For the time being, I think I'll just make placeholder functions for 'next character' and 'previous character', and have those functions simply look for the first unit of the code-point. (I don't yet know how to do that either, but I assume there's some particular set of bits which signal which unit is the first unit; and I'm pretty sure al_draw_text will have that!)

Thanks for the info.

I think even with UTF8, two multi byte sequences can be combined into one composed character.

That is what I thought I said. Two unicode "characters" can actually render as a single visible character. So you can't really just jump around like you suggested.

No, unicode specifies separate characters and modifiers like umlauts, where they are specified separately in the unicode string, but are composed together when drawing.

Not if you implement it properly. As soon as you encode the Unicode it becomes implementation specific. But you can convert between different encodings. E.g. if you are using UTF-16 on Windows and you want to use Allegro, at some point you'll have to convert between encodings. It is still all Unicode, just stored in a different format.

If you use all of the standard functions for wide strings your std::wstring code will be portable. If you save wide string data on Linux and try to load it on Windows it will fail because the encodings are different. If you load the data as UTF-32 (Linux) and convert to UTF-16 (Windows) you will still be able to use it.

If all of your code uses std::wstring and the w-string wide string functions, it will all be portable. If you say it "won't fit", this implies that you are setting hard limits on the size of data items (e.g. in bytes). You shouldn't do this in any encoding, you should calculate that using the coding API, in order to make the code and data portable.

It's plain unicode value of a character code point. It's not encoded. If you want it encoded you should probably use byte string and special handling functions.

http://www.joelonsoftware.com/articles/Unicode.html

The way I see it, the unicode characters must be encoded in order to exist in computer memory at all. The programmer must choose what kind of encoding to use, and what kind of data structure to store the encoded string in. When I said "it won't fit", what I meant was that a 32-bit unit won't fit into a char16_t, and so if I used UTF-32 encoding and stored it in a std::wstring, that would work on Linux but not on Windows.

My point is that I can read UTF-32 straight into a std::wstring iff wchar_t is 32-bits. The 32-bit units of UTF-32 won't fit inside a 16-bit wchar_t.

I think this is the key to understand what you are talking about. As far as I understand, everything else you've said is stuff that I already knew, but here I'm not sure what you mean by "the w-string wide string functions". What functions are you referring to? (Perhaps it's related to codecvt which Elias mentioned?)

Well, in this thread there is some misinformation and a significant amount of talking at cross purposes, but I feel like I've learned a fair bit. For example, I've learned that the maximum number of bytes per character is 4 for any unicode code-point and that some printable characters are actually composed of multiple unicode code-points. And most recently, Elias mentioned something called codecvt, which I hadn't heard of before. I'm reading about it now in the C++11 Standard document. A lot of the information in the document is pretty dense, but from what I can tell codecvt sounds userful, and very relevant to what we're talking about. Check this out:

So, based on that here's my understanding of the situation: std::wstring wide_string = L"Hello"; will store the string in an implementation defined encoding which uses wchar_t. My concern throughout this thread has been that since we don't necessarily know what the implementation defined encoding is, it's difficult to work with it in a platform independent way. For example I can't use al_ustr_new_from_utf16, because wchar_t may or may not be uint16_t; and since I don't know what the encoding is, I don't know how to traverse the wchar_t array to find the starts of characters and things like that which I need for my text box widget that I mentioned. For these reasons, I claimed that it's better to simply avoid using wchar_t and instead use something like char16_t (or whatever) so that one could be sure of which type of unicode encoding is being used.

However, now that I've read that codecvt<wchar_t,char,mbstate_t> will convert between the mystery wchar_t encoding and UTF-8, I see that it is at least possible to right portable code that uses wchar_t. It still seems to me that it's better to pick one of the more explicit character sizes, but at least wchar_t is not fatally flawed.