[A5] TTF Fonts not caching correctly?
jmasterx

I'm using Allegro 5.0.5

In the game I'm making, the game's text dynamically resizes and therefore I load and use a lot of fonts.

Each Gui component, when receiving a resize event accesses the font manager and sets its font to the appropriate font.

I load fonts in the constructor of the core class, after Allegro has been initialized and after a display has been created.

#SelectExpand
1 Core::Core(int argc, char *argv[]) 2 :display(NULL), 3 m_fontManager(NULL) 4 { 5 //initialize Allegro 5 6 _initAllegro(); 7 8 //initialize Agui 9 _initAgui(); 10 11 //create the display 12 Vec2 optimalSize = Display::getOptimalResoluton(); 13 display = new Display(optimalSize.getX(),optimalSize.getY()); 14 appIcon = new Sprite("res/appico.png"); 15 display->setIcon(appIcon); 16 17 //init font manager 18 m_fontManager = new GuiFontManager("res/font.ttf",8,32,17); 19 agui::Widget::setGlobalFont(m_fontManager->getDefaultFont()); 20 21 //create the scene manager 22 sceneManager = new SceneManager(settings, m_dynamicUIManager); 23 display->setSceneMessenger(sceneManager); 24 25 } 26... 27 GuiFontManager::GuiFontManager( 28 const std::string& fontPath, int beginRange, int endRange, int defaultSize ) 29 : m_beginRange(beginRange), m_endRange(endRange), m_defaultSize(defaultSize) 30 { 31 for(int i = beginRange; i <= endRange; ++i) 32 { 33 m_fonts.push_back(agui::Font::load(fontPath,i)); 34 } 35 }

My issue is when I get into a scene and start using a bunch of these fonts. Sometimes only part of a word is rendered, and on occasion, a glyph is totally the wrong one. I also tried, after loading a font to render the alphabet and this actually helped, but there were still certain circumstances when the word 'Across' rendered as 'Acro' or 'o'. I've looked through my code for hours trying to find something wrong with it but I cannot seem to find anything wrong.

Is there some sort of precaution to take when loading and using around 40 TTF fonts?

Thanks

MiquelFire

Sounds like an issue I had with my last SpeedHack game, only I loaded only two. In my case, only some computers failed to load the fonts correctly.

jmasterx

On my PC, the same exe and conditions cause different letters to show and different letters not to... I do not know what to do. It is very important for me to be able to have this feature.

Edit:

I can confirm that this only happens with Direct3D, not OpenGL.

Peter Wang

Best provide a test case.

Trent Gamblin

I think I had a similar issue with the D3D driver at one point. I fixed it by making sure bitmaps were a minimum size, I think 16x16. Not sure if it's related. I don't know for certain but I assumed that fix was either done before 5.0 was released or ported back.

jmasterx

Here is an example that produces the problem.

It seems to happen when calling al_get_text_width and loading many fonts.

#SelectExpand
1#include <stdio.h> 2#include <allegro5/allegro.h> 3#include <allegro5/allegro5.h> 4#include <allegro5/allegro_primitives.h> 5#include <allegro5/allegro_font.h> 6#include <allegro5/allegro_ttf.h> 7#include "stdlib.h" 8#include <vector> 9struct Example 10{ 11 double fps; 12 ALLEGRO_CONFIG *config; 13} ex; 14//Globals are bad but it is an example 15std::vector<ALLEGRO_FONT*> m_fonts; 16 17static const char *get_string(const char *key) 18{ 19 const char *v = al_get_config_value(ex.config, "", key); 20 return (v) ? v : key; 21} 22 23static void render(void) 24{ 25 al_clear_to_color(al_map_rgb(0,0,0)); 26 27 int fontSz = al_get_display_height(al_get_current_display()) * 0.03f; 28 if(fontSz >= m_fonts.size()) 29 { 30 fontSz = m_fonts.size() - 1; 31 } 32 33 al_draw_text((m_fonts[fontSz]),al_map_rgb(255,255,255),0,0,0,"ABCDEFGHIJKLMNOPQRSTUVWXYZ"); 34 35 al_get_text_width(m_fonts[6],"ALLEGRO"); 36} 37 38int main(int argc, const char *argv[]) 39{ 40 const char *font_file = "data/DejaVuSans.ttf"; 41 ALLEGRO_DISPLAY *display; 42 ALLEGRO_TIMER *timer; 43 ALLEGRO_EVENT_QUEUE *queue; 44 int redraw = 0; 45 double t = 0; 46 47 if(!al_init()) 48 { 49 return 1; 50 } 51 52 53 if(!al_init_primitives_addon()) 54 { 55 return 1; 56 } 57 58 al_init_font_addon(); 59 60 if(!al_init_ttf_addon()) 61 { 62 return 1; 63 } 64 65 66 al_set_new_display_flags(ALLEGRO_RESIZABLE); 67 display = al_create_display(640, 480); 68 if (!display) { 69 return 1; 70 } 71 al_install_keyboard(); 72 73 //load the fonts 74 for(int i = 8; i < 32; ++i) 75 { 76 m_fonts.push_back(al_load_font(font_file,i,0)); 77 } 78 79 timer = al_create_timer(1.0 / 60); 80 81 queue = al_create_event_queue(); 82 al_register_event_source(queue, al_get_keyboard_event_source()); 83 al_register_event_source(queue, al_get_display_event_source(display)); 84 al_register_event_source(queue, al_get_timer_event_source(timer)); 85 86 al_start_timer(timer); 87 while (true) { 88 ALLEGRO_EVENT event; 89 al_wait_for_event(queue, &event); 90 if (event.type == ALLEGRO_EVENT_DISPLAY_CLOSE) 91 break; 92 if (event.type == ALLEGRO_EVENT_KEY_DOWN && 93 event.keyboard.keycode == ALLEGRO_KEY_ESCAPE) { 94 break; 95 } 96 if (event.type == ALLEGRO_EVENT_TIMER) 97 redraw++; 98 99 if(event.type == ALLEGRO_EVENT_DISPLAY_RESIZE) 100 { 101 al_acknowledge_resize(display); 102 103 } 104 105 while (redraw > 0 && al_is_event_queue_empty(queue)) { 106 double dt; 107 redraw--; 108 109 dt = al_get_time(); 110 render(); 111 dt = al_get_time() - dt; 112 113 t = 0.99 * t + 0.01 * dt; 114 115 ex.fps = 1.0 / t; 116 al_flip_display(); 117 } 118 } 119 //free mem 120 121 return 0; 122} 123 124/* vim: set sts=4 sw=4 et: */

Just keep resizing the window until you see the image I've attached:
{"name":"605475","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/0\/6057c40463519de97417fc003c88ed78.png","w":742,"h":308,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/0\/6057c40463519de97417fc003c88ed78"}605475

It's quite tricky to reproduce. If I draw the alphabet with all the loaded fonts the issue does not occur.

Trent Gamblin

Something to do with display lost/found probably. When you resize a window D3D has to reinit all bitmaps from system memory, including fonts.

jmasterx

Do you think it can be fixed? I do not want to use GL on Windows because:

When it resizes it flickers. Theres no way to address this because I cant get a resize begin / end events and therefore cannot render black while resizing.

And it reduces a lot of compatibility. If a person does not have an OpenGL driver from NVidia or ATI Allegro does not work well.

Also, it seems like it does have to do with resizing. If I launch the Window at 320x240, it does not happen (but in my game it still does, just differently).

Trent Gamblin

I'm sure it can be fixed. Change the fonts to use memory bitmaps. See if that "fixes" it. I know it'll be slow but that'll narrow down the bug.

jmasterx

Yup
Adding:
al_set_new_bitmap_flags(ALLEGRO_MEMORY_BITMAP);

'fixes' the problem.

It also 'fixes' it for my game.

Trent Gamblin

Now try cycling through different sizes without resizing (using video bitmaps). Just have a font_index, increase it each render and wrap it around to zero. See if you have any problems.

jmasterx

Cycling does not have the issue. However, when cycling, I can resize and the issue does not happen.

Trent Gamblin

Yeah, it seems like a threading issue. WM_SIZE events come in on a separate thread than the user thread (the one you write your code on.) So a bitmap "backup" must be performed before the resize takes place, and _flip() should probably be blocked with a mutex until all of this is done.

Thomas Fjellstrom

If the bitmaps are invalidated, just send down the LOST_DISPLAY event and call it a day ;)

Trent Gamblin

That's not required to be handled by Allegro user programs. They can handle it if they choose, and skip all of the automatic stuff in favor of perhaps a little more performance and flexibility. But I think it's too complicated and different from other platforms that it shouldn't be the only way.

jmasterx

Most games I see using D3D that resize do postpone display flipping until resizing is done. This causes the game to also look better when resizing. Instead of a fuzzy stretched version of your output you get the last frame and some black which to me looks more elegant anyways. It just seems to me like it is better not to render while resizing; On all platforms too.

Trent Gamblin

Can you build 5.1 svn and try that? It probably doesn't work, but you'll be ready to test some patches in case I feel like trying to fix this now :D.

jmasterx

Would you happen to have FreeType and some of Allegro's other deps (for MSVC 9)? Otherwise it might take a while. I can compile core Allegro without trouble, but the TTF and other addons need the deps.

Trent Gamblin

I don't have them, but they come with the A5 binaries on this page that Michal Cichon makes.

jmasterx

Wouldn't I still need the headers though? I added the .libs to my PATH environment variable and CMAKE still cannot find them.

MiquelFire

I was going to say use the libs from the binary package of 5.0.5, but I noticed you would need to deal with static compiling to do that...

[edit] Just noticed that the only header included is the Physfs one (assuming an example needs it)

jmasterx

[edit] Just noticed that the only header included is the Physfs one (assuming an example needs it)

Exactly, that's my problem.

Peter Wang

I seem to have this file on my Windows system. FreeType comes with MSVC build stuff in builds/win32 so it should not be that hard to build yourself anyway.

http://libagar.org/FreeType235-win32-i386.zip

Edit: here is a copy of one of my deps directories (I have a few). I do not remember if it works, where all the files came from, how they were built, if anything is missing, or if it is virus infected (I don't use antivirus). Use at your own risk.

http://uploading.com/files/9b797f94/deps.zip/

jmasterx

Has there been any progress on the issue. I do not know if you saw the similar OpenGL issue here as well:
http://www.allegro.cc/forums/thread/609417

(Maybe this thread and the other should be moved to the Development section since I guess it is an allegro bug and not an issue with my code which had been my initial speculation)

Also, I compiled FreeType, but I'm not sure how to get CMAKE to know where it is.

Elias

Easiest way is to drop it (both the include and lib folder) into a folder called "deps" in the build folder.

jmasterx

Thanks Elias, that worked perfectly!

@Trent
I just tried my little example in latest SVN and it still happens. But at least now if you have any patches for this that need testing I'm ready!

Trent Gamblin

I tried to recreate what's happening in the ttf addon but without using the ttf addon. This example works, though I can confirm that yours doesn't. Right now I'm not sure why, perhaps this example may shed a little light though so I'll post it here.

#SelectExpand
1#include <allegro5/allegro.h> 2#include <allegro5/allegro_direct3d.h> 3 4ALLEGRO_BITMAP *pages[50] = { NULL, }; 5 6void check_page(int index) 7{ 8 if (index < 3) index = 3; 9 int w = al_get_bitmap_width(pages[index]); 10 int h = al_get_bitmap_height(pages[index]); 11 for (int y = 0; y+index < h; y += index) { 12 for (int x = 0; x+index < w; x += index) { 13 ALLEGRO_LOCKED_REGION *lr; 14 lr = al_lock_bitmap_region( 15 pages[index], 16 x, y, index, index, 17 ALLEGRO_PIXEL_FORMAT_ARGB_8888, 18 ALLEGRO_LOCK_WRITEONLY 19 ); 20 for (int j = 0; j < index; j++) { 21 if (j == 0 || j == index-1) continue; 22 for (int i = 0; i < index; i++) { 23 if (i == 0 || i == index-1) continue; 24 uint8_t *p = 25 (uint8_t *)lr->data; 26 p += j * lr->pitch; 27 p += i * 4; 28 *((uint32_t *)p) = 0xFF00FF00; 29 } 30 } 31 al_unlock_bitmap(pages[index]); 32 } 33 } 34} 35 36int main(int argc, char **argv) 37{ 38 al_init(); 39 40 al_set_new_display_flags(ALLEGRO_RESIZABLE | ALLEGRO_DIRECT3D); 41 42 ALLEGRO_DISPLAY *display = al_create_display(500, 500); 43 ALLEGRO_EVENT_QUEUE *queue = al_create_event_queue(); 44 al_register_event_source(queue, al_get_display_event_source(display)); 45 46 for (int i = 0; i < 50; i++) { 47 int w = 100 + i * 10; 48 int h = 100 + i * 10; 49 pages[i] = al_create_bitmap(w, h); 50 al_set_target_bitmap(pages[i]); 51 al_clear_to_color(al_map_rgb(0, 0, 0)); 52 } 53 54 al_set_target_backbuffer(display); 55 56 while (true) { 57 while (!al_event_queue_is_empty(queue)) { 58 ALLEGRO_EVENT e; 59 al_get_next_event(queue, &e); 60 if (e.type == ALLEGRO_EVENT_DISPLAY_RESIZE) { 61 al_acknowledge_resize(display); 62 } 63 else if (e.type == ALLEGRO_EVENT_DISPLAY_CLOSE) { 64 return 0; 65 } 66 } 67 68 float window_width = al_get_display_width(display); 69 window_width -= 100; 70 window_width /= 400; 71 window_width *= 50; 72 if (window_width < 0) window_width = 0; 73 if (window_width > 49) window_width = 49; 74 check_page(window_width); 75 76 al_draw_bitmap(pages[(int)window_width], 0, 0, 0); 77 78 al_flip_display(); 79 }; 80}

jmasterx

Here's what I found and wrote in the other thread. Maybe it helps:

It really has to do with resizing because I had it display the alphabet at varying sizes for a few minutes. The second I resize, the letter Z is no longer rendered.

The problem seems to happen when d3d_acknowledge_resize is called.

I suspect the problem is somewhere in _al_d3d_refresh_texture_memory

I tried having it rest a few seconds but it still caused the problem.

I'm not familiar with D3D so I cannot be much help in debugging this area.

Edit:
I think I've somewhat narrowed it down to d3d_sync_bitmap_texture(), or at least there's something there.

When I remove the first call to _al_convert... , the letter Z stops disappearing. However the letters that I get the width of still exhibit the issue.

Edit 2/10/12:
THIS solution seems to somewhat the trick, when the device is lost the bitmaps were not being prepared for reset:
Change:

   if (hr == D3DERR_DEVICELOST) {
      d3d_display->device_lost = true;
     _al_d3d_prepare_bitmaps_for_reset(d3d_display);
      return;
   }
   else {
      _al_d3d_prepare_bitmaps_for_reset(d3d_display);
   }

New patch attached.

It solves the case where the Z was vanishing but the get text width problem still prevails.

Edit 2/13/12:
I noticed that the results it gives when it messes up are fairly predictable. I tried changing the order that the fonts are loaded in and now it looks like this:
{"name":"605587","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/a\/e\/ae4e7551b10f00cba71ed1fbfde447bc.png","w":634,"h":245,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/a\/e\/ae4e7551b10f00cba71ed1fbfde447bc"}605587

As can be seen here, the problem seems to be that when it either makes the backup or restores from backup it grabs from the wrong font. Hopefully this helps :)

Trent Gamblin

Still not sure what it is. The font addon uses sub-bitmaps and those don't change when backing up/restoring the sheets. Don't know if that's relevant but it just occurred to me. Doesn't seem relevant since the sub bitmap parent pointer doesn't change and the rectangle stays the same. Maybe digging a little deeper with some debugging drawing from within the ttf addon, maybe drawing each sheet that you can cycle through or something, would help.

jmasterx

I did some digging around in the ttf.c and changed:

 glyph_data = alloc_glyph_region(font_data, w, h, false, glyph, lock_more);

to

 glyph_data = alloc_glyph_region(font_data, w, h, true, glyph, lock_more);

And the problem completely disappears, the returned text width is correct and it uses video bitmaps.

Does this indicate anything to you?

I think it causes every glyph to be on its own page, which wastes a ton of memory, but the problem vanishes.

Trent Gamblin

Hm, that piques my interest. Obviously that's not a solution (using true will create a whole new page for each character), but it made me think... maybe Peter Wang who optimized the addon could tell us, does the ttf addon hold a lock on glyphs between drawing calls? Because maybe the page(s) are locked when D3D tries to backup/restore them. Not sure how to work around that since some of the data (relock, possibly other things) is held in the font and the D3D driver doesn't have access. But if it does hold a lock, releasing the lock before backup/restore (in a proper way as to respect all of the ttf internal variables), may solve the problem.

jmasterx

I have found something even better now.

The whole notion of locking got me realizing that the renderGlyph function calls lock_more on the cache function with false.

I created a new function called _renderGlyph which is basically renderGlyph WITHOUT the call to al_draw_bitmap. Calling it like this returns the correct text width and the problem vanishes.

#SelectExpand
1static int ttf_text_length(ALLEGRO_FONT const *f, const ALLEGRO_USTR *text) 2{ 3 ALLEGRO_TTF_FONT_DATA *data = f->data; 4 FT_Face face = data->face; 5 int pos = 0; 6 int prev_ft_index = -1; 7 int x = 0; 8 int32_t ch; 9 bool hold; 10 11 hold = al_is_bitmap_drawing_held(); 12 13 while ((ch = al_ustr_get_next(text, &pos)) >= 0) { 14 int ft_index = FT_Get_Char_Index(face, ch); 15 ALLEGRO_TTF_GLYPH_DATA *glyph = get_glyph(data, ft_index); 16 17 x += _render_glyph(f,al_map_rgb(255,255,255),prev_ft_index,ft_index,0,0); 18 19 prev_ft_index = ft_index; 20 } 21 22 al_hold_bitmap_drawing(hold); 23 24 return x; 25} 26 27static int _render_glyph(ALLEGRO_FONT const *f, 28 ALLEGRO_COLOR color, int prev_ft_index, int ft_index, 29 float xpos, float ypos) 30{ 31 ALLEGRO_TTF_FONT_DATA *data = f->data; 32 FT_Face face = data->face; 33 ALLEGRO_TTF_GLYPH_DATA *glyph = get_glyph(data, ft_index); 34 int advance = 0; 35 36 /* We don't try to cache all glyphs in a pre-pass before drawing them. 37 * While that would indeed save us making separate texture uploads, it 38 * implies two passes over a string even in the common case when all glyphs 39 * are already cached. This turns out to have an measureable impact on 40 * performance. 41 */ 42 cache_glyph(data, face, ft_index, glyph, false); 43 unlock_current_page(data); 44 45 advance += get_kerning(data, face, prev_ft_index, ft_index); 46 advance += glyph->advance; 47 48 return advance; 49}

I can confirm that with this method, the problem in my game completely is gone!

Trent Gamblin

What if you add at line 571 this:

Giving you:

#SelectExpand
1static int ttf_text_length(ALLEGRO_FONT const *f, const ALLEGRO_USTR *text) 2{ 3 ALLEGRO_TTF_FONT_DATA *data = f->data; 4 FT_Face face = data->face; 5 int pos = 0; 6 int prev_ft_index = -1; 7 int x = 0; 8 int32_t ch; 9 bool hold; 10 11 hold = al_is_bitmap_drawing_held(); 12 al_hold_bitmap_drawing(false); 13 14 while ((ch = al_ustr_get_next(text, &pos)) >= 0) { 15 int ft_index = FT_Get_Char_Index(face, ch); 16 ALLEGRO_TTF_GLYPH_DATA *glyph = get_glyph(data, ft_index); 17 18 cache_glyph(data, face, ft_index, glyph, true); 19 20 x += get_kerning(data, face, prev_ft_index, ft_index); 21 x += glyph->advance; 22 23 prev_ft_index = ft_index; 24 } 25 26 al_hold_bitmap_drawing(hold); 27 28 return x; 29}

Added note: I think with your method if you used al_get_text_width before drawing you might run into problems.

jmasterx

Nope that does not work.

I tried my method with getting the width at various places (as I do in my game) and there does not seem to be a problem with it. What sort of problems do you foresee?

What does the magic is

static void unlock_current_page(ALLEGRO_TTF_FONT_DATA *data)
{
   if (data->page_lr) {
      ALLEGRO_BITMAP **back = _al_vector_ref_back(&data->page_bitmaps);
      ASSERT(al_is_bitmap_locked(*back));
      al_unlock_bitmap(*back);
      data->page_lr = NULL;
   }
}

Maybe we should check if a bitmap is locked before making a backup and relock it if it was locked?

Currently the d3d code does:

if (dest->locked) {
      return;
   }

Which does not seem right.

Another option could be to lock the font page back up after getting the text width; That might work.

Trent Gamblin
jmasterx said:

Maybe we should check if a bitmap is locked before making a backup and relock it if it was locked?

That sounds reasonable. So check if it's locked, if it is store the lock info (x, y, width, height, format, other?), unlock it, back it up, then re-lock it. This assumes the ttf addon doesn't hold any pointers within the lock->data region. If that works then it's a good solution.

jmasterx

Locking the ttf as readwrite rather than write only, seems to have a different effect. I'm going to look into trying to understand why the other technique works.

Edit:
After trying several different locking / unlocking, it really feels like my solution is the best solution. It is simply 1 line like this:

#SelectExpand
1static int ttf_text_length(ALLEGRO_FONT const *f, const ALLEGRO_USTR *text) 2{ 3 ALLEGRO_TTF_FONT_DATA *data = f->data; 4 FT_Face face = data->face; 5 int pos = 0; 6 int prev_ft_index = -1; 7 int x = 0; 8 int32_t ch; 9 bool hold; 10 11 hold = al_is_bitmap_drawing_held(); 12 13 while ((ch = al_ustr_get_next(text, &pos)) >= 0) { 14 int ft_index = FT_Get_Char_Index(face, ch); 15 ALLEGRO_TTF_GLYPH_DATA *glyph = get_glyph(data, ft_index); 16 17 cache_glyph(data, face, ft_index, glyph, true); 18 unlock_current_page(data); //Only change 19 x += get_kerning(data, face, prev_ft_index, ft_index); 20 x += glyph->advance; 21 22 prev_ft_index = ft_index; 23 } 24 25 al_hold_bitmap_drawing(hold); 26 27 return x; 28}

literally the only change here is the call to:

unlock_current_page(data);

I have tested in several situations with and without rendering anything and I do not see a performance impact nor are results inconsistent.

If you could tell me what sorts of problems I should be looking for or problems doing this could cause, please let me know. I do not see a problem because when rendering a glyph, this is called. And Allegro's state is always 'ready to render' since a new scene begins when another has ended.

I tested with GL too and results are nice and consistent here too. If you're worried it might affect other platforms somehow you could ifdef the call for D3D only, but I do not think that is necessary.

I also tried 2 calls to al_get_text_width in a row before any rendering even occurs.

Note that it should be done to ttf_get_text_dimensions as well.

Trent Gamblin

Please post a patch to the [AD] mailing list. I'm not comfortable applying this without Peter first having a look at it. He did all of the optimization in the ttf addon. He'd probably know if this change is ok. I don't know if he's following this thread.

jmasterx

I'll just start a thread in the Allegro Development section.
http://www.allegro.cc/forums/thread/609548/947404#target

Thread #609404. Printed from Allegro.cc