So, despite many years of programming experience, I never had anything to do with shaders. I'm more of a game mechanics guy. But I found myself forced to try to write a shader recently, because I want to subtract one texture from another, and on my Tegra-3 based Android device, ALLEGRO_DIST_MINUS_SRC simply don't work at all.

But the problem is, whenever I attach a source for a vertex shader, any kind of output disappears!

Here's the code for drawing:

// Create a sub-bitmap, at coordinates where we want to remove a part of targetBitmap
ALLEGRO_BITMAP *tmp = al_create_sub_bitmap( targetBitmap, x, y, w, h );
al_set_target_bitmap( tmp );
al_use_shader( pShader );

TexPoint tp1( 0, 0, 0, 0 );
TexPoint tp2( w, 0, w, 0 );
TexPoint tp3( w, h, w, h );
TexPoint tp4( 0, h, 0, h );

render.DrawTexturedQuad( m_damageImage, tp1, tp2, tp3, tp4 );

al_set_target_backbuffer( al_get_current_display() );
al_destroy_bitmap( tmp );

if I create the shader in this way:

pShader = al_create_shader( ALLEGRO_SHADER_GLSL );
al_attach_shader_source( pShader, ALLEGRO_VERTEX_SHADER, default_vertex_shader );
al_attach_shader_source( pShader, ALLEGRO_PIXEL_SHADER, my_pixel_shader );
al_build_shader( pShader );

Nothing is visible. However, if I omit attaching vertex shader, then something is drawn at the necessary point.

If I set targetBitmap as a target, instead of sub-bitmap, things seem to work even with vertex shader, but this will lead to problems later, so I would prefer this code to work with sub-bitmap of targetBitmap.

What am I doing wrong here?

I'm pretty sure you can't replicate blenders just like that, so I'm surprised that one of your setups actually worked for you. From my understand of how shaders work, to replicate SRC_MINUS_DEST you'd have to have 3 bitmaps: a destination bitmap (could use your backbuffer), your source bitmap (I imagine that's the targetBitmap) and then the bitmap you will subtract (m_damageImage). Then you'd do something like this:

al_use_shader(shader);
al_set_shader_sampler("tex2", damageImage, 1);
al_draw_bitmap(targetBitmap);

Oh, but the problem is, I didn't even get to the point where I try to replicate the subtraction! For now, my shader just paints everything red And it doesn't work when I use it with sub-bitmap, as described above.

Oh I see. I tried a setup like yours in ex_shader, and it seemed to work ok. Could you paste your pixel shader (I imagine it's pretty simple, but still). Alternatively, also try ex_shader yourself.

OK, I tracked down my problem (problems, actually). First, sub-bitmaps are not very useful with shaders, since they do not change geometry passed to shader. Second, I did not take OpenGL's coordinate system into consideration properly

Is this a recent Allegro 5.1? A week or two ago I landed the projection reform which switched to using viewports for sub-bitmaps, which should have made them operate predictably.

I haven't updated my copy to the latest git yet. Maybe I'll try, though I have a nice working version already. Almost In reality, I'm still missing something: the whole render gets corrupted after I use my newly written shader outside of test in a real game.