I'd like to use a shader to find out if vertices are inside a circle. If not make the vertex transparant. The following compiles and runs but simply shows the full image as I made it.

and

// Vertex
#version 420
varying vec2 v_texCoord;

void main()
{
    v_texCoord = gl_MultiTexCoord0;// * 0.5;// kleinere factor vergroot meer
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
}

Is this any good at all? Is it possible to use a mask image (white circle in black square) and let a shader do the filling in of the .a component?

The backbuffer alpha is undefined. Try drawing to a buffer first, then draw using normal alpha blender to the backbuffer.

This is the shader I use to achieve what you are asking for, but it's an Allegro shader not a pure openGl one - the circle function may be some help though:

This will draw a circle in a bitmap of any size, with the other pixels transparent.

The backbuffer alpha should be known and filled already I think. I have this:

  wtr = al_load_bitmap("data/voor.bmp");
  tmp = al_create_bitmap(al_get_bitmap_width(wtr)*2, al_get_bitmap_height(wtr)*2);

and

I didn't occur to me to post this code. So wtr goes to the shader and it's result will be written to a four times larger bitmap tmp (yes, simulating a scope).

I'll try Dizzy Eggs function for sure but I'd like to know what I'm doing wrong.

- And it does the resize OK, but draws the full square.

>The backbuffer alpha should be known and filled already I think. I have this:

No, it's not defined. The backbuffer may or may not have alpha, depending on the driver and how you set it up, so you can't draw alpha directly to the backbuffer and expect it to work.

You need to respect that the coordinates are 1.0/1.0 for the image in openGl, so you should probably use something like this (this assumes that wtr is 23x23 and tmp is 46x46):

float Cirkel( vec2 p, float mx, float my )
{
  // mx and my will be 23,23 the centre of tmp
  p.x *= 46; //multiply by the size of tmp
  p.y *= 46; //multiply by the size of tmp
  p.y = 46-p.y; //offset for openGl coords

  if(pow(p.x, 2) - 2*mx*p.x + pow(mx, 2) + pow(p.y, 2) - 2*my*p.y + pow(my, 2) < 529)
    return 1.0;
  else
    return 0.0;
}

You always want to multiply p.x and p.y by the size of tmp, and set mx and my to the centre of tmp...

EDIT:

Also, make sure you set:

al_set_blender(ALLEGRO_ADD, ALLEGRO_ALPHA, ALLEGRO_INVERSE_ALPHA);

Before drawing, to make sure alpha will work properly with the shader.

I already had al_set_blender(ALLEGRO_ADD, ALLEGRO_ALPHA, ALLEGRO_INVERSE_ALPHA); before Edgar's second reply but forgot to post that.

And I had done al_get_bitmap_format(al_get_target_bitmap() which reports 9, it is ALLEGRO_PIXEL_FORMAT_ARGB_8888 if read it right. It creates a bigger cirkle now always on a 93x93 bitmap.

float Cirkel( vec2 p)//, float mx, float my )
{
  // mx and my will be 46.5,46.5 the centre of tmp, radius too is 46.5
  p.x *= 93; //multiply by the size of tmp
  p.y *= 93; //multiply by the size of tmp
  p.y = 93-p.y; //offset for openGl coords

//if(pow(p.x, 2) - 2*mx*p.x + pow(mx, 2) + pow(p.y, 2) - 2*my*p.y + pow(my, 2) < 529)
  if(pow(p.x, 2) -   93*p.x              + pow(p.y, 2) -   93*p.y              < -2162.25)
    return 1.0;
  else
    return 0.01;
}

But this is a great function, thanks. I can feed it a smaller image for more magnification.

-edit: I redid the whole thing in a simpler program, now paying attention to do an al_clear_to_color(al_map_rgba_f(0.0, 0.0, 0.0, 0.0)); after setting tmp as target. That did it.