I'm looking for a graphics library that I could use to develop GUI for a VST plugin (Linux initially).

In VST 2.x plugin host creates the window (using XCreateWindow on Linux/X11) and sets is size. The handle/pointer to Window (and X display) is then passed to the child plugin which should use the created window (pointer) and not create its own new window (standard Allegro functions).

Is this possible in Allegro? Would it be easy to hack/modify Allegro library to support this?

cslr

Allegro does actually have API for this dirty-rectangle based drawing... in principle you create a single buffer display and then use al_update_display_region. It only appears to be well implemented on Windows with Direct3D though.

Going back to the original question, there is no exposed way. The only similar thing that's done is how we deal with menus on Linux, where we actually use GTK to create the window, and then connect Allegro to it. This is done here: https://github.com/liballeg/allegro5/blob/master/src/x/xdisplay.c#L380, which calls this function: https://github.com/liballeg/allegro5/blob/master/addons/native_dialog/gtk_xgtk.c#L130. I think in principle you could create generic X11 implementations of the functions in that file and it might work? Honestly, I've never tried.

On Windows, we have a somewhat old patch to deal with external Qt windows here, but the mechanism of Windows is very different: https://github.com/liballeg/allegro5/pull/661