Allegro.cc - Online Community

Allegro.cc Forums » The Depot » Allegro Planet Remote Builds - Project Update

This thread is locked; no one can reply to it. rss feed Print
Allegro Planet Remote Builds - Project Update
Mark Oates
Member #1,146
March 2001
avatar

Allegro Planet Remote Builds

Some of you may recall discussions about Allegro Planet offering automated builds of your Allegro games. I wanted to share my very promising progress on the project so far. :)

Progress

  • Support for project-specific GitHub webhooks has been developed and is in Allegro Planet. Webhook calls from GitHub are collected in Allegro Planet on a per-project basis. (These will be used eventually to trigger builds).

  • An experimental RemoteGithubRepoBuilder demonstrates the proof-of-concept of running a build process.

Plans

  • Move the built file from the tmp/ directory into a publically accessible location.

  • Create a Build class that keeps status of an ongoing build, and notifies the user/subscriber of build events.

  • Add a "watch build" type of page that allows users to follow the progress (and possible error messages) of their builds.

  • Move work done on RemoteGithubRepoBuilder into a background job. Have it create a Build object and update that object during during build process.

  • Investigate Rails' ActionCable to notify subscribers.

  • Attach new builds to a project (ensure no filename collisions), and add a download button to the Game's project page 🎉

Problems

  • Early versions will likely have very limited build options and may impose restrictions on your directory structure and Makefile. Since Makefiles can arbitrarily run any commands on the system, there will need to be safeguards to ensure that the system remains secure. This may include a requirement for container builds, or might introduce an optional .allegro-planet file in your repo that include a limited scope of build settings for your project.

  • Project build dependencies will likely need to be included in the project's .allegro-planet file and may be restricted to allegro and allegro_flare for the time being.

  • Early versions will likely only support Linux builds with the clang/gcc compiler until cross-compilation can be solved.

  • Remote build is an advanced feature, Allegro Planet still doesn't have user accounts built.

  • Remote builds initially adds support for GitHub webhooks, but a more open-ended build system allowing cloning repos from arbitrary locations could be pretty cool, too.

  • Builds should ideally be handled completely on the Allegro Planet (or docker on the Allegro Planet) server. It's possible there will be some pressure to have builds run on remote systems and I would like to avoid that if possible.

  • GitHub allows the remote server to post ongoing status directly back to your repo. This feature would reduce build<->check iteration time substantially but would require users to generate a token and/or integration permissions on a repo and would add an additional time dependency on development.

Ways That You Can Help

  • Cross-platform building is still unexplored and a bit of an unknown. Ideally, the system could cross-compile binaries with the llvm compiler and not rely on any third party OSs to produce the builds. This also means that there are dlls and other dependencies needed for distribution, and that needs to be accounted for as well. None of this has been explored with much depth yet.

Elias
Member #358
May 2000

For my bot in #allegro which allows sending it arbitrary commands I have a Virtual Box instance to execute the binaries. It's very easy:

vboxmanage controlvm botvm poweroff
vboxmanage snapshot botvm restore botsnapshot
vboxmanage startvm botvm --type headless

(botvm is the name of my VM, botsnapshot is the name of a snapshot with the initial state I want it to be in before running the arbitrary command)

Then I run the binary over ssh into that VM. There are solutions which are faster (but the above seems plenty fast, the bot still replies in less than a second) and more secure (there are ways to break out of virtual box). But just saying, it should not be very hard. And with Virtual Box you can also choose to run commands inside of Windows or OSX :)

--
"Either help out or stop whining" - Evert

amarillion
Member #940
January 2001
avatar

If you're using docker, then you could just package up the dependencies in docker images, like I've done here It could potentially be also used to compile for android, which would be awesome!

--
Martijn van Iersel | My Blog | Sin & Cos | Tegel tilemap editor | TINS 2017

beoran
Member #12,636
March 2011

I personally find LXC containers even more useful and flexible than docker. Only downside is that it's only for Linux in Linux containers.

Mark Oates
Member #1,146
March 2001
avatar

If you're using docker, then you could just package up the dependencies in docker images, like I've done here It could potentially be also used to compile for android, which would be awesome!

This is amazing. I need to learn more about how this works because I'm not familiar with it. Do you know if it is possible to build windows binaries in a linux docker?

Elias
Member #358
May 2000

Docker is a huge pain :p

--
"Either help out or stop whining" - Evert

amarillion
Member #940
January 2001
avatar

Elias and I disagree about this :)

I use docker for everything now - my build server, my blog and the TINS website all run on docker now. And on the build server most of the build jobs run in a separate docker container as well. It's the perfect way to keep dependency chains isolated and avoid conflicts. Docker ftw!

Theoretically it would be possible to set up a docker build environment for cross compilation, but I haven't tried this yet. I still do my windows builds on a fully fledged windows VM.

Remember a docker environment is linux (even if it's supported on Win10 and OS X with docker machine, that's just a thinly veiled VM running linux).

--
Martijn van Iersel | My Blog | Sin & Cos | Tegel tilemap editor | TINS 2017

Mark Oates
Member #1,146
March 2001
avatar

And on the build server most of the build jobs run in a separate docker container as well.

What's the configuration of your build server? Are you on a cloud-hosted service?

I'm on Heroku right now, but it doesn't look like there is support to setup some external packages with apt-get for example. Not without an experimental feature, it would seem.

Quote:

Remember a docker environment is linux

Would I essentially still need to setup a complex cross-compile environment? E.g. no native dockers for emulating other OSs?

amarillion
Member #940
January 2001
avatar

What's the configuration of your build server? Are you on a cloud-hosted service?

The build server is an old laptop running under my desk. It uses Jenkins to define and schedule the jobs (you can set it up to track git commits).

My website is hosted on a VPS, which basically gives you a complete linux VM to play with. I can install anything, the sky is the limit (well, as long as it fits in 20G disk space and 1G mem). My provider is Cloudvps, a dutch provider.

Quote:

Would I essentially still need to setup a complex cross-compile environment? E.g. no native dockers for emulating other OSs?

That is correct. The advantage of using docker, though, is that you can separate the maintenance of the build environment from the maintenance of the server, and you can track the maintenance in git.
I think cross-compilation for windows targets should be possible, but I haven't gotten it working yet myself.

--
Martijn van Iersel | My Blog | Sin & Cos | Tegel tilemap editor | TINS 2017

Go to: