Hello fans of the Armed Bear,
As you've probably noticed, Docker and similar technologies are extremely popular these days. Additionally, there is unfortunately no Common Lisp representation in Docker's Official Images program (the Docker standard library, if you will). I'd like to change that and ABCL seems like a good candidate to start with because it's both relatively simple to Dockerize and Mark is supportive of getting CL representation in that program.
I've been maintaining some Common Lisp related Docker images in my personal namespace (daewok) for several years now. I recently moved the repos to a gitlab.common-lisp.net group (mirrored to Github) and have cleaned up/gotten ABCL's repos into a state where I believe they adhere to all of the Official Image program's guidelines and are ready for inclusion.
I wanted to reach out to 1) get any feedback from ABCL devs and users to make sure the images are useful and 2) get consent from the ABCL devs to submit to the program.
First, the code:
The Dockerfiles themselves are located at https://gitlab.common-lisp.net/cl-docker-images/abcl
The manifest for inclusion in the Official Images program is at https://gitlab.common-lisp.net/cl-docker-images/official-images/-/blob/abcl/...
The documentation for inclusion on Docker Hub is at https://gitlab.common-lisp.net/cl-docker-images/docs/-/tree/abcl/abcl
All of these are also mirrored to the cl-docker-images org on Github.
The images are built for the cross product of {buster, windowsservercore-1809, windowsservercore-ltsc2016} x {jdk8, jdk11, jdk15} x {arm64, amd64} (except for all combos containing jdk8 and arm64). The current versions of the images are located at https://hub.docker.com/r/daewok/abcl if you would like to check them out.
I am planning that only the latest version of ABCL will be "supported". The initial request would also backfill 1.7.1. Once a version becomes unsupported it would not be removed from Docker hub, it just receives no more base image updates.
Next, some info about Docker Official Images:
Documentation on the program is located at https://docs.docker.com/docker-hub/official_images/ and https://github.com/docker-library/official-images
In summary, every image in the program is reviewed by a team for best practices and security, built by that team, and is available for pulling without specifying a namespace (e.g., `docker pull abcl:latest`). This helps give users confidence in the code they're running, as opposed to pulling from random personal namespaces.
Even the Official Images team admits that "Official Images" is not a great name as it can imply that the upstream of the software contained in the image maintains the images when that is frequently not the case. As I'm not an ABCL dev myself, the image would be considered community maintained and the description on Docker Hub would clearly point to the cl-docker-images/abcl repo as where to file issues with the images. But it is very conceivable that random users may show up on this list asking for help. I am subscribed to this list and will do my best to redirect/answer any of those types of emails that come in.
Thoughts, questions, concerns?
-Eric
On Nov 7, 2020, at 21:29, Eric Timmons etimmons@mit.edu wrote:
Hello fans of the Armed Bear,
[…]
I am planning that only the latest version of ABCL will be "supported". The initial request would also backfill 1.7.1. Once a version becomes unsupported it would not be removed from Docker hub, it just receives no more base image updates.
Next, some info about Docker Official Images:
Documentation on the program is located at https://docs.docker.com/docker-hub/official_images/ and https://github.com/docker-library/official-images
In summary, every image in the program is reviewed by a team for best practices and security, built by that team, and is available for pulling without specifying a namespace (e.g., `docker pull abcl:latest`). This helps give users confidence in the code they're running, as opposed to pulling from random personal namespaces.
Even the Official Images team admits that "Official Images" is not a great name as it can imply that the upstream of the software contained in the image maintains the images when that is frequently not the case. As I'm not an ABCL dev myself, the image would be considered community maintained and the description on Docker Hub would clearly point to the cl-docker-images/abcl repo as where to file issues with the images. But it is very conceivable that random users may show up on this list asking for help. I am subscribed to this list and will do my best to redirect/answer any of those types of emails that come in.
Thoughts, questions, concerns?
Looks just fine. I think I will adopt some of your conventions for the docker entrypoint and other things from your binary-based Dockerfiles, with the [source-based Dockerfile recipe in the ABCL tree][1] so that I user can use either one interchangibly without surprise.
[1]: https://gitlab.common-lisp.net/abcl/abcl/-/blob/master/Dockerfile
What can I do as the ABCL Release Engineer to get changes to your Dockerfiles when ABCL is released? Submit a pull request? Use a more credentialed public/private key pair when I sign the releases?
Thanks ever so much for charging ahead here, Mark
Mark Evenson evenson@panix.com writes:
Looks just fine. I think I will adopt some of your conventions for the docker entrypoint and other things from your binary-based Dockerfiles, with the [source-based Dockerfile recipe in the ABCL tree][1] so that I user can use either one interchangibly without surprise.
Great! I was doing one last pass on the binary Dockerfiles and realized that the release should *probably* be unpacked in /usr/local/share instead of /usr/local/lib. I'll make that change and submit sometime later.
What can I do as the ABCL Release Engineer to get changes to your Dockerfiles when ABCL is released? Submit a pull request? Use a more credentialed public/private key pair when I sign the releases?
If you could submit a pull request on release that'd be fantastic! For the upcoming 1.8.1 release, I think you should need to do only the following things:
1. Run `./update.sh 1.8.1` 2. Delete the 1.8.0 folder 3. Update the aliases map in generate-stackbrew-library.sh 4. Commit all changes and make a pull request
After merging, all the building for the "unofficial" images should happen automatically. Then presuming evertyhing looks good I'll update the manifest and ask the Docker Official Images team to merge it.
I don't think anything needs to be changed about the key pair. There are no requirements on how trusted it is, etc. only that the full fingerprint be used and easily visible so they and any other user can confirm it's the correct one. If you do decide to change the key for whatever reason, just update it in the Dockerfile templates before running the update.sh script.
-Eric
Thanks for starting this. It would be a great contribution.
Something I've wanted for a very long time is to have a checkpoint/resume for ABCL - would function like save-world in other lisps. Doing this would make it practical to build tools which use lots of code, such as my own, start up quickly and become practical for script usage. Docker allegedly has some support for this but I've never seen it work on MacOS. If this is something you have any awareness of how to do I'd love to see that integrated into an ABCL docker image.
Could you say a few words about the image OS environment and your considerations when choosing it, and a bit on how to understand the tags?
Thanks, Alan
On Sat, Nov 7, 2020 at 3:30 PM Eric Timmons etimmons@mit.edu wrote:
Hello fans of the Armed Bear,
As you've probably noticed, Docker and similar technologies are extremely popular these days. Additionally, there is unfortunately no Common Lisp representation in Docker's Official Images program (the Docker standard library, if you will). I'd like to change that and ABCL seems like a good candidate to start with because it's both relatively simple to Dockerize and Mark is supportive of getting CL representation in that program.
I've been maintaining some Common Lisp related Docker images in my personal namespace (daewok) for several years now. I recently moved the repos to a gitlab.common-lisp.net group (mirrored to Github) and have cleaned up/gotten ABCL's repos into a state where I believe they adhere to all of the Official Image program's guidelines and are ready for inclusion.
I wanted to reach out to 1) get any feedback from ABCL devs and users to make sure the images are useful and 2) get consent from the ABCL devs to submit to the program.
First, the code:
The Dockerfiles themselves are located at https://gitlab.common-lisp.net/cl-docker-images/abcl
The manifest for inclusion in the Official Images program is at
https://gitlab.common-lisp.net/cl-docker-images/official-images/-/blob/abcl/...
The documentation for inclusion on Docker Hub is at https://gitlab.common-lisp.net/cl-docker-images/docs/-/tree/abcl/abcl
All of these are also mirrored to the cl-docker-images org on Github.
The images are built for the cross product of {buster, windowsservercore-1809, windowsservercore-ltsc2016} x {jdk8, jdk11, jdk15} x {arm64, amd64} (except for all combos containing jdk8 and arm64). The current versions of the images are located at https://hub.docker.com/r/daewok/abcl if you would like to check them out.
I am planning that only the latest version of ABCL will be "supported". The initial request would also backfill 1.7.1. Once a version becomes unsupported it would not be removed from Docker hub, it just receives no more base image updates.
Next, some info about Docker Official Images:
Documentation on the program is located at https://docs.docker.com/docker-hub/official_images/ and https://github.com/docker-library/official-images
In summary, every image in the program is reviewed by a team for best practices and security, built by that team, and is available for pulling without specifying a namespace (e.g., `docker pull abcl:latest`). This helps give users confidence in the code they're running, as opposed to pulling from random personal namespaces.
Even the Official Images team admits that "Official Images" is not a great name as it can imply that the upstream of the software contained in the image maintains the images when that is frequently not the case. As I'm not an ABCL dev myself, the image would be considered community maintained and the description on Docker Hub would clearly point to the cl-docker-images/abcl repo as where to file issues with the images. But it is very conceivable that random users may show up on this list asking for help. I am subscribed to this list and will do my best to redirect/answer any of those types of emails that come in.
Thoughts, questions, concerns?
-Eric
Hi Alan,
Alan Ruttenberg alanruttenberg@gmail.com writes:
Thanks for starting this. It would be a great contribution.
Something I've wanted for a very long time is to have a checkpoint/resume for ABCL - would function like save-world in other lisps. Doing this would make it practical to build tools which use lots of code, such as my own, start up quickly and become practical for script usage. Docker allegedly has some support for this but I've never seen it work on MacOS. If this is something you have any awareness of how to do I'd love to see that integrated into an ABCL docker image.
I've never been able to get Docker checkpoint/restore to work for me. I tried a while ago (IIRC, prompted by you asking a similar question on slime-docker!) and ran into https://github.com/moby/moby/issues/35691 and I just tried again and am currently seeing https://github.com/moby/moby/issues/41639. Unfortunately I don't have the time to experiment with various Docker and CRIU version combos to find one where they actually work together.
However, from what knowledge I have been able to gather, it appears that nothing special needs to be added to the image to make C/R work. It happens entirely outside the view of the running processes and is supposed to just enable the process to pick up exactly where the checkpoint happened, none the wiser that it got paused.
Could you say a few words about the image OS environment and your considerations when choosing it, and a bit on how to understand the tags?
Sure! All official images must inherit from other official images and if an official image exists for a needed software stack you're highly encouraged to use it instead of installing the software yourself (to help encourage layer reuse). I'm aware of at least three Java runtime images: openjdk, adoptopenjdk, and amazoncorretto. My decision to use openjdk as the base image was semi-arbitrary in that I've used openjdk before, the in-tree Dockerfile for building ABCL from source uses openjdk, and for other official images that support only one Java runtime, openjdk seemed to be the most used. However, there certainly is precedent for Java based images to support multiple runtimes via tags so if ABCL users think that's important or if the default should be changed, that's definitely something we can do.
For the Windows images, we're limited to what the Official Images team's hardware is. They're running a version of Windows Server that supports 1809 at most, so that's why the Windows versions appear to be so old.
For Linux images, the openjdk image is built for both Debian Buster and Oracle Linux 8. Buster is their default for Java versions < 15 and Oracle Linux is their default for >= 15. I chose to make the ABCL images default to Buster for all Java versions beacuse, personally, it seemed a little ridiculous to change OSes that drastically based only on Java version. I didn't create any Oracle Linux variants of the ABCL image because Buster is almost certainly the most widespread base image and I took a guess that ABCL users would prefer that. But if there's a desire for Oracle Linux based images, they would be easy to add.
The openjdk image also builds for Alpine, but only on the cutting edge versions of Java and they drop support for it as soon the Java version reaches General Availability.
Last, the openjdk image does have tags for either a JRE or JDK install of Java. I chose to use only the JDK since that's the most useful, but if there is desire for images that have only the JRE, that is also possible.
With that out of the way, onto the tags for ABCL! Every tuple in my original email has a canonical tag of the form $ABCL_VERSION-$JAVA_VERSION-$OS (note processor architecture is omitted). For instance: 1.8.0-jdk15-buster. Then, each of these canonical tags have aliases. These aliases either omit a segment of the tag (if it is the latest version of that particular element) or provide an alternate form.
For instance, 1.8.0-jdk15-buster, 1.8-jdk15-buster, and jdk15-buster are all the same image. 1.8 is an alias for the latest release in the 1.8 series and because 1.8.0 is also the latest ABCL release, it's the default ABCL version and is omitted. jdk in the java segment of the tag is an alias for the latest supported JDK (currently 15). Last, there is a latest tag that is an alias for the latest of everything (ABCL 1.8.0, JDK 15, and Buster on Linux or if on Windows, the latest version of Windows your OS supports).
The architecture is handled by pushing the images to separate organizations on Docker Hub (such as arm64v8/ or amd64/ for the Official Images program) and then tied together in the primary organization using manifest lists. This results in Docker pulling the correct version of the image based on your hardware (or what you request using the --platform CLI flag).
Hopefully that helps!
-Eric
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Eric Timmons -
Mark Evenson