Provide a supported way to build docker images in tasks

[petemoore]

NSS currently can't upgrade taskcluster/image_builder from version 0.1.5 to version 1.3.0 since the 1.3.0 version is gecko-specific.

For this reason, it might make sense to move this into its own (generic) tool.

A disadvantage of this is it might be harder to hack on, in-tree, if you are working on gecko.

Without knowing too many details, I'm creating this issue so that people who know better than me can chime in.

Issue came to light today in #taskcluster.

See also bug 1361413.

CC @djmitche @ttaubert

[gregarndt]

+1 for having a generic image building tool, but I think that it would need to wait for the QEMU engine to be done so we have a better environment to offer to people for building images.

CC @jonasfj

[jonasfj]

We already have docker running under qemu in tc-worker... just started using it for CI :)

[djmitche]

@petemoore can you give more detail on what you envision?

[jonasfj]

I could imagine a base ubuntu image for qemu that already contains docker and a script to clone a git repo and run a script...
So that you just write:

task.payload = {
  image: 'https://static-url-to-ubuntu-image-for-qemu.tar.zst',
  env: {
    REPOSITORY: 'https://github.com/<org>/<repo>.git'
    REVISION: '<revision>',
  },
  command: ['clone-and-exec.sh', 'make', 'build-docker-image'],
  artifacts: [
    {type: 'file', path: '/myimage.zst', name: 'public/images/myimage.zst'},
  ]
}

Assuming your repository contains a Makefile that defines the target build-docker-image which builds a docker image and exports it as a file to /myimage.zst...

Spoiler alert, we have this now :)

[djmitche]

I'm not sure tc-worker, and specifically QEMU, is ready for "prime time" yet. And @petemoore seems to be indicating using docker-in-docker rather than QEMU in the title of this issue. I'm not sure what functionality that would include, and what would be left to the user. @jonasfj's suggestion omits (well, leaves to the user) a lot:

• mercurial support
• "recipes" to copy data from elsewhere in the tree into the Dockerfile directory
• hash generation
• zstd compression
• interfacing with docker (is this via the docker cli, or with curl talking to the docker API?)

Outside of QEMU, the equivalent would be a basic ubuntu:16.04 image with docker and mercurial installed. I don't see a lot of value to us writing and owning a 2-line Dockerfile.

So, I think my question still stands:

@petemoore can you give more detail on what you envision?

[jonasfj]

Another option IMO is an image build that reads an env var CONTEXT_URL, downloads and extracts a tarball from given url, and then builds Dockerfile from said tarball...

Then you be a task that extracts things from you repo... Created said tarball and the.create a dependent task..

Note: IMO using docker CLI and zstd is the easy part...

[djmitche]

Tc-worker is certainly more "prime time" now. Maybe it's time to think about this again? I think this will be easier to solve than #51 using something like what @jonasfj suggested in the previous comment.

[jonasfj]

yeah, tc-worker with qemu engine is the future for building docker images. Once it lands in-tree we'll have stable workerType people can use too.

[djmitche]

@jonasfj, @petemoore I think @glandium has done a lot of the work here to make the existing docker-in-docker solution workable in Gecko. But I'm still missing what the proposed change is here.

[glandium]

Note that I've actually originally written the code that is now in Gecko for git-cinnabar CI, meaning that I am also doing docker-in-docker image building on taskcluster-github. I'm using the python:2.7 image to do that, pip install requests-unixsocket zstandard==0.8.1 and then I run a script within my repo that has the python foo now in Gecko to drive docker-in-docker to build the image for me.

[djmitche]

Related to #51, and something we'll probably want with redeployability.

[petemoore]

@SimonSapin, you mentioned today you are building docker image artifacts from a Dockerfile inside a non-gecko task - are your tasks using taskcluster/image_builder for this, or do you have a different solution? I'm wondering if this issue can be closed. I'm not sure we have docs on how to set this up, in which case we should probably only close it once we have docs, so I'm curious how you set this up, and who you got help from (or if you stumbled on some docs)! Many thanks.

[SimonSapin]

I did not use taskcluster/image_builder. I made a minimal image whose Dockerfile is in https://github.com/servo/taskcluster-bootstrap-docker-images/. It is hosted on and built by Docker Hub.

As far as I can tell, the necessary pieces are:

• A worker type where docker-in-docker is enabled

• An image builder image where Docker 1.6 (at least the client parts) is installed. A more recent version of the Docker client might cause "protocol version mismatch" errors, at least with the AMIs currently running on servo-docker-worker. 1.6 happens to be the version packaged in Ubuntu 14.04 as docker.io.

• A docker-worker task with payload.features.dind = true

Then that task should be able to run docker build and docker save. (And save the resulting tarball as an artifact.)

To cache built images, I took inspiration (though not code) from https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/docker-images.html to use the Index service with a hash of the source Dockerfile. The code for that is at:

https://github.com/servo/servo/blob/c33f5cc/etc/ci/taskcluster/decisionlib.py#L60-L131

See also https://github.com/servo/servo/tree/master/etc/ci/taskcluster#in-tree-docker-images

[glandium]

FWIW, for git-cinnabar, I'm using an ad-hoc docker client with docker-in-docker. Similar code is actually now used for gecko too. https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/util/docker.py