Interesting question here - how do we handle this? Do we want to delete the image after every build? michaelneale, jamesdumay, hrmpw - thoughts?

This isn't specific to pipeline-model-definition though is it? This would be true of any build using a dockerfile, no? abayer

Yes, but in Declarative, we possibly generate a lot more of 'em without the user ever having to really think about it.

building the image for every build kind of defeats the purpose doesn't it? Then it would be wiser to just build the image once and publish it to docker hub to use with a regular docker statement.

I think this might fall under general workspace management more than something to be solved in Declarative. We could add some sugar to clean workspace in post but beyond that what can we do?

Preemptive apology for the novel of a comment, but I wanted to be detailed with my thoughts.

hrmpw is absolutely right, building the image every time does mostly defeat the purpose, but being able to store our Docker definition alongside our pipeline definition is exceedingly convenient for both our dev and QA in my experience so far.

This may or may not be hard to do from your end (I'm not too familiar with the code managing the Dockerfile interactions), but here's the way I imagine the Dockerfile flow in declarative pipeline could work:

Each declarative pipeline job run using a Dockerfile would retain 1 or more pairs of fingerprints, where each pair would contain a Dockerfile fingerprint and the fingerprint of the Docker image built from said Dockerfile.

Thus, for each declarative pipeline job run that utilizes Dockerfiles there are two possible paths to follow:

1. The Dockerfile fingerprint does match the fingerprint from the previous job run, meaning that ideally we shouldn't rebuild unless we have to.
   To determine that, we check the current node for the image fingerprint from the previous job run:
   1. If the current node does have an image that matches the image fingerprint from the previous job run, just run that image and continue with the job.
   2. If the current node doesn't have an image that matches the image fingerprint from the previous job run, then we logically need to build it on the current node even though the Dockerfile itself hasn't changed.
2. The Dockerfile fingerprint doesn't match the fingerprint from the previous job run, meaning we should rebuild and clean up previously created Docker images if present.
   Just like the first path we check the current node for the image fingerprint from the previous job run, but this time we focus on cleanup:
   1. If the current node does have an image that matches the image fingerprint from the previous job run, remove that image from the current node and then build and run like normal.
   2. If the current node doesn't have an image that matches the image fingerprint from the previous job run, then we've at least done our due diligence to clean up and just build and run like normal.

Keeping Dockerfile and Docker image fingerprints associated as a pair ensures that you can selectively remove or rebuild per Dockerfile used, and removing images only relative to fingerprints from the last job run handles what I'm guessing is the common case for Dockerfile image management.

I think this gives us the best overall user-friendliness for Dockerfiles in the declarative pipeline syntax, following the mentality that users generally shouldn't have to think too much about managing or selecting the nodes they're utilizing (or what arbitrary Docker images they create).

Let me know what you think or if I obviously missed something!

At the moment, the Dockerfile builds are fairly quick after the first build, as Docker's cache is used to avoid rebuilding any Dockerfile steps that haven't changed. This caching would be defeated if the built image was deleted after each build (unless the Docker node has similar images that are not managed by Jenkins).

Determining whether an image should be rebuilt is not necessarily as simple as checking whether the Dockerfile has changed, or that a previous image built for that same Dockerfile still exists. I think this logic should be left to Docker, and not replicated in Jenkins.

My suggestion would be the following:

• When building a Dockerfile, tag the resulting image with the job name/Id instead of the Dockerfile hash.
• Before building a Dockerfile, check for an existing image using the job's tag. If an image exists, note the image ID.
• After building a Dockerfile, check the image ID of the current image with the job's tag. If the image ID has changed (i.e. the tag has moved), delete the old image.

If a job's name is changed, or a job is deleted, there may be some images left orphaned in the Docker node. However, it will be much easier for an admin to clean these up in the future, as the tags would make it obvious what the images were used for.

Playing devils advocate here but should we really be in the business of docker image lifecycle management? There are tools like docker-gc that can take care of all of that for you.

jamesdumay
I can totally see what you're saying, but to some extent I kinda think yes.

To me, part of the beauty of using Dockerfile(s) in the Declarative Pipeline syntax is that it really does allow me to just stop caring about my nodes. I don't care about their platform, I don't care about what they have installed (beyond Docker, of course), and I don't care about managing a complete pre-built Docker image somewhere.

Not caring too much about the images I generate that way seems like it fits right in to that mentality.

Here's the perspective I think many people might end up seeing this from.

As someone who is a Jenkins user but not a Jenkins admin, working with a pool of generic nodes with Docker installed, I don't want to be that person who used up all of a slave/node's disk space because:

1. I didn't have permission to access the nodes directly and clean up my old images
2. I didn't know how to clean up my old images
3. I didn't have time or care enough to clean up my old images

One solution, just like you suggested, is to run something like docker-gc on each and every node in the pool with a regular cadence.
Frankly, to manage a whole pool of Docker nodes like I'm thinking, that may very well have to be something we do anyway and that would just become part of the requirements to be a Docker node.

I'm just not sure if everyone else thinks that running something like docker-gc totally separate from the Jenkins job creating the images is a suitable solution.
Does that make sense, or am I missing something still?