• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • docker-workflow-plugin
    • Jenkins 2.19.3, pipeline-model-definition-plugin 0.6

      When using a Dockerfile to set up the build environment (as added in JENKINS-39216), a docker image is built and tagged with the SHA1 hash of the Dockerfile. Nothing seems to remove this image once the build competes, so disk space on the docker node will eventually be exhausted.

          [JENKINS-40723] Built Dockerfile images are never removed

          Hrm.  So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX.

          First of all, that's an invalid tag name.  But let's assume we can make it sane by removing the {{/}}s.

          But more importantly, when that PR lands to master, the tag on the docker image will still be for PR-XX even though it will actually now be the new current image being used for master (I think).  But it's not tagged for master on that project, it's tagged for PR-XX on that project.

          Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one.  But that's an extra build that shouldn't be necessary.

          That said, it's probably still better than simply using the "oldest built" garbage collection mechanism that is being suggested in this ticket.

          Also, allanlewis_youview, you didn't respond to my other concern:

          Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?

           

          Brian J Murrell added a comment - Hrm.  So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX. First of all, that's an invalid tag name.  But let's assume we can make it sane by removing the {{/}}s. But more importantly, when that PR lands to master, the tag on the docker image will still be for PR-XX  even though it will actually now be the new current image being used for master (I think).  But it's not tagged for master on that project , it's tagged for PR-XX on that project. Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one.  But that's an extra build that shouldn't be necessary. That said, it's probably still better than simply using the "oldest built" garbage collection mechanism that is being suggested in this ticket. Also, allanlewis_youview , you didn't respond to my other concern: Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?  

          Attila Szeremi added a comment - - edited

          brianjmurrell
          Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?

          First of all, Jenkins doesn't even do that. If a build would be rerun, it would just build the Docker image again. Secondly, it's not like Jenkins does any kind of caching for non-Docker agents (like caching the node_modules/ generated from an npm install). And besides, Docker already has decent caching built into it; any step in the Dockerfile where the involved files did not change since last time, Docker automatically just re-uses the cached layer; so no additional help from Jenkins is needed.

          Attila Szeremi added a comment - - edited brianjmurrell Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed? First of all, Jenkins doesn't even do that. If a build would be rerun, it would just build the Docker image again. Secondly, it's not like Jenkins does any kind of caching for non-Docker agents (like caching the node_modules/ generated from an npm install). And besides, Docker already has decent caching built into it; any step in the Dockerfile where the involved files did not change since last time, Docker automatically just re-uses the cached layer; so no additional help from Jenkins is needed.

          Allan Lewis added a comment -

          Hrm. So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX.

          I'm not using a GitHub org, so I'm not sure about that. My case is using a manually-configured Git URL.

          First of all, that's an invalid tag name. But let's assume we can make it sane by removing the {{/}}s.

          I'm not using the job name as a tag, I'm using it as the image name, and image names can contain slashes.

          Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one. But that's an extra build that shouldn't be necessary.

          Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed?

          No, because when master is built, it will build the image again - cached if it's on the same node or if one implements push-pull with a registry - and tag it as <repo>/master. If we then prune the tag from the branch, we'll still have the image as it will be tagged for master.

          I'm not saying my solution will work for everyone, I just posted it in case it was useful for others.

          Allan Lewis added a comment - Hrm. So for a GitHub Organisation PR, the $JOB_NAME will be github-org/project/PR-XX. I'm not using a GitHub org, so I'm not sure about that. My case is using a manually-configured Git URL. First of all, that's an invalid tag name. But let's assume we can make it sane by removing the {{/}}s. I'm not using the job name as a tag, I'm using it as the image name, and image names can contain slashes. Granted when (the locally written script for) garbage cleanup removes that PR-XX image because PR-XX is closed, the master branch build will build a new (identical) docker image to replace the just removed one. But that's an extra build that shouldn't be necessary. Doesn't deleting all of the Jenkins produced hashy type tags mean that Jenkins won't be able to find an image to re-use for a future run of a job where the Dockerfile has not changed? No, because when master is built, it will build the image again - cached if it's on the same node or if one implements push-pull with a registry - and tag it as <repo>/master . If we then prune the tag from the branch, we'll still have the image as it will be tagged for master. I'm not saying my solution will work for everyone, I just posted it in case it was useful for others.

          Dwight Guth added a comment -

          I definitely side with the people who think that this was resolved prematurely. We have been running docker image prune on our Jenkins node that we recently migrated to dockerfiles, and each time it ran it was freeing up less and less space. What we eventually realized is that because Jenkins is tagging every build with a tag corresponding to the hash of the dockerfile, when the dockerfile changes, it does not untag the old build (because the new tag is different from the old tag), so the old build is not marked as dangling, and you have to run the significantly more aggressive `docker image prune -a` to clean it up. I believe Jenkins should be tagging these builds with something that is deterministic across builds to the same job and branch, and deleting all the tags associated with change requests when the change request closes and the jenkins branch is deleted.

          Dwight Guth added a comment - I definitely side with the people who think that this was resolved prematurely. We have been running docker image prune on our Jenkins node that we recently migrated to dockerfiles, and each time it ran it was freeing up less and less space. What we eventually realized is that because Jenkins is tagging every build with a tag corresponding to the hash of the dockerfile, when the dockerfile changes, it does not untag the old build (because the new tag is different from the old tag), so the old build is not marked as dangling, and you have to run the significantly more aggressive `docker image prune -a` to clean it up. I believe Jenkins should be tagging these builds with something that is deterministic across builds to the same job and branch, and deleting all the tags associated with change requests when the change request closes and the jenkins branch is deleted.

          Michael Harris added a comment - - edited

          Having some mechanism to delete the generated image is really important. If we have a lot of builds in a short time our nodes still run out of disk space before the scheduled job runs and wipes out old images. Jenkins is creating the images, and at a minimum needs to expose a way to clean up the image. Even if it was only added in the scripting syntax that would be sufficient.

          Michael Harris added a comment - - edited Having some mechanism to delete the generated image is really important. If we have a lot of builds in a short time our nodes still run out of disk space before the scheduled job runs and wipes out old images. Jenkins is creating the images, and at a minimum needs to expose a way to clean up the image. Even if it was only added in the scripting syntax that would be sufficient.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

          Don Schiewer added a comment -

          This is still an issue and there is no good solution offered.

          Don Schiewer added a comment - This is still an issue and there is no good solution offered.

          Koen Dierckx added a comment -

          This is the shortest solution I could find, without reverting to the scripted workflow. Would be nice if this could be done automatically ?

           

          agent {
            dockerfile {
              filename 'test.Dockerfile'
              additionalBuildArgs "-t jenkins-test-build:${env.BUILD_NUMBER}"
            }
          }
          post {
            always {
              echo 'Cleaning up'
              sh 'docker rmi --force $(docker images --quiet --filter=reference="jenkins-test-build")' /* clean up dockerfile images*/
            }
          }
          

           

          Koen Dierckx added a comment - This is the shortest solution I could find, without reverting to the scripted workflow. Would be nice if this could be done automatically ?   agent {   dockerfile {     filename 'test.Dockerfile'     additionalBuildArgs "-t jenkins-test-build:${env.BUILD_NUMBER}"   } } post {   always {     echo 'Cleaning up'     sh 'docker rmi --force $(docker images --quiet --filter=reference= "jenkins-test-build" )' /* clean up dockerfile images*/   } }  

          I've also been bitten by this. 

          Currently, I'm solving it labeling & tagging my images in a predictable way, including project & branch name, and then, at post time:

          • generate the list of legal image tags for the project (one for each branch)
          • get the list of of all existing images for the project
          • subtract first list from the second
          • delete those images

          That deletes the jenkins-generated tag for the current build (but not my predictable tag), and also takes care of deleting images belonging to deleted branches.

          I also do the same for named volumes, by always creating them with a common prefix including project and branch name.

          Implementation goes along these lines:

          // Add label in Dockerfile, to be able to
          // easily identify jenkins-generated tags to belong to this project
          LABEL project=PROJECT_NAME
          
          // tag generated image predictably, including project & branch
                agent {
                  dockerfile {
                    additionalBuildArgs "--tag PROJECT_NAME/${env.BRANCH_NAME}:latest"
                  }
                  reuseNode true
                }
          
          // do cleanup at the end of every build
            post {
              always {
                dockerCleanup()
              }
            }
          
          // aux
          
          def dockerCleanup() {
            echo 'Cleaning stale docker images'
            sh oneline('''
              docker image ls -f
                label=project=PROJECT_NAME
              | tail -n +2
              | cut -d ' ' -f 1
              | egrep -v "$(
                  git branch -r
                  | sed -r 's# +origin/##'
                  | xargs -I{} -n 1 echo -n "PROJECT_NAME/{}|"
                  | sed 's/|$//'
                )"
              | xargs --no-run-if-empty docker image remove -f
            ''')
          
            echo 'Cleaning stale docker volumes'
            sh oneline('''
              docker volume ls -q
              | grep '^hyperscan-native'
              | egrep -v "$(
                  git branch -r
                  | sed -r 's# +origin/##'
                  | xargs -I{} -n 1 echo -n
                  "PROJECT_NAME_{}_VOLUME_NAME|"
                  | sed 's/|$//'
                )"
              | xargs --no-run-if-empty docker volume remove -f
            ''')
          }
          
          def oneline(obj) {
            obj.toString().replace("\n", " ").replaceAll("\\s+", " ")
          }
          
          

           

          That is working ok for me. But it's a stretch. I agree that if it's Jenkins the one tagging the images in an unpredictable way, it should be Jenkins to delete them when stale.

          Eliseo Martínez added a comment - I've also been bitten by this.  Currently, I'm solving it labeling & tagging my images in a predictable way, including project & branch name, and then, at post time: generate the list of legal image tags for the project (one for each branch) get the list of of all existing images for the project subtract first list from the second delete those images That deletes the jenkins-generated tag for the current build (but not my predictable tag), and also takes care of deleting images belonging to deleted branches. I also do the same for named volumes, by always creating them with a common prefix including project and branch name. Implementation goes along these lines: // Add label in Dockerfile, to be able to // easily identify jenkins-generated tags to belong to this project LABEL project=PROJECT_NAME // tag generated image predictably, including project & branch agent { dockerfile { additionalBuildArgs "--tag PROJECT_NAME/${env.BRANCH_NAME}:latest" } reuseNode true } // do cleanup at the end of every build post { always { dockerCleanup() } } // aux def dockerCleanup() { echo 'Cleaning stale docker images' sh oneline(''' docker image ls -f label=project=PROJECT_NAME | tail -n +2 | cut -d ' ' -f 1 | egrep -v "$( git branch -r | sed -r 's# +origin/##' | xargs -I{} -n 1 echo -n "PROJECT_NAME/{}|" | sed 's/|$ //' )" | xargs --no-run- if -empty docker image remove -f ''') echo 'Cleaning stale docker volumes' sh oneline(''' docker volume ls -q | grep '^hyperscan- native ' | egrep -v "$( git branch -r | sed -r 's# +origin/##' | xargs -I{} -n 1 echo -n "PROJECT_NAME_{}_VOLUME_NAME|" | sed 's/|$ //' )" | xargs --no-run- if -empty docker volume remove -f ''') } def oneline(obj) { obj.toString().replace( "\n" , " " ).replaceAll( "\\s+" , " " ) }   That is working ok for me. But it's a stretch. I agree that if it's Jenkins the one tagging the images in an unpredictable way, it should be Jenkins to delete them when stale.

          A proper solution would also for jenkins to use DIND (docker-in-docker) before creating the docker agent image so that the whole docker environment use for the job gets deleted afterwards, be it sub-image for the agent or others created by the job.  Or at least provide this as an option.

          Yannick Koehler added a comment - A proper solution would also for jenkins to use DIND (docker-in-docker) before creating the docker agent image so that the whole docker environment use for the job gets deleted afterwards, be it sub-image for the agent or others created by the job.  Or at least provide this as an option.

            Unassigned Unassigned
            gllewellyn Gavin Llewellyn
            Votes:
            5 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated: