Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38369

Unexpected docker-host issues make Jenkins Hang

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: docker-plugin
    • Labels:
      None
    • Environment:
      Jenkins Version: 2.7.4
      Docker Plugin Version: 0.16.2
      Docker Version: 1.12.1 on debian jessie
      Java Version: 1.7 & 1.8 on Jenkins Host
    • Similar Issues:

      Description

      It is observed in two scenarios that Docker plugin is making the entire Jenkins server to get hang.
      Firstly when docker-host faced kernel panic and secondly when docker-daemon is hanged.

      After this Jenkins not able to fail the running builds, even it make it fail after long time, docker nodes remain as dead.
      It is not possible to delete these nodes, abort the running builds manually or do any global configuration change to resume its working.
      It is required to forcefully kill and start the Jenkins process of the server to make it work again.

      Steps to reproduce
      --------------------
      Start 5 to 10 builds parallely on the Jenkins server which runs on the docker instance.
      Enable "Clean local image" option within "Docker Container" under General config of job.
      Now go to the docker host machine, Manally trigger kernel panic with comand "echo c > /proc/sysrq-trigger"
      Then even after 1 hour builds are showing running and it is hard to abort or delete the docker instances.

        Attachments

          Issue Links

            Activity

            Show
            ndeloof Nicolas De Loof added a comment - same as  https://issues.jenkins-ci.org/browse/JENKINS-42187
            Hide
            ndeloof Nicolas De Loof added a comment -

            API calls while owing the queue should all be made asynchronous. 
            known issue (at least) in DockerSakve#_terminate()

            Show
            ndeloof Nicolas De Loof added a comment - API calls while owing the queue should all be made asynchronous.  known issue (at least) in DockerSakve#_terminate()

              People

              Assignee:
              ndeloof Nicolas De Loof
              Reporter:
              deepraut89 Deepak Raut
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: