Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44564

Deadlock restarting jenkins

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      Jenkins 2.163

      Description

      I'm getting errors uploading plugins to Jenkins and then restarting. This is pretty reliable but doesn't happen on every restart, I have seen it happen about 5 times in the last week, restarting about 20 times a day.

      I was able to kill -3 and get a thread-dump (full text below) – the relevant part seems to be a deadlock described here:

       

      Found one Java-level deadlock:
=============================
"Handling GET /static/5adaeac1/images/headshot.png from 172.30.0.86 : RequestHandlerThread[#35]":
waiting to lock monitor 0x00007f104c0047f8 (object 0x0000000091715c10, a hudson.DescriptorExtensionList),
which is held by "Handling GET /static/5adaeac1/images/headshot.png from 172.30.0.86 : RequestHandlerThread[#11]"
"Handling GET /static/5adaeac1/images/headshot.png from 172.30.0.86 : RequestHandlerThread[#11]":
waiting to lock monitor 0x00007f1070c0d018 (object 0x0000000099f9b518, a hudson.ExtensionList$Lock),
which is held by "Initializing plugin copyartifact"
"Initializing plugin copyartifact":
waiting to lock monitor 0x00007f104c0047f8 (object 0x0000000091715c10, a hudson.DescriptorExtensionList),
which is held by "Handling GET /static/5adaeac1/images/headshot.png from 172.30.0.86 : RequestHandlerThread[#11]"

      Our work around is to stop the service sudo service jenkins stop - the process will still be running so we kill -9 it, and then start jenkins again.

      Full thread dump – cannot include inline because it exceeds text limits:

      https://gist.github.com/dalvizu/02071f3c8baf30c71cb3df7d2a53ab64

       

        Attachments

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-55361 DescriptorExtensionList not locking correctly, leading to deadlocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

            Activity

            Ascending order - Click to sort in descending order
            dalvizu Dan Alvizu created issue -
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Component/s jobconfighistory-plugin [ 15683 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels deadlock
            oleg_nenashev Oleg Nenashev made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Stefan Brausch [ stefanbrausch ]
            greybird Arnaud TAMAILLON made changes -
            Link This issue is related to JENKINS-55361 [ JENKINS-55361 ]
            oleg_nenashev Oleg Nenashev made changes -
            Released As Jenkins 2.163
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]

              People

              Assignee:
              stefanbrausch Stefan Brausch
              Reporter:
              dalvizu Dan Alvizu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: