Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18169

Deadlock when running multiple delete

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core, parameterized-trigger-plugin
    • None
    • Jenkins 1.509.1 running on Ubuntu 12.04 with JDK 1.7
      Jenkins 1.501 on Debian 6

      Tried to remove ~20 builds in parallel with 30 days of build logs (a lot of files) and this resulted in a deadlock and we needed to restart the server.
      JavaMelody listed the doDelete threads as deadlocked and showed
      1 x hudson.model.Job.delete(Job.java:587)
      17 x hudson.model.Project.getPublishersList(Project.java:114)
      as a result of the deadlock we can't edit builds anymore

      /configSubmit : http-bio-8080-exec-3552	yes	5	BLOCKED	hudson.model.Project.getPublishersList(Project.java:114)

/doDelete : http-bio-8080-exec-3503	yes	5	BLOCKED	hudson.model.Project.getPublishersList(Project.java:114)

/doDelete : http-bio-8080-exec-3543	yes	5	BLOCKED	hudson.model.Job.delete(Job.java:587)

      Looking in project.java it looks like a deprecated method is used

      /**
 * @deprecated as of 1.463 We will be soon removing the restriction that
 *             only one instance of publisher is allowed per type. Use
 *             {@link #getPublishersList()} instead.
 */
public Map<Descriptor<Publisher>,Publisher> getPublishers() {
return publishers.toMap();
}

        1. build1.txt
          7 kB
        2. build2.txt
          7 kB
        3. jenkins-18169-jstack-deadlock.txt
          20 kB
        4. jenkins-18169-thread-dump-redacted.txt
          187 kB

          [JENKINS-18169] Deadlock when running multiple delete

          Marcus Jacobsson created issue -

          kutzi added a comment -

          Please attach the full thread dump!

          kutzi added a comment - Please attach the full thread dump!

          Emil Sit added a comment -

          We have seen this issue as well. We are using Jenkins 1.510.

          We are using the Job-DSL-Plugin which I believe will bulk delete jobs.

          Emil Sit added a comment - We have seen this issue as well. We are using Jenkins 1.510. We are using the Job-DSL-Plugin which I believe will bulk delete jobs.
          Emil Sit made changes -
          Attachment New: jenkins-18169-thread-dump-redacted.txt [ 23743 ]
          Emil Sit made changes -
          Component/s New: parameterized-trigger [ 15592 ]

          Emil Sit added a comment -

          Some further debugging with jstack reveals that there is a deadlock caused by the RenameListener in the Parameterized Trigger plugin. (See attached jstack output.)

          For each doDelete request that comes in, the thread first holds a lock on itself (via synchronized) in the delete method on Project (extends Job). Each job then winds up calling the RenameListener in the ParameterizedTrigger plugin. The RenameListener iterates over all projects, calling getPublishersList which is also synchronized.

          When delete A and delete B come in at the same time, job A and B are locked, and then B's RenameListener blocks on A and vice versa.

          It looks like getPublishersList() is synchronized just for safety in initializing the publishers list. A potential fix might be to use AtomicReference instead.

          Note that any fix would need to be applied to the other lazy initialized fields in Project (getBuildersList or getBuildWrappersList).

          Emil Sit added a comment - Some further debugging with jstack reveals that there is a deadlock caused by the RenameListener in the Parameterized Trigger plugin. (See attached jstack output.) For each doDelete request that comes in, the thread first holds a lock on itself (via synchronized) in the delete method on Project (extends Job). Each job then winds up calling the RenameListener in the ParameterizedTrigger plugin. The RenameListener iterates over all projects, calling getPublishersList which is also synchronized. When delete A and delete B come in at the same time, job A and B are locked, and then B's RenameListener blocks on A and vice versa. It looks like getPublishersList() is synchronized just for safety in initializing the publishers list. A potential fix might be to use AtomicReference instead. Note that any fix would need to be applied to the other lazy initialized fields in Project (getBuildersList or getBuildWrappersList).
          Emil Sit made changes -
          Attachment New: jenkins-18169-jstack-deadlock.txt [ 23744 ]

          Marcus Jacobsson added a comment -

          I noticed that this is probably the same as https://issues.jenkins-ci.org/browse/JENKINS-18199

          Marcus Jacobsson added a comment - I noticed that this is probably the same as https://issues.jenkins-ci.org/browse/JENKINS-18199

          Marcus Jacobsson added a comment -

          stack trace shown in Javamelody when deleting logs from two builds in prallel

          Marcus Jacobsson added a comment - stack trace shown in Javamelody when deleting logs from two builds in prallel
          Marcus Jacobsson made changes -
          Attachment New: build2.txt [ 23754 ]

            Unassigned Unassigned
            hanabishi Marcus Jacobsson
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: