Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50671

Jenkins won't start after upgrading workflow-cps plugin to 2.47

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • workflow-cps-plugin
    • None
    • Jenkins 2.114
      workflow-cps plugin 2.47 (upgraded from 2.46)

      Using workflow-cps version 2.46 and latest version of Jenkins (2.114) and all other installed plugins, everything works fine. After upgrading workflow-cps to 2.47 and restarting, Jenkins hangs on the "Please wait while Jenkins is getting ready to work" page. I let it sit for about 10 minutes, but no change. It usually starts up in less than two minutes.

      Our theme is also not applied, so it appears that the Simple Theme Plugin is not loaded by the time it hangs.

      Reverting workflow-cps to 2.46 manually gets it to start up fine.

      There doesn't appear to be anything interesting in the jenkins.log, and the jenkins.err.log doesn't appear to be present at all unless I'm looking in the wrong location.

      Any advice on where to look for further information?

          [JENKINS-50671] Jenkins won't start after upgrading workflow-cps plugin to 2.47

          Sam Van Oort added a comment -

          nv035674 I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html

          Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?

          Sam Van Oort added a comment - nv035674 I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?

          Sam Van Oort added a comment -

          Sorry, to clarify: a Thread Dump while it is hung.

          Sam Van Oort added a comment - Sorry, to clarify: a Thread Dump while it is hung.

          Thanks for the quick response svanoort. I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible.

          Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.

          Nathan Vahrenberg added a comment - Thanks for the quick response svanoort . I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible. Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.

          Nathan Vahrenberg added a comment - - edited

          Thread dump: threaddump.txt

          I took a glance at it and it mentions a deadlock, is that what you're looking for?

          Nathan Vahrenberg added a comment - - edited Thread dump: threaddump.txt I took a glance at it and it mentions a deadlock, is that what you're looking for?

          Sam Van Oort added a comment -

          nv035674 Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time.

          The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.

          Sam Van Oort added a comment - nv035674 Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time. The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.

          Sam Van Oort added a comment -

          I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).

          Sam Van Oort added a comment - I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).

          Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix.

          Thanks again!

          Nathan Vahrenberg added a comment - Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix. Thanks again!

            svanoort Sam Van Oort
            nv035674 Nathan Vahrenberg
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: