Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50671

Jenkins won't start after upgrading workflow-cps plugin to 2.47

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.114
      workflow-cps plugin 2.47 (upgraded from 2.46)
    • Similar Issues:

      Description

      Using workflow-cps version 2.46 and latest version of Jenkins (2.114) and all other installed plugins, everything works fine. After upgrading workflow-cps to 2.47 and restarting, Jenkins hangs on the "Please wait while Jenkins is getting ready to work" page. I let it sit for about 10 minutes, but no change. It usually starts up in less than two minutes.

      Our theme is also not applied, so it appears that the Simple Theme Plugin is not loaded by the time it hangs.

      Reverting workflow-cps to 2.46 manually gets it to start up fine.

      There doesn't appear to be anything interesting in the jenkins.log, and the jenkins.err.log doesn't appear to be present at all unless I'm looking in the wrong location.

      Any advice on where to look for further information?

        Attachments

          Issue Links

            Activity

            Hide
            svanoort Sam Van Oort added a comment -

            Nathan Vahrenberg I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html

            Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?

            Show
            svanoort Sam Van Oort added a comment - Nathan Vahrenberg I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?
            Hide
            svanoort Sam Van Oort added a comment -

            Sorry, to clarify: a Thread Dump while it is hung.

            Show
            svanoort Sam Van Oort added a comment - Sorry, to clarify: a Thread Dump while it is hung.
            Hide
            nv035674 Nathan Vahrenberg added a comment -

            Thanks for the quick response Sam Van Oort. I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible.

            Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.

            Show
            nv035674 Nathan Vahrenberg added a comment - Thanks for the quick response Sam Van Oort . I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible. Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.
            Hide
            nv035674 Nathan Vahrenberg added a comment - - edited

            Thread dump: threaddump.txt

            I took a glance at it and it mentions a deadlock, is that what you're looking for?

            Show
            nv035674 Nathan Vahrenberg added a comment - - edited Thread dump: threaddump.txt I took a glance at it and it mentions a deadlock, is that what you're looking for?
            Hide
            svanoort Sam Van Oort added a comment -

            Nathan Vahrenberg Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time.

            The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.

            Show
            svanoort Sam Van Oort added a comment - Nathan Vahrenberg Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time. The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.
            Hide
            svanoort Sam Van Oort added a comment -

            I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).

            Show
            svanoort Sam Van Oort added a comment - I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).
            Hide
            nv035674 Nathan Vahrenberg added a comment -

            Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix.

            Thanks again!

            Show
            nv035674 Nathan Vahrenberg added a comment - Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix. Thanks again!

              People

              Assignee:
              svanoort Sam Van Oort
              Reporter:
              nv035674 Nathan Vahrenberg
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: