Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50671

Jenkins won't start after upgrading workflow-cps plugin to 2.47

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Duplicate
    • workflow-cps-plugin
    • None
    • Jenkins 2.114
      workflow-cps plugin 2.47 (upgraded from 2.46)

    Description

      Using workflow-cps version 2.46 and latest version of Jenkins (2.114) and all other installed plugins, everything works fine. After upgrading workflow-cps to 2.47 and restarting, Jenkins hangs on the "Please wait while Jenkins is getting ready to work" page. I let it sit for about 10 minutes, but no change. It usually starts up in less than two minutes.

      Our theme is also not applied, so it appears that the Simple Theme Plugin is not loaded by the time it hangs.

      Reverting workflow-cps to 2.46 manually gets it to start up fine.

      There doesn't appear to be anything interesting in the jenkins.log, and the jenkins.err.log doesn't appear to be present at all unless I'm looking in the wrong location.

      Any advice on where to look for further information?

      Attachments

        Issue Links

          Activity

            nv035674 Nathan Vahrenberg created issue -
            svanoort Sam Van Oort made changes -
            Field Original Value New Value
            Assignee Sam Van Oort [ svanoort ]
            svanoort Sam Van Oort added a comment -

            nv035674 I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html

            Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?

            svanoort Sam Van Oort added a comment - nv035674 I'm sorry to hear that. Could you please grab and attach a Thread dump – link here: https://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html Also, did you also update workflow-job plugin? If so, do you see higher disk activity on the system? This may be a result of one-time migrations of loaded build records to persist completed status. If not, could you please try with both workflow-job and workflow-cps updated?
            svanoort Sam Van Oort made changes -
            Priority Major [ 3 ] Blocker [ 1 ]
            svanoort Sam Van Oort added a comment -

            Sorry, to clarify: a Thread Dump while it is hung.

            svanoort Sam Van Oort added a comment - Sorry, to clarify: a Thread Dump while it is hung.

            Thanks for the quick response svanoort. I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible.

            Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.

            nv035674 Nathan Vahrenberg added a comment - Thanks for the quick response svanoort . I'll grab the thread dump and upload here as soon as I can - I have to try not to disrupt users of this instance too much if possible. Yes, the workflow-job plugin was also updated this morning. It appears that disk total usage (R/W combined) spiked to about 24% around the upgrade (I can try to get more specific metrics if necessary). Right now it's around 2-5% and it looks like that's about the same as what it was before upgrading. I've left the workflow-job plugin upgraded, the only plugin I have reverted to an older version at this moment is workflow-cps.
            nv035674 Nathan Vahrenberg made changes -
            Attachment stacktrace.txt [ 42123 ]
            nv035674 Nathan Vahrenberg added a comment - - edited

            Thread dump: threaddump.txt

            I took a glance at it and it mentions a deadlock, is that what you're looking for?

            nv035674 Nathan Vahrenberg added a comment - - edited Thread dump: threaddump.txt I took a glance at it and it mentions a deadlock, is that what you're looking for?
            nv035674 Nathan Vahrenberg made changes -
            Attachment stacktrace.txt [ 42123 ]
            nv035674 Nathan Vahrenberg made changes -
            Attachment threaddump.txt [ 42124 ]
            svanoort Sam Van Oort added a comment -

            nv035674 Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time.

            The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.

            svanoort Sam Van Oort added a comment - nv035674 Ah, thank you, that helps a lot – It looks like the root cause of your problem is actually https://issues.jenkins-ci.org/browse/JENKINS-50663 which is caused by the Sonar Plugin triggering a deadlock. You should be able to resolve the issue by downgrading sonar to 2.6.1 and can freely upgrade workflow-cps again at the same time. The workflow-cps upgrade appears to be a red herring – probably since this is a deadlock it just so happens that the timing of the startup process was a bit different between restarts so you got a deadlock in one case but not the other.
            svanoort Sam Van Oort added a comment -

            I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).

            svanoort Sam Van Oort added a comment - I'm closing this one as a duplicate since it appears to be caused by an already-identified bug in the sonar plugin – will be happy to re-open if there proves to be more going on than that (just reply back and re-open it).
            svanoort Sam Van Oort made changes -
            Link This issue is related to JENKINS-50663 [ JENKINS-50663 ]
            svanoort Sam Van Oort made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix.

            Thanks again!

            nv035674 Nathan Vahrenberg added a comment - Thanks for the investigation. Sorry to point blame the wrong direction, but it's definitely curious that downgrading the workflow-cps plugin "fixes" it. I'll keep an eye on the other Jira for a fix. Thanks again!

            People

              svanoort Sam Van Oort
              nv035674 Nathan Vahrenberg
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: