Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30271

Jenkins job hangs and cannot be killed after aborting DSL job

    XMLWordPrintable

Details

    Description

      Running job-dsl-plugin off the master branch (78ae0c67c3d0b3ce630cca0987c5fba869c8dfb1) plus 3 of my outstanding pull requests folded in. I don't think that this issue is related to my changes (which are about adding new DSL configuration options for plugins)

      Steps to reproduce:

      1. Run a job with a Process DSL Script task (ideally long-running)
      2. Abort job while it's executing. Job will not stop, but will finish normally.
      3. Run the job again. Job will hang and cannot be aborted.

      Interestingly, Thread.interrupt() from the script console will not kill it. Using Monitoring plugin's kill feature does kill the job, but the next run will still hang. Only a restart of the Jenkins master fixes the problem.

      UPDATE: also happens without aborting jobs, making this more than minor. BTW, I am running this on two masters and it happened only once on one and a lot on the other.

      Stacktrace for the stuck job:

      Executor #1 for master : executing DSL Job Builder #108
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
      org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:268)
      org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:193)
      org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:231)
      org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:747)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.createPojoSite(CallSiteArray.java:109)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:150)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.checkParameterName(BuildParametersContext.groovy:270)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.simpleParam(BuildParametersContext.groovy:187)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.this$3$simpleParam(BuildParametersContext.groovy)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext$this$3$simpleParam.callCurrent(Unknown Source)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext$this$3$simpleParam.callCurrent(Unknown Source)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.stringParam(BuildParametersContext.groovy:179)
      javaposse.jobdsl.dsl.helpers.BuildParametersContext.stringParam(BuildParametersContext.groovy)
      sun.reflect.GeneratedMethodAccessor459.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11_closure20.doCall(PipelineJobsBuilder.groovy:634)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.invoke(Unknown Source)
      org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
      org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:82)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11.doCall(PipelineJobsBuilder.groovy:633)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10_closure11.doCall(PipelineJobsBuilder.groovy)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:54)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
      javaposse.jobdsl.dsl.ContextHelper.executeInContext(ContextHelper.groovy:14)
      javaposse.jobdsl.dsl.ContextHelper$executeInContext.call(Unknown Source)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      javaposse.jobdsl.dsl.ContextHelper$executeInContext.call(Unknown Source)
      javaposse.jobdsl.dsl.Job.parameters(Job.groovy:468)
      sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
      PipelineJobsBuilder$_run_closure1_closure9_closure10.doCall(PipelineJobsBuilder.groovy:632)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.with(DefaultGroovyMethods.java:196)
      org.codehaus.groovy.runtime.dgm$926.invoke(Unknown Source)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:313)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      javaposse.jobdsl.dsl.JobParent.processJob(JobParent.groovy:108)
      sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
      javaposse.jobdsl.dsl.JobParent.freeStyleJob(JobParent.groovy:42)
      sun.reflect.GeneratedMethodAccessor637.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
      org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
      PipelineJobsBuilder$_run_closure1_closure9.doCall(PipelineJobsBuilder.groovy:615)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.doMethodInvoke(Unknown Source)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      PipelineJobsBuilder$_run_closure1.doCall(PipelineJobsBuilder.groovy:589)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:497)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
      org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.Closure.call(Closure.java:415)
      groovy.lang.Closure.call(Closure.java:428)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
      org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
      org.codehaus.groovy.runtime.dgm$170.doMethodInvoke(Unknown Source)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
      groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
      groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
      org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
      org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      PipelineJobsBuilder.run(PipelineJobsBuilder.groovy:586)
      javaposse.jobdsl.dsl.DslScriptLoader.runDslEngineForParent(DslScriptLoader.java:80)
      javaposse.jobdsl.dsl.DslScriptLoader.runDslEngine(DslScriptLoader.java:123)
      javaposse.jobdsl.plugin.ExecuteDslScripts.perform(ExecuteDslScripts.java:216)
      hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
      hudson.model.Build$BuildExecution.build(Build.java:203)
      hudson.model.Build$BuildExecution.doRun(Build.java:160)
      hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
      hudson.model.Run.execute(Run.java:1741)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:98)
      hudson.model.Executor.run(Executor.java:374)

      Attachments

        1. config.xml
          5 kB
        2. job-dsl.hpi
          2.33 MB
        3. Stuck_seedjob_stacktrace.txt
          20 kB
        4. Thread Dump.txt
          51 kB

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-30621 Job falls to lock on job-dsl view generation

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-21249 Update Groovy

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 24 older comments
            daspilker Daniel Spilker added a comment -

            sebhoss that is a different problem with a different stack trace.

            Can someone reproduce the problem on Jenkins 2.0? If not, I'm going to close this ticket.

            daspilker Daniel Spilker added a comment - sebhoss that is a different problem with a different stack trace. Can someone reproduce the problem on Jenkins 2.0? If not, I'm going to close this ticket.
            pianoroy Roy Tinker added a comment -

            I'm having the exact problem and stack trace Sebastian pasted. Freestyle builds are hanging. This is Jenkins 2.53 on JDK7.

            pianoroy Roy Tinker added a comment - I'm having the exact problem and stack trace Sebastian pasted. Freestyle builds are hanging. This is Jenkins 2.53 on JDK7.
            akom Alexander Komarov added a comment -

            daspilker I have not reproduced the problem in 2.x but mostly because I'm too scared to try to kill a DSL run, I even have a red notice against that in the description.  It's quite possible that it's fixed because I haven't had it happen without aborting anymore.

            akom Alexander Komarov added a comment - daspilker I have not reproduced the problem in 2.x but mostly because I'm too scared to try to kill a DSL run, I even have a red notice against that in the description.  It's quite possible that it's fixed because I haven't had it happen without aborting anymore.
            daspilker Daniel Spilker added a comment -

            Groovy has been updated from 2.4.8 to 2.4.11 in Jenkins 2.61. The issue should be fixed, see https://issues.apache.org/jira/browse/GROOVY-8067. Can anyone reproduce the problem in 2.61 or later? If not, I'm going to close this ticket.

            daspilker Daniel Spilker added a comment - Groovy has been updated from 2.4.8 to 2.4.11 in Jenkins 2.61. The issue should be fixed, see https://issues.apache.org/jira/browse/GROOVY-8067. Can anyone reproduce the problem in 2.61 or later? If not, I'm going to close this ticket.
            daspilker Daniel Spilker added a comment -

            I'm closing the issue because the problem does not occur with newer versions of Jenkins and Job DSL. Please re-open if you can reproduce the problem.

            daspilker Daniel Spilker added a comment - I'm closing the issue because the problem does not occur with newer versions of Jenkins and Job DSL. Please re-open if you can reproduce the problem.

            People

              daspilker Daniel Spilker
              akom Alexander Komarov
              Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: