Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47105

CpsFlowExecutionTest#loaderReleased times out with jenkins-test-harness 2.25+

XMLWordPrintable

      When moving to plugin parent 2.35, I discovered that CpsFlowExecutionTest#loaderReleased hangs - it loops forever with the following in the console:

      Skipped analysing class java.lang.reflect.Proxy because of java.lang.AssertionError
[...]
Skipped analysing class java.lang.reflect.Proxy because of java.lang.AssertionError
Skipped analysing class hudson.model.Descriptor$PropertyType because of java.lang.AssertionError
[...]
Skipped analysing class hudson.model.Descriptor$PropertyType because of java.lang.AssertionError
Failed analysing class org.kohsuke.stapler.JRebelFacet$ReloaderHook because of java.lang.NoClassDefFoundError: org/zeroturnaround/javarebel/ClassEventListener
Failed to read field static final sun.awt.datatransfer.ToolkitThreadBlockedHandler sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.handler because of java.lang.NoClassDefFoundError: Could not initialize class sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher
Failed to read field public static final hudson.util.jna.GNUCLibrary hudson.util.jna.GNUCLibrary.LIBC because of java.lang.NoClassDefFoundError: Could not initialize class hudson.util.jna.GNUCLibrary
Failed analysing class org.kohsuke.stapler.JRebelFacet$ReloaderHook because of java.lang.NoClassDefFoundError: org/zeroturnaround/javarebel/ClassEventListener
Failed to read field private static final org.jvnet.solaris.libzfs.LibZFS hudson.os.solaris.ZFSProvisioner.libzfs because of java.lang.NoClassDefFoundError: Could not initialize class hudson.os.solaris.ZFSProvisioner
Failed to read field public static final org.jvnet.solaris.libzfs.jna.libzfs org.jvnet.solaris.libzfs.jna.libzfs.LIBZFS because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs
Failed to read field public static final java.lang.String org.jvnet.solaris.libzfs.jna.libzfs.ZFS_MOUNTPOINT_NONE because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs
Failed to read field public static final java.lang.String org.jvnet.solaris.libzfs.jna.libzfs.ZFS_MOUNTPOINT_LEGACY because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs
GC after allocation of size 2249
Failed analysing class org.kohsuke.stapler.JRebelFacet$ReloaderHook because of java.lang.NoClassDefFoundError: org/zeroturnaround/javarebel/ClassEventListener
Failed to read field static final sun.awt.datatransfer.ToolkitThreadBlockedHandler sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher.handler because of java.lang.NoClassDefFoundError: Could not initialize class sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher
Failed to read field public static final hudson.util.jna.GNUCLibrary hudson.util.jna.GNUCLibrary.LIBC because of java.lang.NoClassDefFoundError: Could not initialize class hudson.util.jna.GNUCLibrary
Failed analysing class org.kohsuke.stapler.JRebelFacet$ReloaderHook because of java.lang.NoClassDefFoundError: org/zeroturnaround/javarebel/ClassEventListener
Failed to read field private static final org.jvnet.solaris.libzfs.LibZFS hudson.os.solaris.ZFSProvisioner.libzfs because of java.lang.NoClassDefFoundError: Could not initialize class hudson.os.solaris.ZFSProvisioner
Failed to read field public static final org.jvnet.solaris.libzfs.jna.libzfs org.jvnet.solaris.libzfs.jna.libzfs.LIBZFS because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs
Failed to read field public static final java.lang.String org.jvnet.solaris.libzfs.jna.libzfs.ZFS_MOUNTPOINT_NONE because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs
Failed to read field public static final java.lang.String org.jvnet.solaris.libzfs.jna.libzfs.ZFS_MOUNTPOINT_LEGACY because of java.lang.NoClassDefFoundError: Could not initialize class org.jvnet.solaris.libzfs.jna.libzfs

          [JENKINS-47105] CpsFlowExecutionTest#loaderReleased times out with jenkins-test-harness 2.25+

          Andrew Bayer added a comment - - edited

          So this bugger is a bit of a heisenbug. All these tests are coming from the command line, and with groovy-cps 1.19 (rather than the newest 1.20, since testing that is where I encountered this for the first time when I also bumped the plugin parent) fwiw. If you start from scratch, it consistently spits out that noise (and takes up to 75 seconds, vs the normal 10 or less seconds) with plugin parent 2.35 and jenkins-test-harness 2.27. If I run it with plugin parent 2.35/jenkins-test-harness 2.27 and then change jenkins-test-harness to 2.26, it spits out the output/takes longer again...on the first run. On the second run, or subsequent runs with that combination, it's fine. And what's more, changing jenkins-test-harness back to 2.27 after that, the first run is fine. Sometimes the second run is fine too! But eventually it goes back to noise/longer as well.

          With plugin parent 2.33, the behavior is similar, but the test actually ends up failing as well with jenkins-test-harness 2.27, with:

          [ERROR] Failures:
[ERROR]   CpsFlowExecutionTest.loaderReleased Apparent soft references to org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c: {org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c=java.lang.Thread@48e20a51-group->
java.lang.ThreadGroup@5b8c62df-threads->
[Ljava.lang.Thread;@3194e300-[28]->
java.lang.Thread@5f446887-threadLocals->
java.lang.ThreadLocal$ThreadLocalMap@6c54de59-table->
[Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@77c7e4d-[26]->
java.lang.ThreadLocal$ThreadLocalMap$Entry@443f4fd9-value->
java.lang.ref.SoftReference@4119bed4-referent->
org.codehaus.groovy.reflection.ClassInfo$LocalMap@bd22f03-cache->
[Lorg.codehaus.groovy.reflection.ClassInfo;@5cef4aa1-[0]->
org.codehaus.groovy.reflection.ClassInfo@24526e29-klazz->
java.lang.Class@5f8ae2c5-<changed>->
org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c}; apparent weak references: {org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c=private static final java.util.List org.jenkinsci.plugins.workflow.cps.CpsFlowExecutionTest.LOADERS->
java.util.ArrayList@6cd63943-elementData->
[Ljava.lang.Object;@19153301-[1]->
java.lang.ref.WeakReference@73963a4b-referent->
org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c}

          ...except it passes right away when I switch to jenkins-test-harness 2.26. So...yeah. No idea.

          EDIT: And now it started failing for me with plugin parent 2.33 and jenkins-test-harness 2.26 for no obvious reason.

          EDIT 2: And now it's failing with vanilla plugin parent 2.33 (i.e., with jenkins-test-harness 2.23)! What the hell...

          Andrew Bayer added a comment - - edited So this bugger is a bit of a heisenbug. All these tests are coming from the command line, and with groovy-cps 1.19 (rather than the newest 1.20, since testing that is where I encountered this for the first time when I also bumped the plugin parent) fwiw. If you start from scratch, it consistently spits out that noise (and takes up to 75 seconds, vs the normal 10 or less seconds) with plugin parent 2.35 and jenkins-test-harness 2.27. If I run it with plugin parent 2.35/jenkins-test-harness 2.27 and then change jenkins-test-harness to 2.26, it spits out the output/takes longer again...on the first run. On the second run, or subsequent runs with that combination, it's fine. And what's more, changing jenkins-test-harness back to 2.27 after that, the first run is fine. Sometimes the second run is fine too! But eventually it goes back to noise/longer as well. With plugin parent 2.33, the behavior is similar, but the test actually ends up failing as well with jenkins-test-harness 2.27, with: [ERROR] Failures: [ERROR] CpsFlowExecutionTest.loaderReleased Apparent soft references to org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c: {org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c=java.lang. Thread @48e20a51-group-> java.lang. ThreadGroup @5b8c62df-threads-> [Ljava.lang. Thread ;@3194e300-[28]-> java.lang. Thread @5f446887-threadLocals-> java.lang.ThreadLocal$ThreadLocalMap@6c54de59-table-> [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@77c7e4d-[26]-> java.lang.ThreadLocal$ThreadLocalMap$Entry@443f4fd9-value-> java.lang.ref.SoftReference@4119bed4-referent-> org.codehaus.groovy.reflection.ClassInfo$LocalMap@bd22f03-cache-> [Lorg.codehaus.groovy.reflection.ClassInfo;@5cef4aa1-[0]-> org.codehaus.groovy.reflection.ClassInfo@24526e29-klazz-> java.lang. Class @5f8ae2c5-<changed>-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c}; apparent weak references: {org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c= private static final java.util.List org.jenkinsci.plugins.workflow.cps.CpsFlowExecutionTest.LOADERS-> java.util.ArrayList@6cd63943-elementData-> [Ljava.lang. Object ;@19153301-[1]-> java.lang.ref.WeakReference@73963a4b-referent-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader@25276a4c} ...except it passes right away when I switch to jenkins-test-harness 2.26. So...yeah. No idea. EDIT: And now it started failing for me with plugin parent 2.33 and jenkins-test-harness 2.26 for no obvious reason. EDIT 2: And now it's failing with vanilla plugin parent 2.33 (i.e., with jenkins-test-harness 2.23)! What the hell...

          Andrew Bayer added a comment -

          For what it's worth, added to the environment that this is with JDK 1.8.0_131 on my Mac.

          Andrew Bayer added a comment - For what it's worth, added to the environment that this is with JDK 1.8.0_131 on my Mac.

          Jesse Glick added a comment -

          I am not getting failures or even delayed collection on Linux with JDK 8u131 in current trunk after switching to parent 2.35. IIUC https://github.com/jenkinsci/workflow-cps-plugin/pull/177/commits/c482e6fd40210a3cc3bd32c328c565b98ed89561 is your workaround for now.

          Jesse Glick added a comment - I am not getting failures or even delayed collection on Linux with JDK 8u131 in current trunk after switching to parent 2.35. IIUC https://github.com/jenkinsci/workflow-cps-plugin/pull/177/commits/c482e6fd40210a3cc3bd32c328c565b98ed89561  is your workaround for now.

          Andrew Bayer added a comment -

          jglick fwiw, try running it a few times. Like I said, it's bizarrely inconsistent in when it decides to start having issues. I'm trying concurrency experiments on a testbed I've got right now, since I know I'm pretty memory constrained on my laptop so that could be a trigger...

          ...and yeah, now I'm seeing some delayed collections with parent 2.35, though still not every time, and a run with parent 2.34 going at the same time is fine.

          Andrew Bayer added a comment - jglick fwiw, try running it a few times. Like I said, it's bizarrely inconsistent in when it decides to start having issues. I'm trying concurrency experiments on a testbed I've got right now, since I know I'm pretty memory constrained on my laptop so that could be a trigger... ...and yeah, now I'm seeing some delayed collections with parent 2.35, though still not every time, and a run with parent 2.34 going at the same time is fine.

          Jesse Glick added a comment -

          Saw a failure when running the whole test suite, as opposed to the single test case. Right now I am trying to track down why LoggerRule is not working.

          Jesse Glick added a comment - Saw a failure when running the whole test suite, as opposed to the single test case. Right now I am trying to track down why LoggerRule is not working.

          Jesse Glick added a comment -

          I think I have a lead: the test fails iff trustedShell is run first.

          Jesse Glick added a comment - I think I have a lead: the test fails iff trustedShell is run first.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          Jenkinsfile
          pom.xml
          http://jenkins-ci.org/commit/workflow-cps-plugin/942402131d84b961a268a2100a945b3897f121ff
          Log:
          JENKINS-47105 Checking effect of updated parent.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile pom.xml http://jenkins-ci.org/commit/workflow-cps-plugin/942402131d84b961a268a2100a945b3897f121ff Log: JENKINS-47105 Checking effect of updated parent.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionMemoryTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/bbf99da6739b37bbaaba21993f4b8a3604224969
          Log:
          Merge pull request #178 from jglick/parent-JENKINS-47105

          JENKINS-47105 Fixing test failure in loaderReleased

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/aa4c4e78ecf7...bbf99da6739b

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionMemoryTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/bbf99da6739b37bbaaba21993f4b8a3604224969 Log: Merge pull request #178 from jglick/parent- JENKINS-47105 JENKINS-47105 Fixing test failure in loaderReleased Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/aa4c4e78ecf7...bbf99da6739b

          Andrew Bayer added a comment -

          jglick So should this be resolved?

          Andrew Bayer added a comment - jglick So should this be resolved?

            jglick Jesse Glick
            abayer Andrew Bayer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: