Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22525

"incompatible InnerClasses attribute" error in IBM J9 VM

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • windows 2008 R2, Tomcat 6.0.32, IBM Java 1.6 SR15

      When starting Jenkins, it always happen that a Slave disappears. The node is simply vanished.
      I have tried looking for specific errors in the log without success, even looking at all log levels. There is an error while unmarshalling an xml, but unsure if it is related. See the stack trace below in any case.

      I have to note that only one disappears, not all.

      The curious thing comes that if I click "Reload configuration from disk", the disappeared slave is loaded correctly this time (the configuration was in the xml there all along, and I'm not overwriting it).

      Information about my instance:
      Version: 1.547 (happened with 1.532 as well)
      java.vm.info JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr15-20131016_170922 (JIT enabled, AOT enabled)>J9VM - 20131016_170922>JIT - r9_20130920_46510ifx2>GC - GA24_Java6_SR15_20131016_1337_B170922 (happened with an older version as well, tried updating. I'm aware that IBM Java is not officially supported).
      Using tomcat 6.0.32, on a windows 2008 R2 Standard.

      Possible related bug: https://issues.jenkins-ci.org/browse/JENKINS-21867
      (Also on Windows 2008)

      Error that might be related:
      FINE hudson.util.RobustReflectionConverter addErrorInContext
      Failed to load
      java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"
      at java.lang.Class.getDeclaringClass(Class.java:781)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:46)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:89)
      at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:98)
      at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:134)
      at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:43)
      at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:79)
      at java.lang.reflect.Field.getGenericType(Field.java:236)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:207)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:165)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:122)
      at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
      at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:83)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:74)
      at hudson.slaves.NodeList$ConverterImpl.unmarshal(NodeList.java:162)
      at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:338)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:328)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:274)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:221)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061)
      at hudson.util.XStream2.unmarshal(XStream2.java:109)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045)
      at hudson.XmlFile.unmarshal(XmlFile.java:163)
      at jenkins.model.Jenkins$17.run(Jenkins.java:2545)
      at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:906)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:929)
      at java.lang.Thread.run(Thread.java:761)

        1. Queue-compile.javap
          208 kB
        2. Queue-package.javap
          211 kB
        3. Queue.diff
          116 kB

          [JENKINS-22525] "incompatible InnerClasses attribute" error in IBM J9 VM

          Oleg Nenashev added a comment -

          Most probably, the referenced log is unrelated.
          Could you provide the slave's configuration?

          Oleg Nenashev added a comment - Most probably, the referenced log is unrelated. Could you provide the slave's configuration?

          John Tal added a comment - - edited

          Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade.

          Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents.

          With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.

          John Tal added a comment - - edited Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade. Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents. With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.

          Pery Wing added a comment -

          I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost.
          Some time will encounter exception as this ticket show when create the slave node.

          I finally install the 1.528 seems works well for me.

          I also think that a defect in newer vesion

          My isntall evn: RHEL+tomcat8

          Pery Wing added a comment - I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost. Some time will encounter exception as this ticket show when create the slave node. I finally install the 1.528 seems works well for me. I also think that a defect in newer vesion My isntall evn: RHEL+tomcat8

          Daniel Beck added a comment -

          Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?

          Daniel Beck added a comment - Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?

          We first encounter this situation from about Oct/13

          The jvm info of my installation:

          JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA

          Pei-Tang Huang added a comment - We first encounter this situation from about Oct/13 The jvm info of my installation: JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA

          Pei-Tang Huang added a comment - - edited

          I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!

          Pei-Tang Huang added a comment - - edited I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!

          Gabriel Lopez added a comment -

          Hi. Sorry for the delay.
          Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration.

          John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is.

          Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said.

          While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script (https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks.

          My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.

          Gabriel Lopez added a comment - Hi. Sorry for the delay. Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration. John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is. Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said. While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script ( https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script ), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks. My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.

          Oleg Nenashev added a comment -

          There're other issues with classloading on IBM Java. This issue is likely caused by IBM's j9 like it has been mentioned by Pei-Tang and Gabriel.

          I'll create a master-issue, which will reference all others

          Oleg Nenashev added a comment - There're other issues with classloading on IBM Java. This issue is likely caused by IBM's j9 like it has been mentioned by Pei-Tang and Gabriel. I'll create a master-issue, which will reference all others

          Martin Kutter added a comment -

          We're running Jenkins on IBM JDK on AIX.

          I haven't met this issue with Jenkins-1.532.2-LTS - however, I've seen this issue with 1.565. Maybe this helps narrowing down the issue.

          System details:

          AIX 7.1
          IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc-32 20140106_181350 (JIT enabled, AOT enabled)
          Tomcat 1.0.50

          Martin Kutter added a comment - We're running Jenkins on IBM JDK on AIX. I haven't met this issue with Jenkins-1.532.2-LTS - however, I've seen this issue with 1.565. Maybe this helps narrowing down the issue. System details: AIX 7.1 IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc-32 20140106_181350 (JIT enabled, AOT enabled) Tomcat 1.0.50

          Martin Kutter added a comment - - edited

          In my case, reloading the configuration from disk also helps.

          Before I reload the configuration from disk, Jenkins complains that there is Old data on disk.
          It displays the following:

          Type: hudson.model.Hudson
          Name:
          Error: IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue", ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference 
          ---- Debugging information ---- 
          reference : ../../../../../../../slave/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator 
          ------------------------------- 
          message : Could not call hudson.EnvVars.readObject() 
          cause-exception : com.thoughtworks.xstream.converters.ConversionException 
          cause-message : Invalid reference 
          class : hudson.EnvVars 
          required-type : hudson.EnvVars 
          converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter 
          path : /hudson/slaves/slave[2]/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator 
          line number : 243 
          -------------------------------, 
          InstantiationError: null, MissingFieldException: No field 'hudson.tools.ToolLocationNodeProperty' found in class 'hudson.model.Hudson'
          

          Martin Kutter added a comment - - edited In my case, reloading the configuration from disk also helps. Before I reload the configuration from disk, Jenkins complains that there is Old data on disk. It displays the following: Type: hudson.model.Hudson Name: Error: IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue", ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference ---- Debugging information ---- reference : ../../../../../../../slave/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator ------------------------------- message : Could not call hudson.EnvVars.readObject() cause-exception : com.thoughtworks.xstream.converters.ConversionException cause-message : Invalid reference class : hudson.EnvVars required-type : hudson.EnvVars converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter path : /hudson/slaves/slave[2]/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator line number : 243 -------------------------------, InstantiationError: null, MissingFieldException: No field 'hudson.tools.ToolLocationNodeProperty' found in class 'hudson.model.Hudson'

          Daniel Beck added a comment -

          IBM JDK on AIX is probably one of the worst supported environments for Jenkins. Consider moving the master node to e.g. Linux with Oracle JDK 7, which should be fairly reliable.

          Daniel Beck added a comment - IBM JDK on AIX is probably one of the worst supported environments for Jenkins. Consider moving the master node to e.g. Linux with Oracle JDK 7, which should be fairly reliable.

          Martin Kutter added a comment - - edited

          The ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference above is probably unrelated: It looks like my config had invalid references. After cleaning them up, this message disappeared.

          However, the IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" could be the reason for this issue.

          I reproduced the issue with Jenkins 1.554.3 (LTS), 1.565 and 1.570 with the following combinations.

          • Jenkins 1.565 on IBM JDK 6 SR 16 on AIX 7.1
          • Jenkins 1.565 on IBM JDK 7 SR 6 on AIX 7.1
          • Jenkins 1.565 on IBM JDK 7 SR 7 (latest) on AIX 7.1
          • Jenkins 1.554.3 on IBM JDK 7 (build 2.6) on Windows 7
          • Jenkins 1.565 on IBM JDK 7 (build 2.6) on Windows 7
          • Jenkins 1.570 on IBM JDK 7 (build 2.6) on Windows 7
          • Jenkins 1.570 (re-built from source on IBM JDK 6) on IBM JDK 6 on Windows 7

          The issue does not occur on Jenkins-1.532.2 (LTS).

          In my case, the "missing" slave is always the first slave in the config.xml, so I would assume that loading a Slave from disk (and creating a hudsom.model.Queue$ItemList) does not work the first time,
          but works on subsequent tries.

          I've also found that the same error is thrown when creating the first slave node on a fresh jenkins instance (on save / doCreateItem):

           
          Stack trace (excerpt):
          
          Caused by: java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"
          	at java.lang.Class.getDeclaringClass(Class.java:821)
          	at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:52)
          	at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:95)
          	at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
          	at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
          	at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
          	at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:85)
          	at java.lang.reflect.Field.getGenericType(Field.java:237)
          	at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:224)
          	at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182)
          	at com.thoughtworks.xstream.mapper.AnnotationMapper.serializedClass(AnnotationMapper.java:129)
          	at hudson.util.xstream.MapperDelegate.serializedClass(MapperDelegate.java:39)
          	at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:26)
          	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:62)
          	at hudson.slaves.NodeList$ConverterImpl.marshal(NodeList.java:151)
          
          

          The slave is saved nonetheless.

          Subsequent slaves can be created without this error.

          Martin Kutter added a comment - - edited The ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference above is probably unrelated: It looks like my config had invalid references. After cleaning them up, this message disappeared. However, the IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" could be the reason for this issue. I reproduced the issue with Jenkins 1.554.3 (LTS), 1.565 and 1.570 with the following combinations. Jenkins 1.565 on IBM JDK 6 SR 16 on AIX 7.1 Jenkins 1.565 on IBM JDK 7 SR 6 on AIX 7.1 Jenkins 1.565 on IBM JDK 7 SR 7 (latest) on AIX 7.1 Jenkins 1.554.3 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.565 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.570 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.570 (re-built from source on IBM JDK 6) on IBM JDK 6 on Windows 7 The issue does not occur on Jenkins-1.532.2 (LTS). In my case, the "missing" slave is always the first slave in the config.xml, so I would assume that loading a Slave from disk (and creating a hudsom.model.Queue$ItemList) does not work the first time, but works on subsequent tries. I've also found that the same error is thrown when creating the first slave node on a fresh jenkins instance (on save / doCreateItem): Stack trace (excerpt): Caused by: java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" at java.lang.Class.getDeclaringClass(Class.java:821) at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:52) at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:95) at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105) at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140) at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:85) at java.lang.reflect.Field.getGenericType(Field.java:237) at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:224) at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182) at com.thoughtworks.xstream.mapper.AnnotationMapper.serializedClass(AnnotationMapper.java:129) at hudson.util.xstream.MapperDelegate.serializedClass(MapperDelegate.java:39) at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:26) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:62) at hudson.slaves.NodeList$ConverterImpl.marshal(NodeList.java:151) The slave is saved nonetheless. Subsequent slaves can be created without this error.

          Martin Kutter added a comment -

          The error does not appear in Jenkins <= 1.535

          Looks like it has been introduced in 1.536.

          Martin Kutter added a comment - The error does not appear in Jenkins <= 1.535 Looks like it has been introduced in 1.536.

          Martin Kutter added a comment -

          I opened a support request at IBM. I'll update the issue when appropriate.

          Martin Kutter added a comment - I opened a support request at IBM. I'll update the issue when appropriate.

          Jesse Glick added a comment -

          The loss of slave definitions is just the most obvious symptom of the problem with Queue$ItemList, whatever that is.

          Jesse Glick added a comment - The loss of slave definitions is just the most obvious symptom of the problem with Queue$ItemList , whatever that is.

          Jesse Glick added a comment -

          I suspect the problem is related to @WithBridgeMethods or some other bytecode manipulation tricks done in Jenkins; the Eclipse Java compiler also is broken by Queue. It is TBD whether our reworked bytecode is actually invalid according to the JVM spec, or whether it is simply unusual and confuses tools which were not tested against this situation.

          Jesse Glick added a comment - I suspect the problem is related to @WithBridgeMethods or some other bytecode manipulation tricks done in Jenkins; the Eclipse Java compiler also is broken by Queue . It is TBD whether our reworked bytecode is actually invalid according to the JVM spec, or whether it is simply unusual and confuses tools which were not tested against this situation.

          Jesse Glick added a comment -

          Easily reproduced from tests:

          JAVA_HOME=/…/ibm-jdk-71 mvn -f test -Dtest=QueueTest\#testFlyweightTasksWithoutMasterExecutors surefire:test
          

          (Though in this case it does not cause the test to fail.)

          Jesse Glick added a comment - Easily reproduced from tests: JAVA_HOME=/…/ibm-jdk-71 mvn -f test -Dtest=QueueTest\#testFlyweightTasksWithoutMasterExecutors surefire:test (Though in this case it does not cause the test to fail.)

          Jesse Glick added a comment -

          Also reproducible from

          public class Main {
              public static void main(String[] args) throws Exception {
                  System.out.println(Class.forName("hudson.model.Queue$ItemList").getEnclosingClass());
              }
          }
          

          run against core/target/classes, if you have done mvn -f core -DskipTests clean package, but not if the last goal is merely compile. javap confirms that the bytecode of Queue.class is substantially different after the packaging step, presumably due to the bridge method injector running on scheduleMaintenance, though the InnerClasses attribute is not different. In both cases there is no mention of ItemList in InnerClasses, which would seem to violate the JVM spec since it is clearly used, for example as the type of the buildables field. There is a difference in how fields are named, for example

          -         1: getfield      #2                  // Field buildables:Lhudson/model/Queue$ItemList;
          +         1: getfield      #2                  // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;
          

          Jesse Glick added a comment - Also reproducible from public class Main { public static void main( String [] args) throws Exception { System .out.println( Class .forName( "hudson.model.Queue$ItemList" ).getEnclosingClass()); } } run against core/target/classes , if you have done mvn -f core -DskipTests clean package , but not if the last goal is merely compile . javap confirms that the bytecode of Queue.class is substantially different after the packaging step, presumably due to the bridge method injector running on scheduleMaintenance , though the InnerClasses attribute is not different. In both cases there is no mention of ItemList in InnerClasses , which would seem to violate the JVM spec since it is clearly used, for example as the type of the buildables field. There is a difference in how fields are named, for example - 1: getfield #2 // Field buildables:Lhudson/model/Queue$ItemList; + 1: getfield #2 // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;

          Jesse Glick added a comment -

          Attached diff before; now attaching original files for completeness.

          Jesse Glick added a comment - Attached diff before; now attaching original files for completeness.

          After looking further into javap source code, I discovered why the following difference in the output happens:

          -         1: getfield      #2                  // Field buildables:Lhudson/model/Queue$ItemList;
          +         1: getfield      #2                  // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;
          

          When printing the content of constant pool for these instructions, javap attempts to simplify the reference within the same class by omitting the class name as follows:

                  int tag = cpInfo.getTag();
                  switch (tag) {
                      case CONSTANT_Methodref:
                      case CONSTANT_InterfaceMethodref:
                      case CONSTANT_Fieldref:
                          // simplify references within this class
                          CPRefInfo ref = (CPRefInfo) cpInfo;
                          try {
                              if (ref.class_index == classFile.this_class)
                                   cpInfo = classFile.constant_pool.get(ref.name_and_type_index);
                          } catch (ConstantPool.InvalidIndex e) {
                              // ignore, for now
                          }
                  }
                  print(tagName(tag) + " " + stringValue(cpInfo));
          

          This check compares the constant pool index, not the actual content. The fieldref constant pool remains unchanged at #283.#624 before/after transformation, but classFile.this_class that references the class name itself changes from #283 to #400.

          The constant pool entry #400 is "Class #891", just like #283. IOW, javac produces Queue.class that contains two identical constant pool entries, which is suprising.

          It is still unclear as to whether this difference in the class file triggers the "incompatible InnerClasses attribute" problem.

          I think the next step is to modify class file to insert InnerClass attribute for Queue$ItemList, as the lack of this entry does seem to violate JVM spec. If that is the case, the question remains as to why J9VM was happy with the class before transformation.

          Kohsuke Kawaguchi added a comment - After looking further into javap source code, I discovered why the following difference in the output happens: - 1: getfield #2 // Field buildables:Lhudson/model/Queue$ItemList; + 1: getfield #2 // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList; When printing the content of constant pool for these instructions, javap attempts to simplify the reference within the same class by omitting the class name as follows: int tag = cpInfo.getTag(); switch (tag) { case CONSTANT_Methodref: case CONSTANT_InterfaceMethodref: case CONSTANT_Fieldref: // simplify references within this class CPRefInfo ref = (CPRefInfo) cpInfo; try { if (ref.class_index == classFile.this_class) cpInfo = classFile.constant_pool.get(ref.name_and_type_index); } catch (ConstantPool.InvalidIndex e) { // ignore, for now } } print(tagName(tag) + " " + stringValue(cpInfo)); This check compares the constant pool index, not the actual content. The fieldref constant pool remains unchanged at #283.#624 before/after transformation, but classFile.this_class that references the class name itself changes from #283 to #400. The constant pool entry #400 is "Class #891", just like #283. IOW, javac produces Queue.class that contains two identical constant pool entries, which is suprising. It is still unclear as to whether this difference in the class file triggers the "incompatible InnerClasses attribute" problem. I think the next step is to modify class file to insert InnerClass attribute for Queue$ItemList , as the lack of this entry does seem to violate JVM spec. If that is the case, the question remains as to why J9VM was happy with the class before transformation.

          Just for the sake of completeness I manually patched Queue.class after transformation to have its classFile.this_class index points to #283, not #400, and now J9 VM is happy.

          Wow.

          Kohsuke Kawaguchi added a comment - Just for the sake of completeness I manually patched Queue.class after transformation to have its classFile.this_class index points to #283, not #400, and now J9 VM is happy. Wow.

          Relevant StackOverflow thread and BugParade entry, which indicates the fix to javac made it into JDK8 b92.

          Looks like J9 VM is failing to handle this rare situation correctly. I think I'll work around this problem in bridge method injector to reshuffle constant pools and eliminate duplicates.

          Kohsuke Kawaguchi added a comment - Relevant StackOverflow thread and BugParade entry , which indicates the fix to javac made it into JDK8 b92. Looks like J9 VM is failing to handle this rare situation correctly. I think I'll work around this problem in bridge method injector to reshuffle constant pools and eliminate duplicates.

          Jesse Glick added a comment -

          Jesse Glick added a comment - More useful bug link: https://bugs.openjdk.java.net/browse/JDK-5053846

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/pom.xml
          pom.xml
          http://jenkins-ci.org/commit/jenkins/4e23793f62aa5019226f2ba96f8063f01affeb92
          Log:
          [FIXED JENKINS-22525]

          Integrated a bridge-method-injector that rebuilds the constant pool from
          scratch.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml pom.xml http://jenkins-ci.org/commit/jenkins/4e23793f62aa5019226f2ba96f8063f01affeb92 Log: [FIXED JENKINS-22525] Integrated a bridge-method-injector that rebuilds the constant pool from scratch.

          Kohsuke Kawaguchi added a comment - Code change in bridge-method-injector

          Jesse Glick added a comment -

          Jesse Glick added a comment - Also it seems like https://bugs.openjdk.java.net/browse/JDK-6746955 and https://bugs.openjdk.java.net/browse/JDK-8015927 might be the more relevant issues.

          Changed the title to accurately capture the issue.

          Kohsuke Kawaguchi added a comment - Changed the title to accurately capture the issue.

          Kohsuke Kawaguchi added a comment - - edited

          As for ItemList not in the InnerClasses attribute, I'm discovering with the following test case that if a nested/inner class is marked as private, it does not get recorded in the InnerClasses attribute:

          public class App 
          {
              private static class Foo {}
              private class Bar {}
          
              static class Alpha {}
              class Bravo {}
          
              public static void main( String[] args )
              {
                  System.out.println(Foo.class.getEnclosingClass());
                  System.out.println(Bar.class.getEnclosingClass());
          
                  System.out.println(Alpha.class.getEnclosingClass());
                  System.out.println(Bravo.class.getEnclosingClass());
              }
          }
          
          Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App.class
            Last modified Jul 31, 2014; size 1158 bytes
            MD5 checksum 264ed3611a578e1a4c8e2d4e0ea8b378
            Compiled from "App.java"
          public class test.App
            SourceFile: "App.java"
            InnerClasses: 
                 #11= #8 of #9; //Bravo=class test/App$Bravo of class test/App
                 static #13= #7 of #9; //Alpha=class test/App$Alpha of class test/App
            RuntimeInvisibleAnnotations:
              0: #57()
            minor version: 0
            major version: 49
            flags: ACC_PUBLIC, ACC_SUPER
          
          Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Foo.class
            Last modified Jul 31, 2014; size 303 bytes
            MD5 checksum 3f4b082e8d8c5cbf85d4afeb7807b253
            Compiled from "App.java"
          class test.App$Foo
            SourceFile: "App.java"
            minor version: 0
            major version: 49
            flags: ACC_SUPER
          
          Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Alpha.class
            Last modified Jul 31, 2014; size 309 bytes
            MD5 checksum df99610f5b28798c5f07f308b749df89
            Compiled from "App.java"
          class test.App$Alpha
            SourceFile: "App.java"
            InnerClasses: 
                 static #10= #2 of #16; //Alpha=class test/App$Alpha of class test/App
            minor version: 0
            major version: 49
            flags: ACC_SUPER
          

          This seems to violate JVM spec of the InnerClass attribute. J9 VM accepts this without reporting an error. In this case I don't understand how JVM can correctly determine that App$Foo has an enclosing class. From the byte code, there's no way to distinguish this vs compiling the following class:

          class App {}
          class App$Foo {} // '$' is a legal character in a class name in Java
          

          Kohsuke Kawaguchi added a comment - - edited As for ItemList not in the InnerClasses attribute, I'm discovering with the following test case that if a nested/inner class is marked as private, it does not get recorded in the InnerClasses attribute: public class App { private static class Foo {} private class Bar {} static class Alpha {} class Bravo {} public static void main( String [] args ) { System .out.println(Foo. class. getEnclosingClass()); System .out.println(Bar. class. getEnclosingClass()); System .out.println(Alpha. class. getEnclosingClass()); System .out.println(Bravo. class. getEnclosingClass()); } } Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App.class Last modified Jul 31, 2014; size 1158 bytes MD5 checksum 264ed3611a578e1a4c8e2d4e0ea8b378 Compiled from "App.java" public class test.App SourceFile: "App.java" InnerClasses: #11= #8 of #9; //Bravo=class test/App$Bravo of class test/App static #13= #7 of #9; //Alpha=class test/App$Alpha of class test/App RuntimeInvisibleAnnotations: 0: #57() minor version: 0 major version: 49 flags: ACC_PUBLIC, ACC_SUPER Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Foo.class Last modified Jul 31, 2014; size 303 bytes MD5 checksum 3f4b082e8d8c5cbf85d4afeb7807b253 Compiled from "App.java" class test.App$Foo SourceFile: "App.java" minor version: 0 major version: 49 flags: ACC_SUPER Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Alpha.class Last modified Jul 31, 2014; size 309 bytes MD5 checksum df99610f5b28798c5f07f308b749df89 Compiled from "App.java" class test.App$Alpha SourceFile: "App.java" InnerClasses: static #10= #2 of #16; //Alpha=class test/App$Alpha of class test/App minor version: 0 major version: 49 flags: ACC_SUPER This seems to violate JVM spec of the InnerClass attribute . J9 VM accepts this without reporting an error. In this case I don't understand how JVM can correctly determine that App$Foo has an enclosing class. From the byte code, there's no way to distinguish this vs compiling the following class: class App {} class App$Foo {} // '$' is a legal character in a class name in Java

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3569
          [FIXED JENKINS-22525] (Revision 4e23793f62aa5019226f2ba96f8063f01affeb92)

          Result = SUCCESS
          kohsuke : 4e23793f62aa5019226f2ba96f8063f01affeb92
          Files :

          • pom.xml
          • core/pom.xml
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3569 [FIXED JENKINS-22525] (Revision 4e23793f62aa5019226f2ba96f8063f01affeb92) Result = SUCCESS kohsuke : 4e23793f62aa5019226f2ba96f8063f01affeb92 Files : pom.xml core/pom.xml changelog.html

          Martin Kutter added a comment -

          The fix seems to work around the issue - I've been unable to reproduce the error with a build from sources with the fix applied.

          Runtime: Windows 7, IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)

          Martin Kutter added a comment - The fix seems to work around the issue - I've been unable to reproduce the error with a build from sources with the fix applied. Runtime: Windows 7, IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)

          dan tran added a comment -

          dont see it at 1.575 http://jenkins-ci.org/changelog.

          dan tran added a comment - dont see it at 1.575 http://jenkins-ci.org/changelog .

          Jesse Glick added a comment -

          @dantran because it was committed toward 1.576.

          Jesse Glick added a comment - @dantran because it was committed toward 1.576.

          Daniel Beck added a comment -

          Literally one click away: "Upcoming changes"

          Daniel Beck added a comment - Literally one click away: "Upcoming changes"

          dan tran added a comment -

          thanks, I am now running 1.576-SNAPSHOT. Good to see this issue is fixed

          dan tran added a comment - thanks, I am now running 1.576-SNAPSHOT. Good to see this issue is fixed

          Martin Kutter added a comment -

          IBM has tracked the issue with "incompatible inner Class attribute" as APAR IV62298 - see http://www-01.ibm.com/support/docview.wss?uid=swg1IV62298
          I've received an iFix for Java 7SR7 from IBM, which fixes the issue.

          The fix will be applied in the upcoming IBM JDK 7 SR 8.

          Martin Kutter added a comment - IBM has tracked the issue with "incompatible inner Class attribute" as APAR IV62298 - see http://www-01.ibm.com/support/docview.wss?uid=swg1IV62298 I've received an iFix for Java 7SR7 from IBM, which fixes the issue. The fix will be applied in the upcoming IBM JDK 7 SR 8.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/pom.xml
          pom.xml
          http://jenkins-ci.org/commit/jenkins/e4dc3b4210fb6afd80ddf138cafc6bfcb866df5f
          Log:
          [FIXED JENKINS-22525]

          Integrated a bridge-method-injector that rebuilds the constant pool from
          scratch.

          (cherry picked from commit 4e23793f62aa5019226f2ba96f8063f01affeb92)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml pom.xml http://jenkins-ci.org/commit/jenkins/e4dc3b4210fb6afd80ddf138cafc6bfcb866df5f Log: [FIXED JENKINS-22525] Integrated a bridge-method-injector that rebuilds the constant pool from scratch. (cherry picked from commit 4e23793f62aa5019226f2ba96f8063f01affeb92) Conflicts: changelog.html

            kohsuke Kohsuke Kawaguchi
            gabriel Gabriel Lopez
            Votes:
            6 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: