Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22525

"incompatible InnerClasses attribute" error in IBM J9 VM

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      windows 2008 R2, Tomcat 6.0.32, IBM Java 1.6 SR15
    • Similar Issues:

      Description

      When starting Jenkins, it always happen that a Slave disappears. The node is simply vanished.
      I have tried looking for specific errors in the log without success, even looking at all log levels. There is an error while unmarshalling an xml, but unsure if it is related. See the stack trace below in any case.

      I have to note that only one disappears, not all.

      The curious thing comes that if I click "Reload configuration from disk", the disappeared slave is loaded correctly this time (the configuration was in the xml there all along, and I'm not overwriting it).

      Information about my instance:
      Version: 1.547 (happened with 1.532 as well)
      java.vm.info JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr15-20131016_170922 (JIT enabled, AOT enabled)>J9VM - 20131016_170922>JIT - r9_20130920_46510ifx2>GC - GA24_Java6_SR15_20131016_1337_B170922 (happened with an older version as well, tried updating. I'm aware that IBM Java is not officially supported).
      Using tomcat 6.0.32, on a windows 2008 R2 Standard.

      Possible related bug: https://issues.jenkins-ci.org/browse/JENKINS-21867
      (Also on Windows 2008)

      Error that might be related:
      FINE hudson.util.RobustReflectionConverter addErrorInContext
      Failed to load
      java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"
      at java.lang.Class.getDeclaringClass(Class.java:781)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:46)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:89)
      at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:98)
      at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:134)
      at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:43)
      at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:79)
      at java.lang.reflect.Field.getGenericType(Field.java:236)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:207)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:165)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:122)
      at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
      at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:83)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:74)
      at hudson.slaves.NodeList$ConverterImpl.unmarshal(NodeList.java:162)
      at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:338)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:328)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:274)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:221)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061)
      at hudson.util.XStream2.unmarshal(XStream2.java:109)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045)
      at hudson.XmlFile.unmarshal(XmlFile.java:163)
      at jenkins.model.Jenkins$17.run(Jenkins.java:2545)
      at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:906)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:929)
      at java.lang.Thread.run(Thread.java:761)

        Attachments

        1. Queue.diff
          116 kB
        2. Queue-compile.javap
          208 kB
        3. Queue-package.javap
          211 kB

          Issue Links

            Activity

            gabriel Gabriel Lopez created issue -
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Most probably, the referenced log is unrelated.
            Could you provide the slave's configuration?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Most probably, the referenced log is unrelated. Could you provide the slave's configuration?
            Hide
            bluejaguar John Tal added a comment - - edited

            Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade.

            Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents.

            With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.

            Show
            bluejaguar John Tal added a comment - - edited Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade. Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents. With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.
            Hide
            highth Pery Wing added a comment -

            I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost.
            Some time will encounter exception as this ticket show when create the slave node.

            I finally install the 1.528 seems works well for me.

            I also think that a defect in newer vesion

            My isntall evn: RHEL+tomcat8

            Show
            highth Pery Wing added a comment - I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost. Some time will encounter exception as this ticket show when create the slave node. I finally install the 1.528 seems works well for me. I also think that a defect in newer vesion My isntall evn: RHEL+tomcat8
            Hide
            danielbeck Daniel Beck added a comment -

            Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?

            Show
            danielbeck Daniel Beck added a comment - Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?
            danielbeck Daniel Beck made changes -
            Field Original Value New Value
            Link This issue is duplicated by JENKINS-22175 [ JENKINS-22175 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-21867 [ JENKINS-21867 ]
            Hide
            tang Pei-Tang Huang added a comment -

            We first encounter this situation from about Oct/13

            The jvm info of my installation:

            JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA

            Show
            tang Pei-Tang Huang added a comment - We first encounter this situation from about Oct/13 The jvm info of my installation: JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA
            Hide
            tang Pei-Tang Huang added a comment - - edited

            I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!

            Show
            tang Pei-Tang Huang added a comment - - edited I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!
            Hide
            gabriel Gabriel Lopez added a comment -

            Hi. Sorry for the delay.
            Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration.

            John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is.

            Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said.

            While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script (https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks.

            My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.

            Show
            gabriel Gabriel Lopez added a comment - Hi. Sorry for the delay. Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration. John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is. Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said. While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script ( https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script ), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks. My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            There're other issues with classloading on IBM Java. This issue is likely caused by IBM's j9 like it has been mentioned by Pei-Tang and Gabriel.

            I'll create a master-issue, which will reference all others

            Show
            oleg_nenashev Oleg Nenashev added a comment - There're other issues with classloading on IBM Java. This issue is likely caused by IBM's j9 like it has been mentioned by Pei-Tang and Gabriel. I'll create a master-issue, which will reference all others
            oleg_nenashev Oleg Nenashev made changes -
            Summary A Slave disappears every time Jenkins boots up A Slave disappears every time Jenkins boots up on IBM Java
            oleg_nenashev Oleg Nenashev made changes -
            Labels java-ibm
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue depends on JENKINS-23176 [ JENKINS-23176 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-22006 [ JENKINS-22006 ]
            Hide
            martinkutter Martin Kutter added a comment -

            We're running Jenkins on IBM JDK on AIX.

            I haven't met this issue with Jenkins-1.532.2-LTS - however, I've seen this issue with 1.565. Maybe this helps narrowing down the issue.

            System details:

            AIX 7.1
            IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc-32 20140106_181350 (JIT enabled, AOT enabled)
            Tomcat 1.0.50

            Show
            martinkutter Martin Kutter added a comment - We're running Jenkins on IBM JDK on AIX. I haven't met this issue with Jenkins-1.532.2-LTS - however, I've seen this issue with 1.565. Maybe this helps narrowing down the issue. System details: AIX 7.1 IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc-32 20140106_181350 (JIT enabled, AOT enabled) Tomcat 1.0.50
            Hide
            martinkutter Martin Kutter added a comment - - edited

            In my case, reloading the configuration from disk also helps.

            Before I reload the configuration from disk, Jenkins complains that there is Old data on disk.
            It displays the following:

            Type: hudson.model.Hudson
            Name:
            Error: IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue", ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference 
            ---- Debugging information ---- 
            reference : ../../../../../../../slave/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator 
            ------------------------------- 
            message : Could not call hudson.EnvVars.readObject() 
            cause-exception : com.thoughtworks.xstream.converters.ConversionException 
            cause-message : Invalid reference 
            class : hudson.EnvVars 
            required-type : hudson.EnvVars 
            converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter 
            path : /hudson/slaves/slave[2]/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator 
            line number : 243 
            -------------------------------, 
            InstantiationError: null, MissingFieldException: No field 'hudson.tools.ToolLocationNodeProperty' found in class 'hudson.model.Hudson'
            
            Show
            martinkutter Martin Kutter added a comment - - edited In my case, reloading the configuration from disk also helps. Before I reload the configuration from disk, Jenkins complains that there is Old data on disk. It displays the following: Type: hudson.model.Hudson Name: Error: IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue", ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference ---- Debugging information ---- reference : ../../../../../../../slave/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator ------------------------------- message : Could not call hudson.EnvVars.readObject() cause-exception : com.thoughtworks.xstream.converters.ConversionException cause-message : Invalid reference class : hudson.EnvVars required-type : hudson.EnvVars converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter path : /hudson/slaves/slave[2]/nodeProperties/hudson.slaves.EnvironmentVariablesNodeProperty/envVars/tree-map/default/comparator line number : 243 -------------------------------, InstantiationError: null, MissingFieldException: No field 'hudson.tools.ToolLocationNodeProperty' found in class 'hudson.model.Hudson'
            Hide
            danielbeck Daniel Beck added a comment -

            IBM JDK on AIX is probably one of the worst supported environments for Jenkins. Consider moving the master node to e.g. Linux with Oracle JDK 7, which should be fairly reliable.

            Show
            danielbeck Daniel Beck added a comment - IBM JDK on AIX is probably one of the worst supported environments for Jenkins. Consider moving the master node to e.g. Linux with Oracle JDK 7, which should be fairly reliable.
            ediren edwin renner made changes -
            Link This issue is duplicated by JENKINS-22847 [ JENKINS-22847 ]
            Hide
            martinkutter Martin Kutter added a comment - - edited

            The ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference above is probably unrelated: It looks like my config had invalid references. After cleaning them up, this message disappeared.

            However, the IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" could be the reason for this issue.

            I reproduced the issue with Jenkins 1.554.3 (LTS), 1.565 and 1.570 with the following combinations.

            • Jenkins 1.565 on IBM JDK 6 SR 16 on AIX 7.1
            • Jenkins 1.565 on IBM JDK 7 SR 6 on AIX 7.1
            • Jenkins 1.565 on IBM JDK 7 SR 7 (latest) on AIX 7.1
            • Jenkins 1.554.3 on IBM JDK 7 (build 2.6) on Windows 7
            • Jenkins 1.565 on IBM JDK 7 (build 2.6) on Windows 7
            • Jenkins 1.570 on IBM JDK 7 (build 2.6) on Windows 7
            • Jenkins 1.570 (re-built from source on IBM JDK 6) on IBM JDK 6 on Windows 7

            The issue does not occur on Jenkins-1.532.2 (LTS).

            In my case, the "missing" slave is always the first slave in the config.xml, so I would assume that loading a Slave from disk (and creating a hudsom.model.Queue$ItemList) does not work the first time,
            but works on subsequent tries.

            I've also found that the same error is thrown when creating the first slave node on a fresh jenkins instance (on save / doCreateItem):

             
            Stack trace (excerpt):
            
            Caused by: java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"
            	at java.lang.Class.getDeclaringClass(Class.java:821)
            	at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:52)
            	at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:95)
            	at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
            	at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
            	at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
            	at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:85)
            	at java.lang.reflect.Field.getGenericType(Field.java:237)
            	at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:224)
            	at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182)
            	at com.thoughtworks.xstream.mapper.AnnotationMapper.serializedClass(AnnotationMapper.java:129)
            	at hudson.util.xstream.MapperDelegate.serializedClass(MapperDelegate.java:39)
            	at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:26)
            	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:62)
            	at hudson.slaves.NodeList$ConverterImpl.marshal(NodeList.java:151)
            
            

            The slave is saved nonetheless.

            Subsequent slaves can be created without this error.

            Show
            martinkutter Martin Kutter added a comment - - edited The ConversionException: Could not call hudson.EnvVars.readObject() : Invalid reference above is probably unrelated: It looks like my config had invalid references. After cleaning them up, this message disappeared. However, the IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" could be the reason for this issue. I reproduced the issue with Jenkins 1.554.3 (LTS), 1.565 and 1.570 with the following combinations. Jenkins 1.565 on IBM JDK 6 SR 16 on AIX 7.1 Jenkins 1.565 on IBM JDK 7 SR 6 on AIX 7.1 Jenkins 1.565 on IBM JDK 7 SR 7 (latest) on AIX 7.1 Jenkins 1.554.3 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.565 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.570 on IBM JDK 7 (build 2.6) on Windows 7 Jenkins 1.570 (re-built from source on IBM JDK 6) on IBM JDK 6 on Windows 7 The issue does not occur on Jenkins-1.532.2 (LTS). In my case, the "missing" slave is always the first slave in the config.xml, so I would assume that loading a Slave from disk (and creating a hudsom.model.Queue$ItemList) does not work the first time, but works on subsequent tries. I've also found that the same error is thrown when creating the first slave node on a fresh jenkins instance (on save / doCreateItem): Stack trace (excerpt): Caused by: java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue" at java.lang.Class.getDeclaringClass(Class.java:821) at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:52) at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:95) at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105) at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140) at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:85) at java.lang.reflect.Field.getGenericType(Field.java:237) at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:224) at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182) at com.thoughtworks.xstream.mapper.AnnotationMapper.serializedClass(AnnotationMapper.java:129) at hudson.util.xstream.MapperDelegate.serializedClass(MapperDelegate.java:39) at com.thoughtworks.xstream.mapper.MapperWrapper.serializedClass(MapperWrapper.java:26) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:62) at hudson.slaves.NodeList$ConverterImpl.marshal(NodeList.java:151) The slave is saved nonetheless. Subsequent slaves can be created without this error.
            Hide
            martinkutter Martin Kutter added a comment -

            The error does not appear in Jenkins <= 1.535

            Looks like it has been introduced in 1.536.

            Show
            martinkutter Martin Kutter added a comment - The error does not appear in Jenkins <= 1.535 Looks like it has been introduced in 1.536.
            Hide
            martinkutter Martin Kutter added a comment -

            I opened a support request at IBM. I'll update the issue when appropriate.

            Show
            martinkutter Martin Kutter added a comment - I opened a support request at IBM. I'll update the issue when appropriate.
            Hide
            jglick Jesse Glick added a comment -

            The loss of slave definitions is just the most obvious symptom of the problem with Queue$ItemList, whatever that is.

            Show
            jglick Jesse Glick added a comment - The loss of slave definitions is just the most obvious symptom of the problem with Queue$ItemList , whatever that is.
            jglick Jesse Glick made changes -
            Summary A Slave disappears every time Jenkins boots up on IBM Java Jenkins global configuration unreadable on IBM J9 (so slaves lost)
            Hide
            jglick Jesse Glick added a comment -

            I suspect the problem is related to @WithBridgeMethods or some other bytecode manipulation tricks done in Jenkins; the Eclipse Java compiler also is broken by Queue. It is TBD whether our reworked bytecode is actually invalid according to the JVM spec, or whether it is simply unusual and confuses tools which were not tested against this situation.

            Show
            jglick Jesse Glick added a comment - I suspect the problem is related to @WithBridgeMethods or some other bytecode manipulation tricks done in Jenkins; the Eclipse Java compiler also is broken by Queue . It is TBD whether our reworked bytecode is actually invalid according to the JVM spec, or whether it is simply unusual and confuses tools which were not tested against this situation.
            Hide
            jglick Jesse Glick added a comment -

            Easily reproduced from tests:

            JAVA_HOME=/…/ibm-jdk-71 mvn -f test -Dtest=QueueTest\#testFlyweightTasksWithoutMasterExecutors surefire:test
            

            (Though in this case it does not cause the test to fail.)

            Show
            jglick Jesse Glick added a comment - Easily reproduced from tests: JAVA_HOME=/…/ibm-jdk-71 mvn -f test -Dtest=QueueTest\#testFlyweightTasksWithoutMasterExecutors surefire:test (Though in this case it does not cause the test to fail.)
            Hide
            jglick Jesse Glick added a comment -

            Also reproducible from

            public class Main {
                public static void main(String[] args) throws Exception {
                    System.out.println(Class.forName("hudson.model.Queue$ItemList").getEnclosingClass());
                }
            }
            

            run against core/target/classes, if you have done mvn -f core -DskipTests clean package, but not if the last goal is merely compile. javap confirms that the bytecode of Queue.class is substantially different after the packaging step, presumably due to the bridge method injector running on scheduleMaintenance, though the InnerClasses attribute is not different. In both cases there is no mention of ItemList in InnerClasses, which would seem to violate the JVM spec since it is clearly used, for example as the type of the buildables field. There is a difference in how fields are named, for example

            -         1: getfield      #2                  // Field buildables:Lhudson/model/Queue$ItemList;
            +         1: getfield      #2                  // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;
            
            Show
            jglick Jesse Glick added a comment - Also reproducible from public class Main { public static void main( String [] args) throws Exception { System .out.println( Class .forName( "hudson.model.Queue$ItemList" ).getEnclosingClass()); } } run against core/target/classes , if you have done mvn -f core -DskipTests clean package , but not if the last goal is merely compile . javap confirms that the bytecode of Queue.class is substantially different after the packaging step, presumably due to the bridge method injector running on scheduleMaintenance , though the InnerClasses attribute is not different. In both cases there is no mention of ItemList in InnerClasses , which would seem to violate the JVM spec since it is clearly used, for example as the type of the buildables field. There is a difference in how fields are named, for example - 1: getfield #2 // Field buildables:Lhudson/model/Queue$ItemList; + 1: getfield #2 // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;
            jglick Jesse Glick made changes -
            Attachment Queue.diff [ 26461 ]
            Hide
            jglick Jesse Glick added a comment -

            Attached diff before; now attaching original files for completeness.

            Show
            jglick Jesse Glick added a comment - Attached diff before; now attaching original files for completeness.
            jglick Jesse Glick made changes -
            Attachment Queue-compile.javap [ 26463 ]
            Attachment Queue-package.javap [ 26464 ]
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            After looking further into javap source code, I discovered why the following difference in the output happens:

            -         1: getfield      #2                  // Field buildables:Lhudson/model/Queue$ItemList;
            +         1: getfield      #2                  // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList;
            

            When printing the content of constant pool for these instructions, javap attempts to simplify the reference within the same class by omitting the class name as follows:

                    int tag = cpInfo.getTag();
                    switch (tag) {
                        case CONSTANT_Methodref:
                        case CONSTANT_InterfaceMethodref:
                        case CONSTANT_Fieldref:
                            // simplify references within this class
                            CPRefInfo ref = (CPRefInfo) cpInfo;
                            try {
                                if (ref.class_index == classFile.this_class)
                                     cpInfo = classFile.constant_pool.get(ref.name_and_type_index);
                            } catch (ConstantPool.InvalidIndex e) {
                                // ignore, for now
                            }
                    }
                    print(tagName(tag) + " " + stringValue(cpInfo));
            

            This check compares the constant pool index, not the actual content. The fieldref constant pool remains unchanged at #283.#624 before/after transformation, but classFile.this_class that references the class name itself changes from #283 to #400.

            The constant pool entry #400 is "Class #891", just like #283. IOW, javac produces Queue.class that contains two identical constant pool entries, which is suprising.

            It is still unclear as to whether this difference in the class file triggers the "incompatible InnerClasses attribute" problem.

            I think the next step is to modify class file to insert InnerClass attribute for Queue$ItemList, as the lack of this entry does seem to violate JVM spec. If that is the case, the question remains as to why J9VM was happy with the class before transformation.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - After looking further into javap source code, I discovered why the following difference in the output happens: - 1: getfield #2 // Field buildables:Lhudson/model/Queue$ItemList; + 1: getfield #2 // Field hudson/model/Queue.buildables:Lhudson/model/Queue$ItemList; When printing the content of constant pool for these instructions, javap attempts to simplify the reference within the same class by omitting the class name as follows: int tag = cpInfo.getTag(); switch (tag) { case CONSTANT_Methodref: case CONSTANT_InterfaceMethodref: case CONSTANT_Fieldref: // simplify references within this class CPRefInfo ref = (CPRefInfo) cpInfo; try { if (ref.class_index == classFile.this_class) cpInfo = classFile.constant_pool.get(ref.name_and_type_index); } catch (ConstantPool.InvalidIndex e) { // ignore, for now } } print(tagName(tag) + " " + stringValue(cpInfo)); This check compares the constant pool index, not the actual content. The fieldref constant pool remains unchanged at #283.#624 before/after transformation, but classFile.this_class that references the class name itself changes from #283 to #400. The constant pool entry #400 is "Class #891", just like #283. IOW, javac produces Queue.class that contains two identical constant pool entries, which is suprising. It is still unclear as to whether this difference in the class file triggers the "incompatible InnerClasses attribute" problem. I think the next step is to modify class file to insert InnerClass attribute for Queue$ItemList , as the lack of this entry does seem to violate JVM spec. If that is the case, the question remains as to why J9VM was happy with the class before transformation.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Just for the sake of completeness I manually patched Queue.class after transformation to have its classFile.this_class index points to #283, not #400, and now J9 VM is happy.

            Wow.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Just for the sake of completeness I manually patched Queue.class after transformation to have its classFile.this_class index points to #283, not #400, and now J9 VM is happy. Wow.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Relevant StackOverflow thread and BugParade entry, which indicates the fix to javac made it into JDK8 b92.

            Looks like J9 VM is failing to handle this rare situation correctly. I think I'll work around this problem in bridge method injector to reshuffle constant pools and eliminate duplicates.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Relevant StackOverflow thread and BugParade entry , which indicates the fix to javac made it into JDK8 b92. Looks like J9 VM is failing to handle this rare situation correctly. I think I'll work around this problem in bridge method injector to reshuffle constant pools and eliminate duplicates.
            Hide
            jglick Jesse Glick added a comment -
            Show
            jglick Jesse Glick added a comment - More useful bug link: https://bugs.openjdk.java.net/browse/JDK-5053846
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/pom.xml
            pom.xml
            http://jenkins-ci.org/commit/jenkins/4e23793f62aa5019226f2ba96f8063f01affeb92
            Log:
            [FIXED JENKINS-22525]

            Integrated a bridge-method-injector that rebuilds the constant pool from
            scratch.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml pom.xml http://jenkins-ci.org/commit/jenkins/4e23793f62aa5019226f2ba96f8063f01affeb92 Log: [FIXED JENKINS-22525] Integrated a bridge-method-injector that rebuilds the constant pool from scratch.
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Show
            kohsuke Kohsuke Kawaguchi added a comment - Code change in bridge-method-injector
            Hide
            jglick Jesse Glick added a comment -
            Show
            jglick Jesse Glick added a comment - Also it seems like https://bugs.openjdk.java.net/browse/JDK-6746955 and https://bugs.openjdk.java.net/browse/JDK-8015927 might be the more relevant issues.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Changed the title to accurately capture the issue.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Changed the title to accurately capture the issue.
            kohsuke Kohsuke Kawaguchi made changes -
            Assignee Kohsuke Kawaguchi [ kohsuke ]
            Summary Jenkins global configuration unreadable on IBM J9 (so slaves lost) "incompatible InnerClasses attribute" error in IBM J9 VM
            jglick Jesse Glick made changes -
            Labels java-ibm java-ibm lts-candidate
            Hide
            kohsuke Kohsuke Kawaguchi added a comment - - edited

            As for ItemList not in the InnerClasses attribute, I'm discovering with the following test case that if a nested/inner class is marked as private, it does not get recorded in the InnerClasses attribute:

            public class App 
            {
                private static class Foo {}
                private class Bar {}
            
                static class Alpha {}
                class Bravo {}
            
                public static void main( String[] args )
                {
                    System.out.println(Foo.class.getEnclosingClass());
                    System.out.println(Bar.class.getEnclosingClass());
            
                    System.out.println(Alpha.class.getEnclosingClass());
                    System.out.println(Bravo.class.getEnclosingClass());
                }
            }
            
            Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App.class
              Last modified Jul 31, 2014; size 1158 bytes
              MD5 checksum 264ed3611a578e1a4c8e2d4e0ea8b378
              Compiled from "App.java"
            public class test.App
              SourceFile: "App.java"
              InnerClasses: 
                   #11= #8 of #9; //Bravo=class test/App$Bravo of class test/App
                   static #13= #7 of #9; //Alpha=class test/App$Alpha of class test/App
              RuntimeInvisibleAnnotations:
                0: #57()
              minor version: 0
              major version: 49
              flags: ACC_PUBLIC, ACC_SUPER
            
            Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Foo.class
              Last modified Jul 31, 2014; size 303 bytes
              MD5 checksum 3f4b082e8d8c5cbf85d4afeb7807b253
              Compiled from "App.java"
            class test.App$Foo
              SourceFile: "App.java"
              minor version: 0
              major version: 49
              flags: ACC_SUPER
            
            Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Alpha.class
              Last modified Jul 31, 2014; size 309 bytes
              MD5 checksum df99610f5b28798c5f07f308b749df89
              Compiled from "App.java"
            class test.App$Alpha
              SourceFile: "App.java"
              InnerClasses: 
                   static #10= #2 of #16; //Alpha=class test/App$Alpha of class test/App
              minor version: 0
              major version: 49
              flags: ACC_SUPER
            

            This seems to violate JVM spec of the InnerClass attribute. J9 VM accepts this without reporting an error. In this case I don't understand how JVM can correctly determine that App$Foo has an enclosing class. From the byte code, there's no way to distinguish this vs compiling the following class:

            class App {}
            class App$Foo {} // '$' is a legal character in a class name in Java
            
            Show
            kohsuke Kohsuke Kawaguchi added a comment - - edited As for ItemList not in the InnerClasses attribute, I'm discovering with the following test case that if a nested/inner class is marked as private, it does not get recorded in the InnerClasses attribute: public class App { private static class Foo {} private class Bar {} static class Alpha {} class Bravo {} public static void main( String [] args ) { System .out.println(Foo. class. getEnclosingClass()); System .out.println(Bar. class. getEnclosingClass()); System .out.println(Alpha. class. getEnclosingClass()); System .out.println(Bravo. class. getEnclosingClass()); } } Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App.class Last modified Jul 31, 2014; size 1158 bytes MD5 checksum 264ed3611a578e1a4c8e2d4e0ea8b378 Compiled from "App.java" public class test.App SourceFile: "App.java" InnerClasses: #11= #8 of #9; //Bravo=class test/App$Bravo of class test/App static #13= #7 of #9; //Alpha=class test/App$Alpha of class test/App RuntimeInvisibleAnnotations: 0: #57() minor version: 0 major version: 49 flags: ACC_PUBLIC, ACC_SUPER Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Foo.class Last modified Jul 31, 2014; size 303 bytes MD5 checksum 3f4b082e8d8c5cbf85d4afeb7807b253 Compiled from "App.java" class test.App$Foo SourceFile: "App.java" minor version: 0 major version: 49 flags: ACC_SUPER Classfile /files/kohsuke/ws/jenkins/investigations/JENKINS-22525/jenkins22525/target/classes/test/App$Alpha.class Last modified Jul 31, 2014; size 309 bytes MD5 checksum df99610f5b28798c5f07f308b749df89 Compiled from "App.java" class test.App$Alpha SourceFile: "App.java" InnerClasses: static #10= #2 of #16; //Alpha=class test/App$Alpha of class test/App minor version: 0 major version: 49 flags: ACC_SUPER This seems to violate JVM spec of the InnerClass attribute . J9 VM accepts this without reporting an error. In this case I don't understand how JVM can correctly determine that App$Foo has an enclosing class. From the byte code, there's no way to distinguish this vs compiling the following class: class App {} class App$Foo {} // '$' is a legal character in a class name in Java
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3569
            [FIXED JENKINS-22525] (Revision 4e23793f62aa5019226f2ba96f8063f01affeb92)

            Result = SUCCESS
            kohsuke : 4e23793f62aa5019226f2ba96f8063f01affeb92
            Files :

            • pom.xml
            • core/pom.xml
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3569 [FIXED JENKINS-22525] (Revision 4e23793f62aa5019226f2ba96f8063f01affeb92) Result = SUCCESS kohsuke : 4e23793f62aa5019226f2ba96f8063f01affeb92 Files : pom.xml core/pom.xml changelog.html
            Hide
            martinkutter Martin Kutter added a comment -

            The fix seems to work around the issue - I've been unable to reproduce the error with a build from sources with the fix applied.

            Runtime: Windows 7, IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)

            Show
            martinkutter Martin Kutter added a comment - The fix seems to work around the issue - I've been unable to reproduce the error with a build from sources with the fix applied. Runtime: Windows 7, IBM J9 VM (build 2.6, JRE 1.7.0 Windows 7 amd64-64 Compressed References 20130617_152572 (JIT enabled, AOT enabled)
            Hide
            dantran dan tran added a comment -

            dont see it at 1.575 http://jenkins-ci.org/changelog.

            Show
            dantran dan tran added a comment - dont see it at 1.575 http://jenkins-ci.org/changelog .
            Hide
            jglick Jesse Glick added a comment -

            @dantran because it was committed toward 1.576.

            Show
            jglick Jesse Glick added a comment - @dantran because it was committed toward 1.576.
            Hide
            danielbeck Daniel Beck added a comment -

            Literally one click away: "Upcoming changes"

            Show
            danielbeck Daniel Beck added a comment - Literally one click away: "Upcoming changes"
            Hide
            dantran dan tran added a comment -

            thanks, I am now running 1.576-SNAPSHOT. Good to see this issue is fixed

            Show
            dantran dan tran added a comment - thanks, I am now running 1.576-SNAPSHOT. Good to see this issue is fixed
            Hide
            martinkutter Martin Kutter added a comment -

            IBM has tracked the issue with "incompatible inner Class attribute" as APAR IV62298 - see http://www-01.ibm.com/support/docview.wss?uid=swg1IV62298
            I've received an iFix for Java 7SR7 from IBM, which fixes the issue.

            The fix will be applied in the upcoming IBM JDK 7 SR 8.

            Show
            martinkutter Martin Kutter added a comment - IBM has tracked the issue with "incompatible inner Class attribute" as APAR IV62298 - see http://www-01.ibm.com/support/docview.wss?uid=swg1IV62298 I've received an iFix for Java 7SR7 from IBM, which fixes the issue. The fix will be applied in the upcoming IBM JDK 7 SR 8.
            olivergondza Oliver Gondža made changes -
            Labels java-ibm lts-candidate 1.565.3-fixed java-ibm
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/pom.xml
            pom.xml
            http://jenkins-ci.org/commit/jenkins/e4dc3b4210fb6afd80ddf138cafc6bfcb866df5f
            Log:
            [FIXED JENKINS-22525]

            Integrated a bridge-method-injector that rebuilds the constant pool from
            scratch.

            (cherry picked from commit 4e23793f62aa5019226f2ba96f8063f01affeb92)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml pom.xml http://jenkins-ci.org/commit/jenkins/e4dc3b4210fb6afd80ddf138cafc6bfcb866df5f Log: [FIXED JENKINS-22525] Integrated a bridge-method-injector that rebuilds the constant pool from scratch. (cherry picked from commit 4e23793f62aa5019226f2ba96f8063f01affeb92) Conflicts: changelog.html
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 154627 ] JNJira + In-Review [ 194969 ]

              People

              Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              gabriel Gabriel Lopez
              Votes:
              6 Vote for this issue
              Watchers:
              16 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: