Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22525

"incompatible InnerClasses attribute" error in IBM J9 VM

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • windows 2008 R2, Tomcat 6.0.32, IBM Java 1.6 SR15

      When starting Jenkins, it always happen that a Slave disappears. The node is simply vanished.
      I have tried looking for specific errors in the log without success, even looking at all log levels. There is an error while unmarshalling an xml, but unsure if it is related. See the stack trace below in any case.

      I have to note that only one disappears, not all.

      The curious thing comes that if I click "Reload configuration from disk", the disappeared slave is loaded correctly this time (the configuration was in the xml there all along, and I'm not overwriting it).

      Information about my instance:
      Version: 1.547 (happened with 1.532 as well)
      java.vm.info JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr15-20131016_170922 (JIT enabled, AOT enabled)>J9VM - 20131016_170922>JIT - r9_20130920_46510ifx2>GC - GA24_Java6_SR15_20131016_1337_B170922 (happened with an older version as well, tried updating. I'm aware that IBM Java is not officially supported).
      Using tomcat 6.0.32, on a windows 2008 R2 Standard.

      Possible related bug: https://issues.jenkins-ci.org/browse/JENKINS-21867
      (Also on Windows 2008)

      Error that might be related:
      FINE hudson.util.RobustReflectionConverter addErrorInContext
      Failed to load
      java.lang.IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"
      at java.lang.Class.getDeclaringClass(Class.java:781)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:46)
      at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:89)
      at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:98)
      at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:134)
      at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:43)
      at sun.reflect.generics.repository.FieldRepository.getGenericType(FieldRepository.java:79)
      at java.lang.reflect.Field.getGenericType(Field.java:236)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:207)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:165)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:122)
      at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
      at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:83)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:74)
      at hudson.slaves.NodeList$ConverterImpl.unmarshal(NodeList.java:162)
      at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:338)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:328)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:274)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:221)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061)
      at hudson.util.XStream2.unmarshal(XStream2.java:109)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045)
      at hudson.XmlFile.unmarshal(XmlFile.java:163)
      at jenkins.model.Jenkins$17.run(Jenkins.java:2545)
      at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:906)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:929)
      at java.lang.Thread.run(Thread.java:761)

        1. Queue-compile.javap
          208 kB
        2. Queue-package.javap
          211 kB
        3. Queue.diff
          116 kB

          [JENKINS-22525] "incompatible InnerClasses attribute" error in IBM J9 VM

          Gabriel Lopez created issue -

          Oleg Nenashev added a comment -

          Most probably, the referenced log is unrelated.
          Could you provide the slave's configuration?

          Oleg Nenashev added a comment - Most probably, the referenced log is unrelated. Could you provide the slave's configuration?

          John Tal added a comment - - edited

          Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade.

          Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents.

          With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.

          John Tal added a comment - - edited Was this an in-place update/upgrade that done recently from a previous version of Jenkins (or even hudson)? At least on RHEL with tomcat we have had to remove the old directory under webapps when dropping a new jenkins.war in place. Otherwise we can get these same symptoms (disappearing nodes) and see this same error with a slightly different stack trace immediately after a fresh in-place upgrade. Also note there is a ~/.jenkins/cache directory where jar file versions may be. Don't suggest you play in there unless step above has been tried and then, if still failing, after you have stopped jenkins to rename/delete cache contents. With further testing we can get this same error to occur with different versions of Jenkins. For an existing installation we are seeing this same issue when we update to Jenkins 1.562 ("incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue"). Whereas dropping in-place a lower Jenkins version such as 1.528 does not show this issue.

          Pery Wing added a comment -

          I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost.
          Some time will encounter exception as this ticket show when create the slave node.

          I finally install the 1.528 seems works well for me.

          I also think that a defect in newer vesion

          My isntall evn: RHEL+tomcat8

          Pery Wing added a comment - I also meet same problems. install both 1.558 version and 1.554.1(LTS). create a slave node then restart web server the slave node lost. Some time will encounter exception as this ticket show when create the slave node. I finally install the 1.528 seems works well for me. I also think that a defect in newer vesion My isntall evn: RHEL+tomcat8

          Daniel Beck added a comment -

          Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?

          Daniel Beck added a comment - Is there an entry in Manage Jenkins » Manage Old Data for the logged exception?
          Daniel Beck made changes -
          Link New: This issue is duplicated by JENKINS-22175 [ JENKINS-22175 ]
          Daniel Beck made changes -
          Link New: This issue is related to JENKINS-21867 [ JENKINS-21867 ]

          We first encounter this situation from about Oct/13

          The jvm info of my installation:

          JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA

          Pei-Tang Huang added a comment - We first encounter this situation from about Oct/13 The jvm info of my installation: JRE 1.6.0 IBM J9 2.4 Windows Server 2008 R2 amd64-64 jvmwa6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)>J9VM - 20120202_101568>JIT - r9_20111107_21307ifx1>GC - 20120202_AA

          Pei-Tang Huang added a comment - - edited

          I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!

          Pei-Tang Huang added a comment - - edited I changed the JRE from IBM's j9 to Oracle's one (jdk1.7.0_55 x64), restart, restart, restart, none of my slaves have been taken!

          Gabriel Lopez added a comment -

          Hi. Sorry for the delay.
          Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration.

          John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is.

          Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said.

          While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script (https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks.

          My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.

          Gabriel Lopez added a comment - Hi. Sorry for the delay. Oleg- The slave is nothing out of the ordinary, just a simple, Java Web Start slave. It is not happening with the "same" slave always, just one that gets disappeared. If I have 4, and jenkins is restarted, 3 appears. If I restart again, I'm left with 2. My feeling is that it depends on the order in the xml configuration. John - No, it was a new installation. I recently updated to 1.554.1 LTS, and removed the webapps folder. The issue is still there. I'll try removing the cache directory, though. It is a Windows machine, so I'll see where that folder is. Daniel- Yes, there is. It says: hudson.model.Hudson IncompatibleClassChangeError: incompatible InnerClasses attribute between "hudson.model.Queue$ItemList" and "hudson.model.Queue.". But I'm not too convinced that it is related, as Oleg said. While troubleshooting the issue, I found out that the missing slave configuration was in the jenkins' config XML, and discovered that hitting "Reload configuration from disk" makes the slave to appear again. As a workaround, I created a Groovy post-init script ( https://wiki.jenkins-ci.org/display/JENKINS/Post-initialization+script ), that calls the Reloading function.. So after restarting and just after finishing the initialization process, my jenkins reloads the configuration again, making the slave to appear again. It is a dirty workaround, but has been working for several weeks. My suspicion is that it is a problem when running Jenkins on IBM Java, maybe even specific with IBM java versions.

            kohsuke Kohsuke Kawaguchi
            gabriel Gabriel Lopez
            Votes:
            6 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: