Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51965

Add support of JAXB unpackaging for Java 9+ in WAR Exploder

      As discussed in JENKINS-51821, Jenkins won't work on Java 11 due to removed JAXB modules. I propose the following approach:

      • Package 3 JAXB libs (jaxb-api-2.3.0.jar, jaxb-core-2.3.0.1.jar, jaxb-impl-2.3.0.1.jar) to a WAR resource which does not get classloaded by default. E.g. "WEB-INF/platform-compat/java11" (or 9 ?)
      • When Jenkins starts up on Java 9+, Jenkins WAR exploder copies the libraries to "war/WEB-INF/lib" so that they get picked by classloader automatically

      It will allow to have a transparent support of Java 9..11 within the WAR file (without adding modules, etc.)

      A better way forward would be Multi-Version JAR, but we need a massive tooling update to support it

          [JENKINS-51965] Add support of JAXB unpackaging for Java 9+ in WAR Exploder

          Oleg Nenashev added a comment - - edited

          kohsuke We also have other options which may be preferable && compatible

          • Cleanup usages of JAXB from Jenkins Core and Modules (like you proposed)
          • Create a new JAXB API Plugin
            • The plugin is compatible with Java 8
            • The plugin implements multi-release JAR (will require tooling updates IIUC)
            • For Java 8 the plugin does nothing
            • For Java 9+ we bundle the libraries into the plugin
          • Release the plugin and mark it as detached from the core

          In such case we will have a clean WAR startup without JAXB modules + the plugin classloaders will be used to add the dependencies from JAXB API Plugin. It will allow running the approach in all Web containers.

          The main problem is that JAXB API Plugin will be present on many instances until we update all main plugins beyond the version with detached version. But it applies to all detached plugins so it should be fine

          WDYT?

          Oleg Nenashev added a comment - - edited kohsuke We also have other options which may be preferable && compatible Cleanup usages of JAXB from Jenkins Core and Modules (like you proposed) Create a new JAXB API Plugin The plugin is compatible with Java 8 The plugin implements multi-release JAR (will require tooling updates IIUC) For Java 8 the plugin does nothing For Java 9+ we bundle the libraries into the plugin Release the plugin and mark it as detached from the core In such case we will have a clean WAR startup without JAXB modules + the plugin classloaders will be used to add the dependencies from JAXB API Plugin. It will allow running the approach in all Web containers. The main problem is that JAXB API Plugin will be present on many instances until we update all main plugins beyond the version with detached version. But it applies to all detached plugins so it should be fine WDYT?

          Today we made a good progress on the proposed approach of detached JAXB plugin, but in the very end I discovered that this didn't work.

          The way plugins use JAXB makes JAXB to look up the implementation through thread context classloader, which is servlet classloader and not uber classloader. So it fails to find it. Plugins can invoke JAXB API a little differently, but it'd be nice if plugins need not make any changes.

          So after all that, I come full circle back to my original suggestion of just bundling JAXB in WEB-INF/lib. I discussed this with Paul, and this is a very common practice for JAXB, because JavaSE carries one version of it, EE container carries another version of it, and because those two platforms move very slowly, webapp often wants to use its own version of it. The servlet container is used to handle this kind of situation, and I'm saying this as a former JAXB RI & spec lead, so I know a thing or two about JAXB!

          We just need to bundle the version of JAXB that works with Java 8 (which is 2.2.x) in order to work in any Java 8+ runtime.

          I'll check in with Oleg tomorrow.

           

           

          Kohsuke Kawaguchi added a comment - Today we made a good progress on the proposed approach of detached JAXB plugin, but in the very end I discovered that this didn't work. The way plugins use JAXB makes JAXB to look up the implementation through thread context classloader, which is servlet classloader and not uber classloader. So it fails to find it. Plugins can invoke JAXB API a little differently, but it'd be nice if plugins need not make any changes. So after all that, I come full circle back to my original suggestion of just bundling JAXB in WEB-INF/lib. I discussed this with Paul, and this is a very common practice for JAXB, because JavaSE carries one version of it, EE container carries another version of it, and because those two platforms move very slowly, webapp often wants to use its own version of it. The servlet container is used to handle this kind of situation, and I'm saying this as a former JAXB RI & spec lead, so I know a thing or two about JAXB! We just need to bundle the version of JAXB that works with Java 8 (which is 2.2.x) in order to work in any Java 8+ runtime. I'll check in with Oleg tomorrow.    

          Oleg Nenashev added a comment -

          kohsuke works for me if it works without breaking on Java 8, 10 and 11
          I doubt we want to tweak plugin classloading even more to have this plugin working somehow.

          Oleg Nenashev added a comment - kohsuke works for me if it works without breaking on Java 8, 10 and 11 I doubt we want to tweak plugin classloading even more to have this plugin working somehow.

          Jesse Glick added a comment -

          thread context classloader, which is servlet classloader and not uber classloader

          I wonder if we should revisit that decision.

          Plugins can invoke JAXB API a little differently, but it'd be nice if plugins need not make any changes.

          Which plugins would actually be affected? It is maybe a five-line change to switch the thread context loader¹, and it would be nice to get JAXB out of core if we could.

          ¹And could be made even simpler with a utility in core:

          public final class WithJenkinsContextClassLoader implements AutoCloseable {
              private final Thread t;
              private final ClassLoader orig;
              public WithJenkinsContextClassLoader() {
                  this(Jenkins.get().getPluginManager().uberClassLoader);
              }
              public WithJenkinsContextClassLoader(Class<?> pluginClass) {
                  this(pluginClass.getClassLoader());
              }
              private WithJenkinsContextClassLoader(ClassLoader loader) {
                  t = Thread.currentThread();
                  orig = t.getContextClassLoader();
                  t.setContextClassLoader(loader);
              }
              @Override public void close() {
                  t.setContextClassLoader(orig);
              }
          }
          

          Thus:

          try (WithJenkinsContextClassLoader context = new WithJenkinsContextClassLoader(ThisClass.class)) {
              myUsageOfJAXB();
          }
          

          Jesse Glick added a comment - thread context classloader, which is servlet classloader and not uber classloader I wonder if we should revisit that decision. Plugins can invoke JAXB API a little differently, but it'd be nice if plugins need not make any changes. Which plugins would actually be affected? It is maybe a five-line change to switch the thread context loader¹, and it would be nice to get JAXB out of core if we could. ¹And could be made even simpler with a utility in core: public final class WithJenkinsContextClassLoader implements AutoCloseable { private final Thread t; private final ClassLoader orig; public WithJenkinsContextClassLoader() { this (Jenkins.get().getPluginManager().uberClassLoader); } public WithJenkinsContextClassLoader( Class <?> pluginClass) { this (pluginClass.getClassLoader()); } private WithJenkinsContextClassLoader( ClassLoader loader) { t = Thread .currentThread(); orig = t.getContextClassLoader(); t.setContextClassLoader(loader); } @Override public void close() { t.setContextClassLoader(orig); } } Thus: try (WithJenkinsContextClassLoader context = new WithJenkinsContextClassLoader(ThisClass.class)) { myUsageOfJAXB(); }

          Oleg Nenashev added a comment -

          I am still a fan of the "Multi-Release WAR" in this case (or of embedding bootstrap loading into the core). https://gist.github.com/rkx-bloodshed/badc6d0798b145123897a68ae4e3d6bd shows example of how modules could be loaded dynamically. Our main problem is that we may end up bundling all modules being detached from Java 11+ if we want to retain full compatibility going forward

          I suspect that we will need to tweak the Servlet Context Classloader anyway in this case (similar to the bootstrap code in https://github.com/jenkinsci/jenkins/pull/2754/files ?)

          > Which plugins would actually be affected?

          Many: https://github.com/search?l=Java&q=org%3Ajenkinsci+java.xml.bind&type=Code . There is a lot of test code and runtime annotations referenced there, so creating wrappers is unlikely a good option. 

           

          Oleg Nenashev added a comment - I am still a fan of the "Multi-Release WAR" in this case (or of embedding bootstrap loading into the core). https://gist.github.com/rkx-bloodshed/badc6d0798b145123897a68ae4e3d6bd  shows example of how modules could be loaded dynamically. Our main problem is that we may end up bundling all modules being detached from Java 11+ if we want to retain full compatibility going forward I suspect that we will need to tweak the Servlet Context Classloader anyway in this case (similar to the bootstrap code in  https://github.com/jenkinsci/jenkins/pull/2754/files  ?) > Which plugins would actually be affected? Many: https://github.com/search?l=Java&q=org%3Ajenkinsci+java.xml.bind&type=Code  . There is a lot of test code and runtime annotations referenced there, so creating wrappers is unlikely a good option.   

          I've worked a bit in the last few days on checking the current state of Jenkins without the --add-modules switches: I've tried looking for an easy case to reproduce this issue, but it's actually not anymore easy to make Jenkins fail for a JAXB error, which is good. That is mostly because since when the issues were filed, many were fixed in plac to not use JAXB anymore where this was possible.

          On the opensource side, it's close to impossible to check through the UI because of a few plugins have the majority javax.xml.bind imports
          https://github.com/search?q=org%3Ajenkinsci+%22javax.xml.bind%22&unscoped_q=%22javax.xml.bind%22, so I had to go through the GitHub API using curl and some shell scripting to get the full list.

          On the OSS side, the full list of repositories using "javax.xml.bind":

              286 "coverity-plugin"
              194 "inflectra-spira-integration-plugin"
              105 "zephyr-enterprise-test-management-plugin"
               24 "codebeamer-coverage-publisher-plugin"
               21 "codesonar-plugin"
               14 "java-client-api"
               13 "packageversion-plugin"
                9 "performance-signature-dynatrace-plugin"
                9 "hp-operations-orchestration-plugin"
                8 "meliora-testlab-plugin"
                8 "mabl-integration-plugin"
                8 "jelly"
                8 "app.io-plugin"
                7 "sloccount-plugin"
                7 "jira-issues-versioning-plugin"
                7 "dom4j"
                6 "extras-ec2-launcher"
                6 "dynatrace-plugin"
                5 "xlrelease-plugin"
                5 "xcode-plugin"
                5 "libdtkit"
                5 "github-coverage-reporter-plugin"
                4 "zanata-plugin"
                3 "cppcheck-plugin"
                2 "ucm4svn-plugin"
                2 "tfs-plugin"
                2 "stride-notification-plugin"
                2 "refit-plugin"
                2 "notification-plugin"
                2 "maven-metadata-plugin"
                2 "hp-quality-center-plugin"
                2 "ec2-deployment-dashboard"
                2 "docker-workflow-plugin"
                2 "docker"
                1 "windows-azure-storage-plugin"
                1 "timestamper-plugin"
                1 "testinium-plugin"
                1 "testcomplete-plugin"
                1 "testcomplete11-xunit-plugin"
                1 "qualityclouds-plugin"
                1 "policycenter-gate-validator-plugin"
                1 "pipeline-aws-plugin"
                1 "performance-plugin"
                1 "maven-hudson-dev-plugin"
                1 "maven-artifact-choicelistprovider-plugin"
                1 "localization-plugin"
                1 "klocwork-plugin"
                1 "jwsdp-sqe-plugin"
                1 "job-import-plugin"
                1 "jenkow-plugin"
                1 "jclouds-plugin"
                1 "jaxb-plugin"
                1 "ibm-cloud-devops-plugin"
                1 "fortify360-plugin"
                1 "dtkit-plugin"
                1 "agile-cockpit-notification-plugin"
                1 "acceptance-test-harness"
          

          Some notes

          Conclusion:

          Given:

          • the long term right fix is likely to actually not need --add-modules anymore and allow plugins to do like nothing changed
          • we already somehow communicated the fact we were shifting gears, and interpreting to my advantage "we are willing to break some stuff to gain what we really need" . In other words: yes we could obviously fix this, but do we really need/want to spend more time on making JAXB usable like before when there seems to be no real need .

          I'm inclined to say we should not bother adapting JAXB in Jenkins so plugins can work like before forever. At least I hope my summary above shows there's no real traction to do this.

          I would rather encourage plugins to not depend on JAXB anymore.

          What do you think?

          Baptiste Mathus added a comment - I've worked a bit in the last few days on checking the current state of Jenkins without the --add-modules switches: I've tried looking for an easy case to reproduce this issue, but it's actually not anymore easy to make Jenkins fail for a JAXB error, which is good. That is mostly because since when the issues were filed, many were fixed in plac to not use JAXB anymore where this was possible. On the opensource side, it's close to impossible to check through the UI because of a few plugins have the majority javax.xml.bind imports https://github.com/search?q=org%3Ajenkinsci+%22javax.xml.bind%22&unscoped_q=%22javax.xml.bind%22 , so I had to go through the GitHub API using curl and some shell scripting to get the full list. On the OSS side, the full list of repositories using "javax.xml.bind": 286 "coverity-plugin" 194 "inflectra-spira-integration-plugin" 105 "zephyr-enterprise-test-management-plugin" 24 "codebeamer-coverage-publisher-plugin" 21 "codesonar-plugin" 14 "java-client-api" 13 "packageversion-plugin" 9 "performance-signature-dynatrace-plugin" 9 "hp-operations-orchestration-plugin" 8 "meliora-testlab-plugin" 8 "mabl-integration-plugin" 8 "jelly" 8 "app.io-plugin" 7 "sloccount-plugin" 7 "jira-issues-versioning-plugin" 7 "dom4j" 6 "extras-ec2-launcher" 6 "dynatrace-plugin" 5 "xlrelease-plugin" 5 "xcode-plugin" 5 "libdtkit" 5 "github-coverage-reporter-plugin" 4 "zanata-plugin" 3 "cppcheck-plugin" 2 "ucm4svn-plugin" 2 "tfs-plugin" 2 "stride-notification-plugin" 2 "refit-plugin" 2 "notification-plugin" 2 "maven-metadata-plugin" 2 "hp-quality-center-plugin" 2 "ec2-deployment-dashboard" 2 "docker-workflow-plugin" 2 "docker" 1 "windows-azure-storage-plugin" 1 "timestamper-plugin" 1 "testinium-plugin" 1 "testcomplete-plugin" 1 "testcomplete11-xunit-plugin" 1 "qualityclouds-plugin" 1 "policycenter-gate-validator-plugin" 1 "pipeline-aws-plugin" 1 "performance-plugin" 1 "maven-hudson-dev-plugin" 1 "maven-artifact-choicelistprovider-plugin" 1 "localization-plugin" 1 "klocwork-plugin" 1 "jwsdp-sqe-plugin" 1 "job-import-plugin" 1 "jenkow-plugin" 1 "jclouds-plugin" 1 "jaxb-plugin" 1 "ibm-cloud-devops-plugin" 1 "fortify360-plugin" 1 "dtkit-plugin" 1 "agile-cockpit-notification-plugin" 1 "acceptance-test-harness" Some notes the biggest user above, coverity-plugin, is actually already deprecated: https://github.com/jenkinsci/coverity-plugin and replaced by https://github.com/jenkinsci/synopsys-coverity-plugin the second one, inflectra-spira-integration-plugin, is installed 7 times worldwide https://plugins.jenkins.io/inflectra-spira-integration https://plugins.jenkins.io/zephyr-enterprise-test-management is installed 163 times codebeamer-coverage-publisher-plugin installed 62 times codesonar has a decent install number (2200+). This plugin is maintained by Praqma, generally a very active contributor in the Jenkins community, so I filed https://issues.jenkins-ci.org/browse/JENKINS-55572 to see if this could be fixed there too the rest look generally like plugins that are either not used a lot, or quite easily fixable given they use JAXB in very few places, ala https://github.com/jenkinsci/instance-identity-module/pull/11 for instance that was fixed by Kohsuke. Conclusion: Given: the long term right fix is likely to actually not need --add-modules anymore and allow plugins to do like nothing changed we already somehow communicated the fact we were shifting gears , and interpreting to my advantage "we are willing to break some stuff to gain what we really need" . In other words: yes we could obviously fix this, but do we really need/want to spend more time on making JAXB usable like before when there seems to be no real need . I'm inclined to say we should not bother adapting JAXB in Jenkins so plugins can work like before forever. At least I hope my summary above shows there's no real traction to do this. I would rather encourage plugins to not depend on JAXB anymore. What do you think?

          (BTW, I'm now thinking fixing the list above might be a good candidate for FOSDEM's hackfest. It's easy enough from a code perspective and then would allow people to go through the PR cycle, and a bit more code-oriented than the usual typo-fixes we do in such events)

          Baptiste Mathus added a comment - (BTW, I'm now thinking fixing the list above might be a good candidate for FOSDEM's hackfest. It's easy enough from a code perspective and then would allow people to go through the PR cycle, and a bit more code-oriented than the usual typo-fixes we do in such events)

          A new update: after Oleg recommended I try the SlocCount plugin, I did. And it does fail as expected on Java 11 by default. I filed JENKINS-55620 to track this.

          So then I tried dropping directly jaxb-api-2.3.0.jar and jaxb-core-2.3.0.1.jar into various locations to see how it behaves. Everything worked fine.

            Java 8 Java 11
          Plugin's WEB-INF/lib  OK OK
          Jenkins' war/WEB-INF/lib OK OK

          To summarize, it seems like hardcoding the dependency against JAXB directly in the plugin would be simpler. I'm not saying it will work in every single case, where/if a given plugin does something more convoluted, trying for instance to use JAXB instances coming from another plugin (which I think, but untested yet, would result in having the classes seen as different given classloaders would be different).

          Circling back to my previous point, and adding up to Jesse's comment above:

          which plugins would actually be affected? It is maybe a five-line change to switch the thread context loader¹, and it would be nice to get JAXB out of core if we could.

          • I think I provided some data about "which plugins would actually be affected?": the answer is likely "very few" or something along this.
          • "get JAXB out of core": given my testing on sloccount-plugin, I think this is possible.

          Now, I'm aware I've not tested the whole Jenkins ecosystem, but so I'm still wondering if this subject is as prevalent as we were imagining. I guess, again, the fact many fixes were pushed already makes this less and less of an issue.

          I'm eager to hear what people think.

          Baptiste Mathus added a comment - A new update: after Oleg recommended I try the SlocCount plugin, I did. And it does fail as expected on Java 11 by default. I filed JENKINS-55620 to track this. So then I tried dropping directly jaxb-api-2.3.0.jar and jaxb-core-2.3.0.1.jar into various locations to see how it behaves. Everything worked fine.   Java 8 Java 11 Plugin's WEB-INF/lib  OK OK Jenkins' war/WEB-INF/lib OK OK To summarize, it seems like hardcoding the dependency against JAXB directly in the plugin would be simpler. I'm not saying it will work in every single case, where/if a given plugin does something more convoluted, trying for instance to use JAXB instances coming from another plugin (which I think, but untested yet, would result in having the classes seen as different given classloaders would be different). Circling back to my previous point, and adding up to Jesse's comment above: which plugins would actually be affected? It is maybe a five-line change to switch the thread context loader¹, and it would be nice to get JAXB out of core if we could. I think I provided some data about "which plugins would actually be affected?": the answer is likely "very few" or something along this. "get JAXB out of core": given my testing on sloccount-plugin, I think this is possible. Now, I'm aware I've not tested the whole Jenkins ecosystem, but so I'm still wondering if this subject is as prevalent as we were imagining. I guess, again, the fact many fixes were pushed already makes this less and less of an issue. I'm eager to hear what people think.

          Filed JENKINS-55944 to track removing JAXB usages where possible.

          Baptiste Mathus added a comment - Filed  JENKINS-55944 to track removing JAXB usages where possible.

          AFAICT, this is superseded by JENKINS-55681, which landed in 2.163 as per https://github.com/jenkinsci/jenkins/pull/3865

          Baptiste Mathus added a comment - AFAICT, this is superseded by JENKINS-55681 , which landed in 2.163 as per https://github.com/jenkinsci/jenkins/pull/3865

            kohsuke Kohsuke Kawaguchi
            oleg_nenashev Oleg Nenashev
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: