Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57833

All builds with maven pipeline plugin fail with java.lang.AbstractMethodError while parsing event spy log

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • pipeline-maven-plugin
    • None
    • Internal

      Environment:

      Jenkins 2.179

      Runtime: Docker jenkins standard container from the offical repository (jenkins/jenkins)

      Plugins

      Ant Plugin - 1.9
Apache HttpComponents Client 4.x API Plugin - 4.5.5-3.0
Authentication Tokens API Plugin - 1.3
Autofavorite for Blue Ocean - 1.2.4
Basic Branch Build Strategies Plugin - 1.3.2
BD Protex Plugin for Jenkins CI - 1.4.1
Bitbucket Branch Source Plugin - 2.4.4
Bitbucket Pipeline for Blue Ocean - 1.16.0
Black Duck Hub Plug-In for Jenkins - 1.5.4
Blue Ocean - 1.16.0
Blue Ocean Core JS - 1.16.0
Blue Ocean Executor Info - 1.16.0
Blue Ocean Pipeline Editor - 1.16.0
bouncycastle API Plugin - 2.17
Branch API Plugin - 2.5.2
Build Blocker Plugin - 1.7.3
Build Monitor View - 1.12+build.201809061734
Command Agent Launcher Plugin - 1.3
Common API for Blue Ocean - 1.16.0
Conditional BuildStep - 1.3.6
Config API for Blue Ocean - 1.16.0
Config File Provider Plugin - 3.6
Credentials Binding Plugin - 1.19
Credentials Plugin - 2.2.0
Dashboard for Blue Ocean - 1.16.0
Design Language - 1.16.0
Discard Old Build plugin - 1.05
Display URL API - 2.3.1
Display URL for Blue Ocean - 2.3.0
Docker API Plugin - 3.0.14
Docker Commons Plugin - 1.15
Docker Pipeline - 1.18
Docker plugin - 1.1.6
Durable Task Plugin - 1.29
Email Extension Plugin - 2.66
Embeddable Build Status Plugin - 2.0.1
Events API for Blue Ocean - 1.16.0
External Monitor Job Type Plugin - 1.7
Favorite - 2.3.2
Folders Plugin - 6.8
Git client plugin - 2.7.7
Git Pipeline for Blue Ocean - 1.16.0
Git plugin - 3.10.0
GIT server Plugin - 1.7
GitHub API Plugin - 1.95
GitHub Branch Source Plugin - 2.5.3
GitHub Pipeline for Blue Ocean - 1.16.0
GitHub plugin - 1.29.4
Gradle Plugin - 1.32
H2 API Plugin - 1.4.199
Handy Uri Templates 2.x API Plugin - 2.1.7-1.0
HTML Publisher plugin - 1.18
i18n for Blue Ocean - 1.16.0
Icon Shim Plugin - 2.0.3
Jackson 2 API Plugin - 2.9.9
Javadoc Plugin - 1.5
JavaScript GUI Lib: ACE Editor bundle plugin - 1.1
JavaScript GUI Lib: Handlebars bundle plugin - 1.1.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin - 1.2.1
JavaScript GUI Lib: Moment.js bundle plugin - 1.1.1
JDK Tool Plugin - 1.2
JIRA Integration for Blue Ocean - 1.16.0
JIRA plugin - 3.0.7
Job DSL - 1.74
jQuery plugin - 1.12.4-0
jQuery UI plugin - 1.0.2
JSch dependency plugin - 0.1.55
JUnit Plugin - 1.28
JWT for Blue Ocean - 1.16.0
Keycloak Authentication Plugin - 2.3.0
Kubernetes Credentials Plugin - 0.4.0
Kubernetes plugin - 1.15.5
LDAP Plugin - 1.20
Mail Watcher Plugin - 1.16
Mailer Plugin - 1.23
Mask Passwords Plugin - 2.12.0
Matrix Authorization Strategy Plugin - 2.4.2
Matrix Project Plugin - 1.14
Maven Integration plugin - 3.2
Maven Release Plug-in Plug-in - 0.14.0
Mercurial plugin - 2.6
Nexus Platform Plugin - 3.5.20190425-152158.c63841b
OWASP Markup Formatter Plugin - 1.5
PAM Authentication plugin - 1.5.1
Parameterized Trigger plugin - 2.35.2
Personalization for Blue Ocean - 1.16.0
Pipeline Graph Analysis Plugin - 1.10
Pipeline implementation for Blue Ocean - 1.16.0
Pipeline Maven Integration Plugin - 3.7.1
Pipeline SCM API for Blue Ocean - 1.16.0
Pipeline Utility Steps - 2.3.0
Pipeline: API - 2.34
Pipeline: Basic Steps - 2.17
Pipeline: Build Step - 2.9
Pipeline: Declarative - 1.3.8
Pipeline: Declarative Agent API - 1.1.1
Pipeline: Declarative Extension Points API - 1.3.8
Pipeline: Groovy - 2.70
Pipeline: Input Step - 2.10
Pipeline: Job - 2.32
Pipeline: Milestone Step - 1.3.1
Pipeline: Model API - 1.3.8
Pipeline: Multibranch - 2.21
Pipeline: Nodes and Processes - 2.31
Pipeline: REST API Plugin - 2.11
Pipeline: SCM Step - 2.7
Pipeline: Shared Groovy Libraries - 2.13
Pipeline: Stage Step - 2.3
Pipeline: Stage Tags Metadata - 1.3.8
Pipeline: Stage View Plugin - 2.11
Pipeline: Step API - 2.20
Pipeline: Supporting APIs - 3.3
Plain Credentials Plugin - 1.5
Pub-Sub "light" Bus - 1.12
REST API for Blue Ocean - 1.16.0
REST Implementation for Blue Ocean - 1.16.0
RocketChat Notifier - 1.4.3
Role-based Authorization Strategy - 2.11
Run Condition Plugin - 1.2
SCM API Plugin - 2.4.1
Script Security Plugin - 1.60
Server Sent Events (SSE) Gateway Plugin - 1.17
SSH Credentials Plugin - 1.16
SSH Slaves plugin - 1.29.4
Structs Plugin - 1.19
Token Macro Plugin - 2.7
Trilead API Plugin - 1.0.3
Variant Plugin - 1.2
Web for Blue Ocean - 1.16.0
WMI Windows Agents Plugin - 1.4

      Dependencies:

       antlr:antlr:2.7.6
 aopalliance:aopalliance:1.0
 args4j:args4j:2.0.31
 com.github.jnr:jffi:1.2.16
 com.github.jnr:jffi:1.2.17
 com.github.jnr:jnr-constants:0.9.9
 com.github.jnr:jnr-ffi:2.1.8
 com.github.jnr:jnr-posix:3.0.45
 com.github.jnr:jnr-x86asm:1.0.2
 com.google.code.findbugs:annotations:3.0.1
 com.google.code.findbugs:jsr305:3.0.1
 com.google.guava:guava:11.0.1
 com.google.inject:guice:4.0
 com.infradna.tool:bridge-method-annotation:1.13
 com.jcraft:jzlib:1.1.3-kohsuke-1
 commons-beanutils:commons-beanutils:1.9.3
 commons-codec:commons-codec:1.9
 commons-collections:commons-collections:3.2.2
 commons-digester:commons-digester:2.1
 commons-discovery:commons-discovery:0.4
 commons-fileupload:commons-fileupload:1.3.1-jenkins-2
 commons-httpclient:commons-httpclient:3.1-jenkins-1
 commons-io:commons-io:2.6
 commons-jelly:commons-jelly-tags-fmt:1.0
 commons-jelly:commons-jelly-tags-xml:1.1
 commons-lang:commons-lang:2.6
 com.sun.solaris:embedded_su4j:1.1
 com.sun.xml.txw2:txw2:20110809
 io.github.stephenc.crypto:self-signed-cert-generator:1.0.0
 io.jenkins.stapler:jenkins-stapler-support:1.0
 javax.annotation:javax.annotation-api:1.2
 javax.inject:javax.inject:1
 javax.mail:mail:1.4.4
 javax.servlet.jsp.jstl:javax.servlet.jsp.jstl-api:1.2.1
 javax.xml.stream:stax-api:1.0-2
 jaxen:jaxen:1.1-beta-11
 jfree:jcommon:1.0.12
 jfree:jfreechart:1.0.9
 jline:jline:2.12
 junit:junit:4.12
 net.i2p.crypto:eddsa:0.3.0
 net.java.dev.jna:jna:4.5.2
 net.java.sezpoz:sezpoz:1.13
 net.jcip:jcip-annotations:1.0
 net.sf.ezmorph:ezmorph:1.0.6
 net.sf.kxml:kxml2:2.3.0
 org.acegisecurity:acegi-security:1.0.7
 org.apache.ant:ant:1.9.2
 org.apache.ant:ant-launcher:1.9.2
 org.apache.commons:commons-compress:1.10
 org.apache.sshd:sshd-core:1.7.0
 org.codehaus.groovy:groovy-all:2.4.12
 org.codehaus.woodstox:wstx-asl:3.2.9
 org.connectbot.jbcrypt:jbcrypt:1.0.0
 org.fusesource.jansi:jansi:1.11
 org.hamcrest:hamcrest-core:1.3
 org.jenkins-ci:annotation-indexer:1.12
 org.jenkins-ci:bytecode-compatibility-transformer:2.0-beta-2
 org.jenkins-ci:commons-jelly:1.1-jenkins-20120928
 org.jenkins-ci:commons-jexl:1.1-jenkins-20111212
 org.jenkins-ci:constant-pool-scanner:1.2
 org.jenkins-ci:crypto-util:1.1
 org.jenkins-ci.dom4j:dom4j:1.6.1-jenkins-4
 org.jenkins-ci.main:cli:2.179
 org.jenkins-ci.main:jenkins-core:2.179
org.jenkins-ci.main:jenkins-war:2.179
 org.jenkins-ci.main:remoting:3.30
 org.jenkins-ci:memory-monitor:1.9
 org.jenkins-ci.modules:instance-identity:2.2
 org.jenkins-ci.modules:launchd-slave-installer:1.2
 org.jenkins-ci.modules:slave-installer:1.6
 org.jenkins-ci.modules:ssh-cli-auth:1.5
 org.jenkins-ci.modules:sshd:2.6
 org.jenkins-ci.modules:systemd-slave-installer:1.1
 org.jenkins-ci.modules:upstart-slave-installer:1.1
 org.jenkins-ci.modules:windows-slave-installer:1.11
 org.jenkins-ci.plugins.icon-shim:icon-set:1.0.5
 org.jenkins-ci:symbol-annotation:1.1
 org.jenkins-ci:task-reactor:1.5
 org.jenkins-ci:trilead-ssh2:build-217-jenkins-14
 org.jenkins-ci.ui:bootstrap:1.3.2
 org.jenkins-ci.ui:handlebars:1.1.1
 org.jenkins-ci.ui:jquery-detached:1.2
 org.jenkins-ci.ui:jquery-detached:1.2.1
 org.jenkins-ci:version-number:1.6
 org.jenkins-ci:winstone:5.3
 org.jmdns:jmdns:3.5.5
 org.jruby.ext.posix:jna-posix:1.0.3-jenkins-1
 org.jvnet.hudson:activation:1.1.1-hudson-1
 org.jvnet.hudson:commons-jelly-tags-define:1.0.1-hudson-20071021
 org.jvnet.hudson:jtidy:4aug2000r7-dev-hudson-1
 org.jvnet.hudson:xstream:1.4.7-jenkins-1
 org.jvnet.localizer:localizer:1.26
 org.jvnet.robust-http-client:robust-http-client:1.2
 org.jvnet:tiger-types:2.2
 org.jvnet.winp:winp:1.28
 org.kohsuke:access-modifier-annotation:1.14
 org.kohsuke:akuma:1.10
 org.kohsuke:asm5:5.0.1
 org.kohsuke:asm6:6.2
 org.kohsuke.jinterop:j-interop:2.0.6-kohsuke-1
 org.kohsuke.jinterop:j-interopdeps:2.0.6-kohsuke-1
 org.kohsuke:libpam4j:1.11
 org.kohsuke:libzfs:0.8
 org.kohsuke.stapler:json-lib:2.4-jenkins-2
 org.kohsuke.stapler:stapler:1.257
 org.kohsuke.stapler:stapler-adjunct-codemirror:1.3
 org.kohsuke.stapler:stapler-adjunct-timeline:1.5
 org.kohsuke.stapler:stapler-adjunct-zeroclipboard:1.3.5-1
 org.kohsuke.stapler:stapler-groovy:1.257
 org.kohsuke.stapler:stapler-jelly:1.257
 org.kohsuke.stapler:stapler-jrebel:1.257
 org.kohsuke:trilead-putty-extension:1.2
 org.kohsuke:windows-package-checker:1.2
 org.ow2.asm:asm:5.0.3
 org.ow2.asm:asm-analysis:5.0.3
 org.ow2.asm:asm-commons:5.0.3
 org.ow2.asm:asm-tree:5.0.3
 org.ow2.asm:asm-util:5.0.3
 org.samba.jcifs:jcifs:1.3.17-kohsuke-1
 org.slf4j:jcl-over-slf4j:1.7.25
 org.slf4j:log4j-over-slf4j:1.7.25
 org.slf4j:slf4j-api:1.7.25
 org.slf4j:slf4j-jdk14:1.7.25
 org.springframework:spring-aop:2.5.6.SEC03
 org.springframework:spring-beans:2.5.6.SEC03
 org.springframework:spring-context:2.5.6.SEC03
 org.springframework:spring-context-support:2.5.6.SEC03
 org.springframework:spring-core:2.5.6.SEC03
 org.springframework:spring-dao:1.2.9
 org.springframework:spring-jdbc:1.2.9
 org.springframework:spring-web:2.5.6.SEC03
 org.springframework:spring-webmvc:2.5.6.SEC03
 oro:oro:2.0.8
 relaxngDatatype:relaxngDatatype:20020414
 stax:stax-api:1.0.1
 xpp3:xpp3:1.1.4c
 

      Description

      We are facing an issue since the latest wave of plugin and jenkins updates. The failure occurs at the end of every build while parsing the event spy log within the maven pipeline code even when all steps seem to run successful

       Error

      java.lang.AbstractMethodError
 at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:83)
 at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1108)
 at org.jenkinsci.plugins.pipeline.maven.fix.jenkins49337.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:142)
 at org.jenkinsci.plugins.pipeline.maven.fix.jenkins49337.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:85)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 Finished: FAILURE
  

       

       

          [JENKINS-57833] All builds with maven pipeline plugin fail with java.lang.AbstractMethodError while parsing event spy log

          Markus Till added a comment -

          I get the following output. But why is "source not found" the expected behavoir? shouldn't there be a valid class loaded?

          DocumentBuilderFactory: com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl@57dc5f4f
class com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -> source not found

          Markus Till added a comment - I get the following output. But why is "source not found" the expected behavoir? shouldn't there be a valid class loaded? DocumentBuilderFactory: com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl@57dc5f4f class com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -> source not found

          Markus Till added a comment -

          I don't really get how to resolve this. Should I add -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl to the starting Jenkins Process or to the maven run?

          Markus Till added a comment - I don't really get how to resolve this. Should I add -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl to the starting Jenkins Process or to the maven run?

          Cyrille Le Clerc added a comment -

          Which jvm version are you using?

          Cyrille Le Clerc added a comment - Which jvm version are you using?

          Markus Till added a comment -

          seems adding the -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl  to the JAVA_OPTS seem to work. But I'm still confused why this is needed. 

          Markus Till added a comment - seems adding the -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl  to the JAVA_OPTS seem to work. But I'm still confused why this is needed. 

          Markus Till added a comment - - edited

          the one bundeld with the offical docker image 

          openjdk version "1.8.0_212"
          OpenJDK Runtime Environment (build 1.8.0_212-b04)
          OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

          Markus Till added a comment - - edited the one bundeld with the offical docker image  openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-b04) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

          Cyrille Le Clerc added a comment - - edited

          > But why is "source not found" the expected behavoir? shouldn't there be a valid class loaded?

          "source not found" means that it is provided by the JVM jar(s) directly and this is what we desire. Sometimes, it's an old/outdated XML parsing implementation that it loaded and you get a source such as ".../xercesImpl-2.6.2.jar". It is often caused by plugins forgetting to exclude old versions of Xerces from their dependencies.

          > openjdk version "1.8.0_212"

          This is a JDK that works well. The problem is likely to be a plugin that pull an outdated version of XercesImpl

          class org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -> file:/var/jenkins_home/plugins/registry_reporter/WEB-INF/lib/xercesImpl-2.6.2.jar

          https://issues.jenkins-ci.org/browse/JENKINS-47486?focusedCommentId=318000&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-318000

          Cyrille Le Clerc added a comment - - edited > But why is "source not found" the expected behavoir? shouldn't there be a valid class loaded? "source not found" means that it is provided by the JVM jar(s) directly and this is what we desire. Sometimes, it's an old/outdated XML parsing implementation that it loaded and you get a source such as ".../xercesImpl-2.6.2.jar". It is often caused by plugins forgetting to exclude old versions of Xerces from their dependencies. > openjdk version "1.8.0_212" This is a JDK that works well. The problem is likely to be a plugin that pull an outdated version of XercesImpl class org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -> file:/var/jenkins_home/plugins/registry_reporter/WEB-INF/lib/xercesImpl-2.6.2.jar https://issues.jenkins-ci.org/browse/JENKINS-47486?focusedCommentId=318000&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-318000

          Cyrille Le Clerc added a comment -

          matill can you please search for any xerces jar file in the plugins that you have loaded in your instance. It is typically outdated versions of xerces that cause issues with the DocumentBuilderFactory. Can you on the master run "find -iname "xerces"" under "JENKINS_HOME/plugins".

          It looks like

          $ cd <JENKINS_HOME>/plugins
# verify that we are in the right folder verifying the number of jar files we found
$ find -iname "*.jar" | wc -
$ find -iname "*xerces*.jar"

          Result typically looks like:

          $ find -iname "*.jar" | wc -l
1262
$ find -iname "xerces*.jar"
./analysis-core/WEB-INF/lib/xercesImpl-2.11.0.jar
$

          Cyrille Le Clerc added a comment - matill can you please search for any xerces jar file in the plugins that you have loaded in your instance. It is typically outdated versions of xerces that cause issues with the DocumentBuilderFactory. Can you on the master run " find -iname " xerces " " under " JENKINS_HOME/plugins ". It looks like $ cd <JENKINS_HOME>/plugins # verify that we are in the right folder verifying the number of jar files we found $ find -iname "*.jar" | wc - $ find -iname "*xerces*.jar" Result typically looks like: $ find -iname "*.jar" | wc -l 1262 $ find -iname "xerces*.jar" ./analysis-core/WEB-INF/lib/xercesImpl-2.11.0.jar $

          Martin Dönicke added a comment -

          We had the same issue after updating the the pipeline-maven-plugin

          find -iname "xerces*.jar"
./plugins/analysis-core/WEB-INF/lib/xercesImpl-2.11.0.jar
./plugins/emma/WEB-INF/lib/xercesImpl-2.9.1.jar
./plugins/jdepend/WEB-INF/lib/xercesImpl-2.11.0.jar
./plugins/build-pipeline-plugin/WEB-INF/lib/xercesImpl-2.9.1.jar
./plugins/discard-old-build/WEB-INF/lib/xercesImpl-2.6.2.jar

          Removing the plugin discard-old-build (since we didn't use it anyway) has solved the issue on our side.

          Martin Dönicke added a comment - We had the same issue after updating the the pipeline-maven-plugin find -iname "xerces*.jar" ./plugins/analysis-core/WEB-INF/lib/xercesImpl-2.11.0.jar ./plugins/emma/WEB-INF/lib/xercesImpl-2.9.1.jar ./plugins/jdepend/WEB-INF/lib/xercesImpl-2.11.0.jar ./plugins/build-pipeline-plugin/WEB-INF/lib/xercesImpl-2.9.1.jar ./plugins/discard-old-build/WEB-INF/lib/xercesImpl-2.6.2.jar Removing the plugin discard-old-build (since we didn't use it anyway) has solved the issue on our side.

          Opa added a comment -

          martindoenicke Thanks, that did the trick for us too !

          Opa added a comment - martindoenicke Thanks, that did the trick for us too !

          Cyrille Le Clerc added a comment -

          Closing the issue. The issue is to identify the outdated "xerces jar file" brought by another plugin.

          Cyrille Le Clerc added a comment - Closing the issue. The issue is to identify the outdated "xerces jar file" brought by another plugin.

            cleclerc Cyrille Le Clerc
            matill Markus Till
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: