Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46579

Maven Event Spy seems to still have a thread safe problem

      Exception:

      ERROR: [withMaven] WARNING Exception parsing the logs generated by the Jenkins Maven Event Spy /var/lib/jenkins/workspace/_appia=gui/F__appia_-_gui_-_master@tmp/withMaven1afdee4c/maven-spy-20170831-215714-662.log, ignore file.  Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org 
      ERROR: org.xml.sax.SAXParseException; lineNumber: 3824; columnNumber: 53; XML document structures must start and end within the same entity.
      

          [JENKINS-46579] Maven Event Spy seems to still have a thread safe problem

          Cyrille Le Clerc added a comment - - edited

          Hello pino, we have fixed such issue in version 2.0.1 on April 2017. What version do you use?

          As you say "still have", you seem to be in a more recent version.

          Cyrille Le Clerc added a comment - - edited Hello pino , we have fixed such issue in version 2.0.1 on April 2017. What version do you use? As you say "still have", you seem to be in a more recent version.

          Pino Silvaggio added a comment - - edited

          I am aware that an issue regarding this was resolved. However, running 3.0.0 with a threaded build results in this error from time to time.

          Pino Silvaggio added a comment - - edited I am aware that an issue regarding this was resolved. However, running 3.0.0 with a threaded build results in this error from time to time.

          Alvaro Lobato added a comment -

          Hi,

           

          Can you attach one of the broken log files? Also any information you could give us about the pipeline you are using and the pom, etc would be helpful to try to reproduce it.

           

          Alvaro Lobato added a comment - Hi,   Can you attach one of the broken log files? Also any information you could give us about the pipeline you are using and the pom, etc would be helpful to try to reproduce it.  

          Yes I will try to give you more info.

          I am debugging right now as this is important to us to keep the trust in the ci.

          However, quickly glancing at the code I am wondering if the source of the problem is not the onEvent method. Nothing prevents the different threads to output content in the middle of another unfinished task. I know the underlying classes use thread safe operations but handlers will use more than 1 output operation and therefore another thread can output stuff right in the middle of it.

          But that's just speculation after looking at the code on my cellphone fast fast.

          Pino Silvaggio added a comment - Yes I will try to give you more info. I am debugging right now as this is important to us to keep the trust in the ci. However, quickly glancing at the code I am wondering if the source of the problem is not the onEvent method. Nothing prevents the different threads to output content in the middle of another unfinished task. I know the underlying classes use thread safe operations but handlers will use more than 1 output operation and therefore another thread can output stuff right in the middle of it. But that's just speculation after looking at the code on my cellphone fast fast.

          Alvaro Lobato added a comment -

          I'm not sure if that is the problem given that all the handlers print in a single operation passing the Xpp3Dom object and the method is thread safe.

          Alvaro Lobato added a comment - I'm not sure if that is the problem given that all the handlers print in a single operation passing the Xpp3Dom object and the method is thread safe.

          Not an easy one.

          • I'm wondering if there could be 2 maven builds starting at the very same milisecond (here).
          • Can you share with us more details? the pom.xml, the spy logs (see here)...

          Cyrille Le Clerc added a comment - Not an easy one. I'm wondering if there could be 2 maven builds starting at the very same milisecond ( here ). Can you share with us more details? the pom.xml, the spy logs (see here )...

          That's funny.

          I was about to write you about the exact same thing. Because I just isolated the problem and it looks to be it.

          I don't think it's ever a good idea to use a timestamp as a "unique" id for anything. I will run some tests but I am pretty sure that's the problem as I have added output and see 2 outFile with the same name.

          Pino Silvaggio added a comment - That's funny. I was about to write you about the exact same thing. Because I just isolated the problem and it looks to be it. I don't think it's ever a good idea to use a timestamp as a "unique" id for anything. I will run some tests but I am pretty sure that's the problem as I have added output and see 2 outFile with the same name.

          Ok, so yeah I've got more than 1 case. There one where if the build is reentrant and 2 or more concurrent builds run we might get the same file name.

          However, this is not the only case. I am trying to package details on the problem.

          Pino Silvaggio added a comment - Ok, so yeah I've got more than 1 case. There one where if the build is reentrant and 2 or more concurrent builds run we might get the same file name. However, this is not the only case. I am trying to package details on the problem.

          Alvaro Lobato added a comment -

          Yes I found that one and already have a fix. Will push that fix while waiting for the details on the other

          Alvaro Lobato added a comment - Yes I found that one and already have a fix. Will push that fix while waiting for the details on the other

          Alvaro Lobato added a comment -

          I don't think it is two builds at the same time but a fork on the maven process. Anyway I had already proved that issue with a unit test and have a fix, I'm sending the PR in a while.

          Alvaro Lobato added a comment - I don't think it is two builds at the same time but a fork on the maven process. Anyway I had already proved that issue with a unit test and have a fix, I'm sending the PR in a while.

          Thanks alobato!

          Cyrille Le Clerc added a comment - Thanks alobato !

          Alvaro Lobato added a comment -

          Alvaro Lobato added a comment - https://github.com/jenkinsci/pipeline-maven-plugin/pull/91

          So, after many many builds the pattern seems to be that maven spy log is incomplete and of course it fails.

          Looks like this:

          <ExecutionEvent type="MojoStarted" class="org.apache.maven.lifecycle.internal.DefaultExecutionEvent" _time="2017-09-01 21:05:52.118">
              <project baseDir="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client" file="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/pom.xml" groupId="com.expretio.appia.ejbs" name="Appia Platform : EJB : User Client" artifactId="appia-user-ejb-client" version="4.54.0-SNAPSHOT">
                <build sourceDirectory="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/src/main/java" directory="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/target"/>
              </project>
              <plugin executionId="default-cli" goal="jar" groupId="org.apache.maven.plugins" artifactId="maven-javadoc-plugin" version="2.10.4">
                <additionalJOption>${additionalJOption}</additionalJOption>
                <additionalparam>${additionalparam}</additionalparam>
                <aggregate>${aggregate}</aggregate>
                <applyJavadocSecurityFix>${maven.javadoc.applyJavadocSecurityFix}</applyJavadocSecurityFix>
                <attach>${attach}</attach>
                <author>${author}</author>
                <bootclasspath>${bootclasspath}</bootclasspath>
                <bootclasspathArtifacts>${bootclasspathArtifacts}</bootclasspathArtifacts>
                <bottom>${bottom}</bottom>
                <breakiterator>${breakiterator}</breakiterator>
                <charset>${charset}</charset>
                <classifier>${maven.javadoc.classifier}</classifier>
                <debug>${debug}</debug>
                <defaultManifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</defaultManifestFile>
                <destDir>${destDir}</destDir>
                <detectJavaApiLink>${detectJavaApiLink}</detectJavaApiLink>
                <detectLinks>${detectLinks}</detectLinks>
                <detectOfflineLinks>${detectOfflineLinks}</detectOfflineLinks>
                <docencoding>${docencoding}</docencoding>
                <docfilessubdirs>${docfilessubdirs}</docfilessubdirs>
                <doclet>${doclet}</doclet>
                <docletArtifact>${docletArtifact}</docletArtifact>
                <docletArtifacts>${docletArtifacts}</docletArtifacts>
                <docletPath>${docletPath}</docletPath>
                <doctitle>${doctitle}</doctitle>
                <encoding>${encoding}</encoding>
                <excludePackageNames>${excludePackageNames}</excludePackageNames>
                <excludedocfilessubdir>${excludedocfilessubdir}</excludedocfilessubdir>
                <extdirs>${extdirs}</extdirs>
                <failOnError>${maven.javadoc.failOnError}</failOnError>
                <finalName>${project.build.finalName}</finalName>
                <footer>${footer}</footer>
                <groups>${groups}</groups>
                <header>${header}</header>
                <helpfile>${helpfile}</helpfile>
                <includeDependencySources>false</includeDependencySources>
                <includeTransitiveDependencySources>false</includeTransitiveDependencySources>
                <isOffline>${settings.offline}</isOffline>
                <jarOutputDirectory>${project.build.directory}</jarOutputDirectory>
                <javaApiLinks>${javaApiLinks}</javaApiLinks>
                <javadocDirectory>${basedir}/src/main/javadoc</javadocDirectory>
                <javadocExecutable>${javadocExecutable}</javadocExecutable>
                <javadocOptionsDir>${project.build.directory}/javadoc-bundle-options</javadocOptionsDir>
                <javadocVersion>${javadocVersion}</javadocVersion>
                <keywords>${keywords}</keywords>
                <links>${links}</links>
                <linksource>${linksource}</linksource>
                <localRepository>${localRepository}</localRepository>
                <locale>${locale}</locale>
                <maxmemory>${maxmemory}</maxmemory>
                <minmemory>${minmemory}</minmemory>
                <nocomment>${nocomment}</nocomment>
                <nodeprecated>${nodeprecated}</nodeprecated>
                <nodeprecatedlist>${nodeprecatedlist}</nodeprecatedlist>
                <nohelp>${nohelp}</nohelp>
                <noindex>${noindex}</noindex>
                <nonavbar>${nonavbar}</nonavbar>
                <nooverview>${nooverview}</nooverview>
                <noqualifier>${noqualifier}</noqualifier>
                <nosince>${nosince}</nosince>
                <notimestamp>${notimestamp}</notimestamp>
                <notree>${no
          

          Pino Silvaggio added a comment - So, after many many builds the pattern seems to be that maven spy log is incomplete and of course it fails. Looks like this: <ExecutionEvent type="MojoStarted" class="org.apache.maven.lifecycle.internal.DefaultExecutionEvent" _time="2017-09-01 21:05:52.118"> <project baseDir="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client" file="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/pom.xml" groupId="com.expretio.appia.ejbs" name="Appia Platform : EJB : User Client" artifactId="appia-user-ejb-client" version="4.54.0-SNAPSHOT"> <build sourceDirectory="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/src/main/java" directory="/var/lib/jenkins/workspace/_appia=platform/F__appia_-_platform_-_master/ejb/user/ejb-client/target"/> </project> <plugin executionId="default-cli" goal="jar" groupId="org.apache.maven.plugins" artifactId="maven-javadoc-plugin" version="2.10.4"> <additionalJOption>${additionalJOption}</additionalJOption> <additionalparam>${additionalparam}</additionalparam> <aggregate>${aggregate}</aggregate> <applyJavadocSecurityFix>${maven.javadoc.applyJavadocSecurityFix}</applyJavadocSecurityFix> <attach>${attach}</attach> <author>${author}</author> <bootclasspath>${bootclasspath}</bootclasspath> <bootclasspathArtifacts>${bootclasspathArtifacts}</bootclasspathArtifacts> <bottom>${bottom}</bottom> <breakiterator>${breakiterator}</breakiterator> <charset>${charset}</charset> <classifier>${maven.javadoc.classifier}</classifier> <debug>${debug}</debug> <defaultManifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</defaultManifestFile> <destDir>${destDir}</destDir> <detectJavaApiLink>${detectJavaApiLink}</detectJavaApiLink> <detectLinks>${detectLinks}</detectLinks> <detectOfflineLinks>${detectOfflineLinks}</detectOfflineLinks> <docencoding>${docencoding}</docencoding> <docfilessubdirs>${docfilessubdirs}</docfilessubdirs> <doclet>${doclet}</doclet> <docletArtifact>${docletArtifact}</docletArtifact> <docletArtifacts>${docletArtifacts}</docletArtifacts> <docletPath>${docletPath}</docletPath> <doctitle>${doctitle}</doctitle> <encoding>${encoding}</encoding> <excludePackageNames>${excludePackageNames}</excludePackageNames> <excludedocfilessubdir>${excludedocfilessubdir}</excludedocfilessubdir> <extdirs>${extdirs}</extdirs> <failOnError>${maven.javadoc.failOnError}</failOnError> <finalName>${project.build.finalName}</finalName> <footer>${footer}</footer> <groups>${groups}</groups> <header>${header}</header> <helpfile>${helpfile}</helpfile> <includeDependencySources>false</includeDependencySources> <includeTransitiveDependencySources>false</includeTransitiveDependencySources> <isOffline>${settings.offline}</isOffline> <jarOutputDirectory>${project.build.directory}</jarOutputDirectory> <javaApiLinks>${javaApiLinks}</javaApiLinks> <javadocDirectory>${basedir}/src/main/javadoc</javadocDirectory> <javadocExecutable>${javadocExecutable}</javadocExecutable> <javadocOptionsDir>${project.build.directory}/javadoc-bundle-options</javadocOptionsDir> <javadocVersion>${javadocVersion}</javadocVersion> <keywords>${keywords}</keywords> <links>${links}</links> <linksource>${linksource}</linksource> <localRepository>${localRepository}</localRepository> <locale>${locale}</locale> <maxmemory>${maxmemory}</maxmemory> <minmemory>${minmemory}</minmemory> <nocomment>${nocomment}</nocomment> <nodeprecated>${nodeprecated}</nodeprecated> <nodeprecatedlist>${nodeprecatedlist}</nodeprecatedlist> <nohelp>${nohelp}</nohelp> <noindex>${noindex}</noindex> <nonavbar>${nonavbar}</nonavbar> <nooverview>${nooverview}</nooverview> <noqualifier>${noqualifier}</noqualifier> <nosince>${nosince}</nosince> <notimestamp>${notimestamp}</notimestamp> <notree>${no

          Pino Silvaggio added a comment - - edited

          Is it possible the StepContext is notified that the executioon ended before it actually ended? Or maybe the something is not properly "released"...

          Pino Silvaggio added a comment - - edited Is it possible the StepContext is notified that the executioon ended before it actually ended? Or maybe the something is not properly "released"...

          Alvaro Lobato added a comment -

          How does your pipeline look? What maven goals are you executing?

          Alvaro Lobato added a comment - How does your pipeline look? What maven goals are you executing?

          Pino Silvaggio added a comment - - edited

          I have attached a build log where it fails, there's some debugging output. It looks like maven isn't done when maven spy fails.

          mvn -Dmaven.test.failure.ignore=true -Dmaven.javadoc.failOnError=false -e -U -T 2 clean deploy -DaltDeploymentRepository=xo-snapshots::default::http://repo.expretio.com/artifactory/m2-xo-snapshots
          

          build-console.zip

          Pino Silvaggio added a comment - - edited I have attached a build log where it fails, there's some debugging output. It looks like maven isn't done when maven spy fails. mvn -Dmaven.test.failure.ignore=true -Dmaven.javadoc.failOnError=false -e -U -T 2 clean deploy -DaltDeploymentRepository=xo-snapshots::default::http://repo.expretio.com/artifactory/m2-xo-snapshots build-console.zip

          Code changed in jenkins
          User: Alvaro Lobato
          Path:
          maven-spy/src/main/java/org/jenkinsci/plugins/pipeline/maven/eventspy/reporter/FileMavenEventReporter.java
          maven-spy/src/test/java/org/jenkinsci/plugins/pipeline/maven/eventspy/JenkinsMavenEventSpyMTTest.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/810fe9a4c7a61c4c7ccb7a35636811e22d437480
          Log:
          JENKINS-46579 - Concurrency problems with spy log (#91)

          JENKINS-46579 maven event spy: Fix concurrency issue in the handling of the output file

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alvaro Lobato Path: maven-spy/src/main/java/org/jenkinsci/plugins/pipeline/maven/eventspy/reporter/FileMavenEventReporter.java maven-spy/src/test/java/org/jenkinsci/plugins/pipeline/maven/eventspy/JenkinsMavenEventSpyMTTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/810fe9a4c7a61c4c7ccb7a35636811e22d437480 Log: JENKINS-46579 - Concurrency problems with spy log (#91) JENKINS-46579 maven event spy: Fix concurrency issue in the handling of the output file

          FYI I have created JENKINS-46609 . pino could the Maven process could have been killed by the operating system? I saw this problem in docker/cgroup environments.

          Cyrille Le Clerc added a comment - FYI I have created JENKINS-46609 . pino could the Maven process could have been killed by the operating system? I saw this problem in docker/cgroup environments.

          Well well, that was it. The maven process was getting killed for no apparent reason. I played with the java Xmx and it looks much better now. I don't understand why it would fail with Xmx3G when in dev we use Xmx2G but I have increased this to 4G and it seems to work.

          Pino Silvaggio added a comment - Well well, that was it. The maven process was getting killed for no apparent reason. I played with the java Xmx and it looks much better now. I don't understand why it would fail with Xmx3G when in dev we use Xmx2G but I have increased this to 4G and it seems to work.

          Can you clarify which Xmx you are referring to?
          Do you run this process in a Docker container or in cgroups?
          Do the memory of this agents fits with the sumof the memory of all the known processes ? Slave.jar + mvn process + maybe mvn sub processes
          Can you monitor the memory, CPU and load of this agent to detect problems?

          Cyrille Le Clerc added a comment - Can you clarify which Xmx you are referring to? Do you run this process in a Docker container or in cgroups? Do the memory of this agents fits with the sumof the memory of all the known processes ? Slave.jar + mvn process + maybe mvn sub processes Can you monitor the memory, CPU and load of this agent to detect problems?

          Cyrille Le Clerc added a comment - - edited

          Fixed by https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pipeline-maven/3.0.1-beta-1/pipeline-maven-3.0.1-beta-1.hpi
          We detect when the Maven build has been killed and don't display this confusing XML parsing exception. See Pipeline Maven Plugin> Why do I see messages...

          Cyrille Le Clerc added a comment - - edited Fixed by https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pipeline-maven/3.0.1-beta-1/pipeline-maven-3.0.1-beta-1.hpi We detect when the Maven build has been killed and don't display this confusing XML parsing exception. See Pipeline Maven Plugin> Why do I see messages...

          Cyrille Le Clerc added a comment - pino could you test the 3.0.1-beta? https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.1-beta-2

          I will, I am however on vacation at this time.

          Pino Silvaggio added a comment - I will, I am however on vacation at this time.

          Thanks pino, as I have already tested, we may release before you are back from holidays

          Cyrille Le Clerc added a comment - Thanks pino , as I have already tested, we may release before you are back from holidays

          v3.0.1

          Cyrille Le Clerc added a comment - v3.0.1

          we run version 3.6.11 and still see this from time to time:

           

          ERROR: [withMaven] WARNING Exception parsing the logs generated by the Jenkins Maven Event Spy /home/ec2-user/workspace/org_yooture_develop@tmp/withMaven5705f0d0/maven-spy-20190520-135648-9715492809875305253764.log, ignore file.  Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org 
          ERROR: org.xml.sax.SAXParseException; lineNumber: 18407; columnNumber: 263; XML document structures must start and end within the same entity. 

          Dominik Bartholdi added a comment - we run version 3.6.11 and still see this from time to time:   ERROR: [withMaven] WARNING Exception parsing the logs generated by the Jenkins Maven Event Spy /home/ec2-user/workspace/org_yooture_develop@tmp/withMaven5705f0d0/maven-spy-20190520-135648-9715492809875305253764.log, ignore file. Please report a bug associated for the component 'pipeline-maven-plugin' at https: //issues.jenkins-ci.org ERROR: org.xml.sax.SAXParseException; lineNumber: 18407; columnNumber: 263; XML document structures must start and end within the same entity.

            Unassigned Unassigned
            pino Pino Silvaggio
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: