Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11256

Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave

      linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave
      build fails with

      — maven-install-plugin:2.3.1:install (default-install) @ xxxxxxxx —
      Waiting for Jenkins to finish collecting data
      mavenExecutionResult exceptions not empty
      message : Internal error: java.util.ConcurrentModificationException
      cause : null
      Stack trace :
      org.apache.maven.InternalErrorException: Internal error: java.util.ConcurrentModificationException
      at org.apache.maven.lifecycle.internal.BuilderCommon.handleBuildError(BuilderCommon.java:128)[INFO] Installing xxxxxxxx.jar to xxxxxxxx

      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:95)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:164)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
      at java.util.concurrent.FutureTask.run(FutureTask.java:123)[INFO] Compiling 1 source file to xxxxxxxx

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.util.ConcurrentModificationException
      at java.util.ArrayList.writeObject(ArrayList.java:572)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.ArrayList.writeObject(ArrayList.java:569)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.HashMap.writeObject(HashMap.java:1039)
      at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      at hudson.remoting.Channel.callAsync(Channel.java:708)
      at hudson.maven.Maven3Builder$FilterImpl.executeAsync(Maven3Builder.java:229)
      at hudson.maven.reporters.MavenFingerprinter.postBuild(MavenFingerprinter.java:96)
      at hudson.maven.Maven3Builder$MavenExecutionListener.recordProjectSucceeded(Maven3Builder.java:431)
      at hudson.maven.Maven3Builder$MavenExecutionListener.projectSucceeded(Maven3Builder.java:405)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:74)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:91)
      ... 7 more
      [INFO]
      [INFO] — maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ xxxxxxxx —
      channel stopped
      Archiviere Artefakte
      Finished: FAILURE

          [JENKINS-11256] Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins

          kutzi added a comment -

          Does it happen frequently or only once?

          kutzi added a comment - Does it happen frequently or only once?

          kisa added a comment -

          Every 10-15 builds ... bad enough

          kisa added a comment - Every 10-15 builds ... bad enough

          All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places.

          Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation.

          I will create a pull request for this.

          Stephan Pauxberger added a comment - All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places. Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation. I will create a pull request for this.

          Stephan Pauxberger added a comment - - edited

          Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e, however it landed in the open pull request concerning Parent POM fingerprinting.

          Kutzi, could you look at both of them together or should I split it out again?

          Stephan Pauxberger added a comment - - edited Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e , however it landed in the open pull request concerning Parent POM fingerprinting. Kutzi, could you look at both of them together or should I split it out again?

          kutzi added a comment - - edited

          Thanks for the pull request.
          Yes, please create a separate pull request for each issue.

          Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem?

          I think I'd rather put a synchronized block inside the BuildCallable in postBuild()

          kutzi added a comment - - edited Thanks for the pull request. Yes, please create a separate pull request for each issue. Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem? I think I'd rather put a synchronized block inside the BuildCallable in postBuild()

          Stephan Pauxberger added a comment - - edited

          Old problem with race conditions - if you need them, they don't show up.

          I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module.

          As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something).

          Which leads back to the question: where does ist happen.

          The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace)

          However, I did not found anyone modifying one of the valuelists.

          Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises.

          I will further look into this.

          UPDATE:

          Its an ArrayList in an ArrayList in the reporters map could be some plugin.

          Stephan Pauxberger added a comment - - edited Old problem with race conditions - if you need them, they don't show up. I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module. As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something). Which leads back to the question: where does ist happen. The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace) However, I did not found anyone modifying one of the valuelists. Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises. I will further look into this. UPDATE: Its an ArrayList in an ArrayList in the reporters map could be some plugin.

          kutzi added a comment - - edited

          According to Olivier this might be a bug in Maven 3.0.3:
          http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html

          kutzi added a comment - - edited According to Olivier this might be a bug in Maven 3.0.3: http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html

          Short update:

          I could not reproduce the problem using a test installation, I will try another test using the exact plugin configuration of the actual system.

          Stephan Pauxberger added a comment - Short update: I could not reproduce the problem using a test installation, I will try another test using the exact plugin configuration of the actual system.

          Found the culprit (no plugin):

          hudson.maven.reporters.SurefireArchiver.result is a TestResult object containing an ArrayList (suites). This list is updated whenever a module containing tests completes. Since The Async invoke serializes the Maven3Builder and therefore all its reporters (along with the SurefireArchivers). If the fingerprinter runs during the update of the test result, this error occurs.

          Solution: Make the field hudson.tasks.junit.TestResult.suites a synchronized list (Collections.synchronizedList)

          Stephan Pauxberger added a comment - Found the culprit (no plugin): hudson.maven.reporters.SurefireArchiver.result is a TestResult object containing an ArrayList (suites). This list is updated whenever a module containing tests completes. Since The Async invoke serializes the Maven3Builder and therefore all its reporters (along with the SurefireArchivers). If the fingerprinter runs during the update of the test result, this error occurs. Solution: Make the field hudson.tasks.junit.TestResult.suites a synchronized list (Collections.synchronizedList)

          Stephan Pauxberger added a comment - - edited

          @kutzi: My development environment seems broken. Could you do that change?

          Thx

          Stephan Pauxberger added a comment - - edited @kutzi: My development environment seems broken. Could you do that change? Thx

          kutzi added a comment - - edited

          Ah, forget what I previously wrote here, I missed an important part of your comments.

          Did you manage to reproduce it reliably? If so, do you have a test case?

          kutzi added a comment - - edited Ah, forget what I previously wrote here, I missed an important part of your comments. Did you manage to reproduce it reliably? If so, do you have a test case?

          kutzi added a comment -

          While it might make sense to make TestResult thread-safe, I'd also ask if it's actually needed that SurefireArchiver#result is serialized.
          Maybe that could be changed, too?

          kutzi added a comment - While it might make sense to make TestResult thread-safe, I'd also ask if it's actually needed that SurefireArchiver#result is serialized. Maybe that could be changed, too?

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/jenkins/1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
          Log:
          [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter

          Compare: https://github.com/jenkinsci/jenkins/compare/796918d...1476b2e

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4 Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Compare: https://github.com/jenkinsci/jenkins/compare/796918d...1476b2e

          kutzi added a comment -

          I've removed the serialization of 'result' and local tests indicate that this fixes the issue.
          As a positive side effect, Jenkins has to serialize less stuff on each remote call, which should reduce the Maven job overhead a bit

          kutzi added a comment - I've removed the serialization of 'result' and local tests indicate that this fixes the issue. As a positive side effect, Jenkins has to serialize less stuff on each remote call, which should reduce the Maven job overhead a bit

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/b835fb15497cf31a5069804a9920a747053be3c4
          Log:
          Changelog entry for JENKINS-11256

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html http://jenkins-ci.org/commit/jenkins/b835fb15497cf31a5069804a9920a747053be3c4 Log: Changelog entry for JENKINS-11256

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1303
          [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
          Changelog entry for JENKINS-11256

          Christoph Kutzinski : 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
          Files :

          • maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java

          Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #1303 [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Changelog entry for JENKINS-11256 Christoph Kutzinski : 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4 Files : maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4 Files : changelog.html

          swolk added a comment -

          When will this make it into an LTS build? I really need this fix. Thanks

          swolk added a comment - When will this make it into an LTS build? I really need this fix. Thanks

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/jenkins/46629db8d493f67c5b8ce1c14465b3c6a117e847
          Log:
          [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter(cherry picked from commit 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/jenkins/46629db8d493f67c5b8ce1c14465b3c6a117e847 Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter(cherry picked from commit 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4)

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/b39a310c5165234dab6d7acd3ed82bd399d255eb
          Log:
          Changelog entry for JENKINS-11256(cherry picked from commit b835fb15497cf31a5069804a9920a747053be3c4)

          Conflicts:

          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html http://jenkins-ci.org/commit/jenkins/b39a310c5165234dab6d7acd3ed82bd399d255eb Log: Changelog entry for JENKINS-11256 (cherry picked from commit b835fb15497cf31a5069804a9920a747053be3c4) Conflicts: changelog.html

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          src/main/java/hudson/maven/reporters/SurefireArchiver.java
          http://jenkins-ci.org/commit/maven-plugin/00f1af0b4e71f8c691e36a1bb741eeff7bd512ee
          Log:
          [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
          Originally-Committed-As: 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java http://jenkins-ci.org/commit/maven-plugin/00f1af0b4e71f8c691e36a1bb741eeff7bd512ee Log: [FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter Originally-Committed-As: 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4

            kutzi kutzi
            kisa kisa
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: