Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11256

Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave

      linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave
      build fails with

      — maven-install-plugin:2.3.1:install (default-install) @ xxxxxxxx —
      Waiting for Jenkins to finish collecting data
      mavenExecutionResult exceptions not empty
      message : Internal error: java.util.ConcurrentModificationException
      cause : null
      Stack trace :
      org.apache.maven.InternalErrorException: Internal error: java.util.ConcurrentModificationException
      at org.apache.maven.lifecycle.internal.BuilderCommon.handleBuildError(BuilderCommon.java:128)[INFO] Installing xxxxxxxx.jar to xxxxxxxx

      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:95)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
      at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:164)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
      at java.util.concurrent.FutureTask.run(FutureTask.java:123)[INFO] Compiling 1 source file to xxxxxxxx

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.util.ConcurrentModificationException
      at java.util.ArrayList.writeObject(ArrayList.java:572)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.ArrayList.writeObject(ArrayList.java:569)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at java.util.HashMap.writeObject(HashMap.java:1039)
      at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:592)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      at hudson.remoting.Channel.callAsync(Channel.java:708)
      at hudson.maven.Maven3Builder$FilterImpl.executeAsync(Maven3Builder.java:229)
      at hudson.maven.reporters.MavenFingerprinter.postBuild(MavenFingerprinter.java:96)
      at hudson.maven.Maven3Builder$MavenExecutionListener.recordProjectSucceeded(Maven3Builder.java:431)
      at hudson.maven.Maven3Builder$MavenExecutionListener.projectSucceeded(Maven3Builder.java:405)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:74)
      at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:91)
      ... 7 more
      [INFO]
      [INFO] — maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ xxxxxxxx —
      channel stopped
      Archiviere Artefakte
      Finished: FAILURE

          [JENKINS-11256] Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins

          kisa created issue -

          kutzi added a comment -

          Does it happen frequently or only once?

          kutzi added a comment - Does it happen frequently or only once?

          kisa added a comment -

          Every 10-15 builds ... bad enough

          kisa added a comment - Every 10-15 builds ... bad enough
          kutzi made changes -
          Assignee New: kutzi [ kutzi ]

          All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places.

          Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation.

          I will create a pull request for this.

          Stephan Pauxberger added a comment - All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places. Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation. I will create a pull request for this.

          Stephan Pauxberger added a comment - - edited

          Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e, however it landed in the open pull request concerning Parent POM fingerprinting.

          Kutzi, could you look at both of them together or should I split it out again?

          Stephan Pauxberger added a comment - - edited Created https://github.com/pauxus/jenkins/commit/f63bdf359b348707e35d3333a2da22d23397530e , however it landed in the open pull request concerning Parent POM fingerprinting. Kutzi, could you look at both of them together or should I split it out again?

          kutzi added a comment - - edited

          Thanks for the pull request.
          Yes, please create a separate pull request for each issue.

          Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem?

          I think I'd rather put a synchronized block inside the BuildCallable in postBuild()

          kutzi added a comment - - edited Thanks for the pull request. Yes, please create a separate pull request for each issue. Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem? I think I'd rather put a synchronized block inside the BuildCallable in postBuild()

          Stephan Pauxberger added a comment - - edited

          Old problem with race conditions - if you need them, they don't show up.

          I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module.

          As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something).

          Which leads back to the question: where does ist happen.

          The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace)

          However, I did not found anyone modifying one of the valuelists.

          Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises.

          I will further look into this.

          UPDATE:

          Its an ArrayList in an ArrayList in the reporters map could be some plugin.

          Stephan Pauxberger added a comment - - edited Old problem with race conditions - if you need them, they don't show up. I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module. As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something). Which leads back to the question: where does ist happen. The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace) However, I did not found anyone modifying one of the valuelists. Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises. I will further look into this. UPDATE: Its an ArrayList in an ArrayList in the reporters map could be some plugin.

          kutzi added a comment - - edited

          According to Olivier this might be a bug in Maven 3.0.3:
          http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html

          kutzi added a comment - - edited According to Olivier this might be a bug in Maven 3.0.3: http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html
          kutzi made changes -
          Assignee Original: kutzi [ kutzi ] New: Stephan Pauxberger [ paux ]

            kutzi kutzi
            kisa kisa
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: