Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55146

PCT sometimes crashes on Java 11 due to module read errors when using the docker image

    XMLWordPrintable

Details

    Description

      https://issues.apache.org/jira/browse/SUREFIRE-1496

       

      Command: // custom image is used for error propagation

      docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out -e ARTIFACT_ID=ssh-slaves -e VERSION=ssh-slaves-1.24 -e JDK_VERSION=11 onenashev/pct
      

       

      Output:

       

      + echo 'ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed' ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed + read file + find /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream + echo 'Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream' + cat /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream # Created on 2018-12-12T11:08:55.538 Corrupted stdin stream in forked JVM 1. Stream 'Error occurred during initialization of boot layer'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Error occurred during initialization of boot layer'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.541 Corrupted stdin stream in forked JVM 1. Stream 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.542 Corrupted stdin stream in forked JVM 1. Stream 'Caused by: java.util.zip.ZipException: zip END header not found'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Caused by: java.util.zip.ZipException: zip END header not found'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) 

       

      Acceptance criteria

      • Figure out if it's failing under other conditions
      • PCT runs with Java 11 and docker image

      Attachments

        Issue Links

          Activity

            oleg_nenashev Oleg Nenashev created issue -
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Component/s plugin-pom [ 21982 ]
            oleg_nenashev Oleg Nenashev added a comment -

            According to https://github.com/junit-team/junit5/issues/855 , this is actually a valid issue in Maven Surefire Plugin which is fixed in 2.21: https://github.com/apache/maven-surefire/commit/7176d3c17b438c60e48a525ef96e6812f00bec46 . Please correct me if I am wrong tibor17

             

            We cannot easily update to 3.0.0-M1 (see https://github.com/jenkinsci/plugin-pom/pull/131 from jglick), but probably we could pick up the 2.21 patch in Plugin POM. CC batmat

            oleg_nenashev Oleg Nenashev added a comment - According to https://github.com/junit-team/junit5/issues/855  , this is actually a valid issue in Maven Surefire Plugin which is fixed in 2.21: https://github.com/apache/maven-surefire/commit/7176d3c17b438c60e48a525ef96e6812f00bec46  . Please correct me if I am wrong tibor17   We cannot easily update to 3.0.0-M1 (see https://github.com/jenkinsci/plugin-pom/pull/131  from jglick ), but probably we could pick up the 2.21 patch in Plugin POM. CC batmat
            oleg_nenashev Oleg Nenashev made changes -
            Labels java11 java11-devtools-compatibility
            oleg_nenashev Oleg Nenashev made changes -
            Summary PCT crashes on Java 11 in Docker due to SUREFIRE-1496 PCT crashes on Java 11 in Docker on Maven Surefire 2.20
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-55147 [ JENKINS-55147 ]
            tibor17 Tibor Digana added a comment -

            oleg_nenashev
            For the external users it must be very uncertain to read if we mix unrelated issues.
            Yes, SUREFIRE-855 was fixed in 2.21.0.
            The another issue https://issues.apache.org/jira/browse/SUREFIRE-1496 is about Jigsaw modularity, see my comment in SUREFIRE-1496.
            And third unrelated cause error is in your description in this issue:
            Caused by: java.util.zip.ZipException: zip END header not found
            The Surefire is not packaging classes in JAR file. It is the responsibility of maven-jar-plugin.
            Please see the release 3.1.1 of JAR Plugin we did yesterday, I guess.
            https://issues.apache.org/jira/browse/MJAR-241
            After seen deeper, the path is pointing to JDK, this means the file /pct/jdk11-libs/javax.activation.jar and I would say that Oracle or OpenJDK has to be asked.

            Check it out again with new version of JAR Plugin and possibly Surefire plugin 3.0.0-M2, but you may try to also switch to another JDK to come over the ZIP error.

            tibor17 Tibor Digana added a comment - oleg_nenashev For the external users it must be very uncertain to read if we mix unrelated issues. Yes, SUREFIRE-855 was fixed in 2.21.0 . The another issue https://issues.apache.org/jira/browse/SUREFIRE-1496 is about Jigsaw modularity, see my comment in SUREFIRE-1496. And third unrelated cause error is in your description in this issue: Caused by: java.util.zip.ZipException: zip END header not found The Surefire is not packaging classes in JAR file. It is the responsibility of maven-jar-plugin . Please see the release 3.1.1 of JAR Plugin we did yesterday, I guess. https://issues.apache.org/jira/browse/MJAR-241 After seen deeper, the path is pointing to JDK, this means the file /pct/jdk11-libs/javax.activation.jar and I would say that Oracle or OpenJDK has to be asked. Check it out again with new version of JAR Plugin and possibly Surefire plugin 3.0.0-M2, but you may try to also switch to another JDK to come over the ZIP error.
            oleg_nenashev Oleg Nenashev added a comment -

            tibor17 thanks for the clarification!

            oleg_nenashev Oleg Nenashev added a comment - tibor17 thanks for the clarification!
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Apparently I cannot reproduce the issue outside Docker, so probably it's another flavor of SUREFIRE-1588 fun

            oleg_nenashev Oleg Nenashev added a comment - - edited java.activation error is gone after updating to the recent release of javarta.aactivation: https://repo1.maven.org/maven2/com/sun/activation/jakarta.activation/1.2.1/  . This is good, but we will need to update it everywhere in Docker packaging I'd guess Maven JAR Plugin surely needs to be updated. https://github.com/jenkinsci/plugin-pom/blob/1f78cd35f318480942f82af836ed31ab98b27a87/pom.xml#L318  is where it is defined for Jenkins plugins. Likely won't help in the case of our tests taking issues referenced in https://github.com/jenkinsci/remoting/pull/315  and other core tests by jglick   Apparently I cannot reproduce the issue outside Docker, so probably it's another flavor of SUREFIRE-1588 fun
            oleg_nenashev Oleg Nenashev added a comment -

            Also CC batmat

            oleg_nenashev Oleg Nenashev added a comment - Also CC batmat
            oleg_nenashev Oleg Nenashev added a comment -

            https://github.com/jenkinsci/plugin-compat-tester/pull/97/commits/1299954b4f83f72409d5ad1a4db6efd01c2fbbef helps for the java.activation issue, but I really really doubt we want to update production packages for now. Will create follow-ups

            oleg_nenashev Oleg Nenashev added a comment - https://github.com/jenkinsci/plugin-compat-tester/pull/97/commits/1299954b4f83f72409d5ad1a4db6efd01c2fbbef  helps for the java.activation issue, but I really really doubt we want to update production packages for now. Will create follow-ups
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-55162 [ JENKINS-55162 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Link JENKINS-51805 [ 191255 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary PCT crashes on Java 11 in Docker on Maven Surefire 2.20 PCT sometimes crashes on Java 11 due to module read errors
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev added a comment -

            I had to revert the patch. It would be applicable until the Jakarta Team fixes https://github.com/eclipse-ee4j/jaf/issues/13 and https://github.com/eclipse-ee4j/jaxb-ri/issues/1236 

            Unassigning it for now so that others from Java 11 support team can pick it up

             

            oleg_nenashev Oleg Nenashev added a comment - I had to revert the patch. It would be applicable until the Jakarta Team fixes https://github.com/eclipse-ee4j/jaf/issues/13  and https://github.com/eclipse-ee4j/jaxb-ri/issues/1236   Unassigning it for now so that others from Java 11 support team can pick it up  
            oleg_nenashev Oleg Nenashev added a comment -

            Summarized the result in https://github.com/eclipse-ee4j/jaf/issues/13 . One of possible solutions would be to update the entire PCT to run with Java 11 (not tested, but it apparently does not work according to kearls3).

            oleg_nenashev Oleg Nenashev added a comment - Summarized the result in https://github.com/eclipse-ee4j/jaf/issues/13  . One of possible solutions would be to update the entire PCT to run with Java 11 (not tested, but it apparently does not work according to kearls3 ).
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-55296 [ JENKINS-55296 ]
            oleg_nenashev Oleg Nenashev added a comment -

            I updated PCT to support running the entire codebase with JDK 11 in JENKINS-55296 . It does not resolve the issue. So we will need something better to run PCT in Docker

            oleg_nenashev Oleg Nenashev added a comment - I updated PCT to support running the entire codebase with JDK 11 in  JENKINS-55296  . It does not resolve the issue. So we will need something better to run PCT in Docker
            batmat Baptiste Mathus made changes -
            Labels java11 java11-devtools-compatibility java11 java11-devtools-compatibility triaged
            alecharp Adrien Lecharpentier made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            batmat Baptiste Mathus made changes -
            Labels java11 java11-devtools-compatibility triaged java11 java11-devtools-compatibility scrub triaged
            batmat Baptiste Mathus added a comment - Looks like https://issues.apache.org/jira/browse/SUREFIRE-1601
            vilacides Isa Vilacides made changes -
            Summary PCT sometimes crashes on Java 11 due to module read errors PCT sometimes crashes on Java 11 due to module read errors when using the docker image
            vilacides Isa Vilacides made changes -
            Description [https://issues.apache.org/jira/browse/SUREFIRE-1496]

             

            Command: // custom image is used for error propagation
            {code:java}
            docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out -e ARTIFACT_ID=ssh-slaves -e VERSION=ssh-slaves-1.24 -e JDK_VERSION=11 onenashev/pct
            {code}
             

            Output:

             
            {code:java}
            + echo 'ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed' ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed + read file + find /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream + echo 'Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream' + cat /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream # Created on 2018-12-12T11:08:55.538 Corrupted stdin stream in forked JVM 1. Stream 'Error occurred during initialization of boot layer'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Error occurred during initialization of boot layer'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.541 Corrupted stdin stream in forked JVM 1. Stream 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.542 Corrupted stdin stream in forked JVM 1. Stream 'Caused by: java.util.zip.ZipException: zip END header not found'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Caused by: java.util.zip.ZipException: zip END header not found'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) {code}
            [https://issues.apache.org/jira/browse/SUREFIRE-1496]

             

            Command: // custom image is used for error propagation
            {code:java}
            docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out -e ARTIFACT_ID=ssh-slaves -e VERSION=ssh-slaves-1.24 -e JDK_VERSION=11 onenashev/pct
            {code}
             

            Output:

             
            {code:java}
            + echo 'ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed' ERROR: PCT failed with code 1. Will check for Maven Surefire dumps if it crashed + read file + find /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream + echo 'Found Maven Surefire dump file: /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream' + cat /pct/tmp/work/ssh-slaves/target/surefire-reports/2018-12-12T11-08-54_738-jvmRun1.dumpstream # Created on 2018-12-12T11:08:55.538 Corrupted stdin stream in forked JVM 1. Stream 'Error occurred during initialization of boot layer'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Error occurred during initialization of boot layer'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.541 Corrupted stdin stream in forked JVM 1. Stream 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.module.FindException: Error reading /pct/jdk11-libs/javax.activation.jar'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) # Created on 2018-12-12T11:08:55.542 Corrupted stdin stream in forked JVM 1. Stream 'Caused by: java.util.zip.ZipException: zip END header not found'. java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Caused by: java.util.zip.ZipException: zip END header not found'. at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:469) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:191) at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:158) at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:87) at java.lang.Thread.run(Thread.java:748) {code}
             

            *Acceptance criteria*
             * Figure out if it's failing under other conditions
             * PCT runs with Java 11 and docker image
            vilacides Isa Vilacides made changes -
            Labels java11 java11-devtools-compatibility scrub triaged java11 java11-devtools-compatibility triaged
            tibor17 Tibor Digana added a comment - - edited

            We will use tcp/ip sockets in Surefire/Failsafe (3.0.0-M4 or M5) instead of interprocess pipes which is nowadays sollution for interprocess communication.

            tibor17 Tibor Digana added a comment - - edited We will use tcp/ip sockets in Surefire/Failsafe (3.0.0-M4 or M5) instead of interprocess pipes which is nowadays sollution for interprocess communication.
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Adrien Lecharpentier [ alecharp ]
            oleg_nenashev Oleg Nenashev added a comment -

            It was actually caused by a glitch in the Docker packaging. And it was fixed by alecharp in https://github.com/jenkinsci/plugin-compat-tester/pull/111 . I still confirm there is similar behavior outside Docker rarely, but I believe it is related to another issue

            oleg_nenashev Oleg Nenashev added a comment - It was actually caused by a glitch in the Docker packaging. And it was fixed by alecharp in https://github.com/jenkinsci/plugin-compat-tester/pull/111  . I still confirm there is similar behavior outside Docker rarely, but I believe it is related to another issue
            oleg_nenashev Oleg Nenashev made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            tibor17 Tibor Digana added a comment - - edited

            oleg_nenashev

            Regarding the Alpine/BusyBox we made three fixes

            https://issues.apache.org/jira/browse/SUREFIRE-1702

            https://issues.apache.org/jira/browse/SUREFIRE-1703

            https://issues.apache.org/jira/browse/SUREFIRE-1704

            and we are finishing the last few issues to cut a new release version 3.0.0-M4.

            tibor17 Tibor Digana added a comment - - edited oleg_nenashev Regarding the Alpine/BusyBox we made three fixes https://issues.apache.org/jira/browse/SUREFIRE-1702 https://issues.apache.org/jira/browse/SUREFIRE-1703 https://issues.apache.org/jira/browse/SUREFIRE-1704 and we are finishing the last few issues to cut a new release version 3.0.0-M4.
            basil Basil Crow made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

              alecharp Adrien Lecharpentier
              oleg_nenashev Oleg Nenashev
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: