-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
windows 10.
the following pipeline hung indefinable and could not be aborted correctly on windows (needed a hard kill)
windows master (no executors), one windows agents launched using a command on the master (java -jar ..../slave.jar)
node {
withMaven(maven: 'default') {
git changelog: false, poll: false, url: 'https://github.com/jtnord/maven-test-project.git'
bat "mvn package"
writeFile file: '.archive-jenkins-maven-event-spy-logs', text: ''
}
}
thread dump from pipeline is
Thread #4 at WorkflowScript.run(WorkflowScript:7) at DSL.withMaven(Native Method) at WorkflowScript.run(WorkflowScript:2) at DSL.node(running on agent) at WorkflowScript.run(WorkflowScript:1)
[JENKINS-43654] adding a simple writeFile inside withMaven causes pipeline to hang
We have similar hangs on a CentOS box when using the
writeFile file: '.archive-jenkins-maven-event-spy-logs', text: ''
This only happens on mvn package and install.
mvn test works ok.
If the created file does not start with a dot, both package and install goals run ok.
writeFile file: 'archive-jenkins-maven-event-spy-logs', text: ''
Not sure if related with JENKINS-32986 and --JENKINS-31484
I'm trying to identify the code that triggers the issue on the prod instance.
I can also replicate the issue on my personal jenkins test instance with:
node('mydocker') { //CentOS build node stage('build') { sh 'echo $PATH' sh 'which nohup' sh 'which sh' docker.image('maven:3.5.0-jdk-8').inside { git 'https://github.com/jtnord/maven-test-project.git' //git 'https://github.com/spring-projects/spring-petclinic.git' sh 'echo $PATH' sh 'which nohup' sh 'which sh' withEnv(['PATH=/usr/local/bin:/usr/bin:/bin']) { //workwaround for CentOS build node regarding nohup not finding sh withMaven() { sh 'mvn clean package' //writeFile file: 'archive-jenkins-maven-event-spy-logs', text: '' writeFile file: '.archive-jenkins-maven-event-spy-logs', text: '' } } } } }
If I use the project from git repo https://github.com/spring-projects/spring-petclinic.git, the build finishes ok.
Nuno Costa
added a comment - - edited We have similar hangs on a CentOS box when using the
writeFile file: '.archive-jenkins-maven-event-spy-logs', text: ''
This only happens on mvn package and install.
mvn test works ok.
If the created file does not start with a dot, both package and install goals run ok.
writeFile file: 'archive-jenkins-maven-event-spy-logs', text: ''
Not sure if related with JENKINS-32986 and -- JENKINS-31484 --.
I'm trying to identify the code that triggers the issue on the prod instance.
I can also replicate the issue on my personal jenkins test instance with:
node( 'mydocker' ) { //CentOS build node
stage( 'build' ) {
sh 'echo $PATH'
sh 'which nohup'
sh 'which sh'
docker.image( 'maven:3.5.0-jdk-8' ).inside {
git 'https: //github.com/jtnord/maven-test-project.git'
//git 'https://github.com/spring-projects/spring-petclinic.git'
sh 'echo $PATH'
sh 'which nohup'
sh 'which sh'
withEnv([ 'PATH=/usr/local/bin:/usr/bin:/bin' ]) { //workwaround for CentOS build node regarding nohup not finding sh
withMaven() {
sh 'mvn clean package '
//writeFile file: 'archive-jenkins-maven-event-spy-logs' , text: ''
writeFile file: '.archive-jenkins-maven-event-spy-logs' , text: ''
}
}
}
}
}
If I use the project from git repo https://github.com/spring-projects/spring-petclinic.git , the build finishes ok.
Nuno Costa
added a comment - After some tests using https://github.com/jtnord/maven-test-project.git as base, this were my findings:
- The hang is not triggered by the writeFile itself because the file is created correctly (issuing sh 'ls -la' after writeFile shows that the file exists)
- The hang takes exactly 5 minutes (timestamp {} can be used to validate) and happens when the main pom.xml points to modules/other pom.xml files
- Their is no time difference between calling only one or all modules.
- The pom files/module folders can be checkout to workspace because the hang only happens if the modules are called on the main pom.xml.
- When comment all module block in the main pom.xml, the build finishes sucessfully (without the 5m hang) and maven-spy file is gathered.
- If module block is kept, the build finishes sucessfully after the 5m hang but maven-spy file is not archived.
- Artifacts (pom and jar) are archived sucessfully unless told not to (.skip-archive-generated-artifacts)
Nuno Costa
added a comment - After some tests using https://github.com/jtnord/maven-test-project.git as base, this were my findings:
The hang is not triggered by the writeFile itself because the file is created correctly (issuing sh 'ls -la' after writeFile shows that the file exists)
The hang takes exactly 5 minutes (timestamp {} can be used to validate) and happens when the main pom.xml points to modules/other pom.xml files
Their is no time difference between calling only one or all modules.
The pom files/module folders can be checkout to workspace because the hang only happens if the modules are called on the main pom.xml.
When comment all module block in the main pom.xml, the build finishes sucessfully (without the 5m hang) and maven-spy file is gathered.
If module block is kept, the build finishes sucessfully after the 5m hang but maven-spy file is not archived.
Artifacts (pom and jar) are archived sucessfully unless told not to (.skip-archive-generated-artifacts)
Thanks ncosta, I'll try to work on it ASAP. Could you attach the result of "/threadDump" including the dump on the agent itself?
cleclerc , threadDump during the hang:
Thread #12 at WorkflowScript.run(WorkflowScript:26) at DSL.withMaven(Native Method) at WorkflowScript.run(WorkflowScript:16) at DSL.withEnv(Native Method) at WorkflowScript.run(WorkflowScript:15) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:129) at DSL.withDockerContainer(Native Method) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:128) at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:63) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:116) at WorkflowScript.run(WorkflowScript:8) at DSL.stage(Native Method) at WorkflowScript.run(WorkflowScript:4) at DSL.node(running on centos-buildserver-03) at WorkflowScript.run(WorkflowScript:2) at DSL.timestamps(Native Method) at WorkflowScript.run(WorkflowScript:1)
Added tail of console output, jenkins and node threadDump in attach.
Not sure if this is the correct data you asked but let me know if other info is needed.
This is from my test jenkins instance on virtualbox VMs.
I see similar hangs on our Prod instance but I did not find any modules being called. Only pom type dependencies.
Nuno Costa
added a comment - - edited cleclerc , threadDump during the hang:
Thread #12
at WorkflowScript.run(WorkflowScript:26)
at DSL.withMaven(Native Method)
at WorkflowScript.run(WorkflowScript:16)
at DSL.withEnv(Native Method)
at WorkflowScript.run(WorkflowScript:15)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/ var /lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:129)
at DSL.withDockerContainer(Native Method)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/ var /lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:128)
at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/ var /lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:63)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/ var /lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:116)
at WorkflowScript.run(WorkflowScript:8)
at DSL.stage(Native Method)
at WorkflowScript.run(WorkflowScript:4)
at DSL.node(running on centos-buildserver-03)
at WorkflowScript.run(WorkflowScript:2)
at DSL.timestamps(Native Method)
at WorkflowScript.run(WorkflowScript:1)
Added tail of console output, jenkins and node threadDump in attach.
Not sure if this is the correct data you asked but let me know if other info is needed.
This is from my test jenkins instance on virtualbox VMs.
I see similar hangs on our Prod instance but I did not find any modules being called. Only pom type dependencies.
Nuno Costa
added a comment - Still an issue with Jenkins 2.108 and fully updated plugins as of today.
Nuno Costa
added a comment - Still an issue with Jenkins 2.108 and fully updated plugins as of today. 
I can't reproduce and several users seem to build on Windows build agents. Could it be caused by something else than the pipeline-maven-plugin?