Major The workflow throws an NPE and hangs when it gets to the node block at the end of the workflow. The stack trace for the npe suggests is is related to copyLogs and SimpleXStreamFlowNodeStorage. The contents of my workflow are listed at the end.
The workflow has 3 main parts. The first part runs on a node which I'll call node A. Only 1 slave with 1 executor matches this node. The second part runs 55 sub-jobs in parallel across 18 executors on about a dozen slaves. The third part runs on node 'A' again and is intended to pull down the results from the parallel sub-jobs. The first and second parts of the workflow succeed. The workflow fails to start the 3rd part, as it throws a null pointer exception from SimpleXStreamFlowNodeStorage. I have a work around which involves using a separate job for part 3. However, I wanted to post this issue in case it provides details that help with the hardening of the workflow plugins.
One other bit of environment information. My master cannot directly see the slaves due to a firewall. The slaves are started via a script on the slave and they are able to see and connect to the master. Usage of cloudant does not appear to be related to the issue. Cloudant is used to store files for sharing because in the workflow archiving to the master was slow and had other issues as well.
log excerpt:
Done parsing cookbook-recipes-0054.txt Running: Allocate node : Body : End Running: Allocate node : End Running: Execute sub-workflows in parallel : Body : End ERROR: failed to start build java.lang.NullPointerException at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:48) at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.set(UnsafeQualifiedObjectFieldAccessorImpl.java:71) at java.lang.reflect.Field.set(Field.java:670) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadInner(SimpleXStreamFlowNodeStorage.java:218) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.loadOuter(SimpleXStreamFlowNodeStorage.java:205) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$PersistenceContext.access$100(SimpleXStreamFlowNodeStorage.java:177) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:83) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:624) at org.jenkinsci.plugins.workflow.job.WorkflowRun.copyLogs(WorkflowRun.java:233) at org.jenkinsci.plugins.workflow.job.WorkflowRun.waitForCompletion(WorkflowRun.java:217) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:178) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) at hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)
def recipeFileList def CATEGORY_ID = '' def branches = [:] def cloudantSnippets = new com.ibm.devops.CloudantSnippets() node('A') { checkout scm: [ $class : 'GitSCM', branches: [[$class: 'hudson.plugins.git.BranchSpec', name: '*/develop']], userRemoteConfigs : [[ url: 'https://***', credentialsId: '***' ] ] ] dir('resources') { echo ':::::::::::::::::::: divide list of recipes into files each containing 200 ::::::::::::::::::::' sh 'rm -f cookbook-recipes*.txt' sh 'cat all-cookbook-recipes-*.txt >cookbook-recipes.txt' sh 'awk \'NR%200==1{x=sprintf("cookbook-recipes-%04d.txt", ++i);}{print > x}\' cookbook-recipes.txt' sh 'ls cookbook-recipes-*.txt | sort >sorted-recipe-list.txt' recipeFileList = readFile('sorted-recipe-list.txt').tokenize() } sh 'rm -f workspace.b*' sh 'tar cjf workspace.bz2 config data deepqa.home logging.properties pom.xml resources settings.arti3.xml.template src test descriptors' CATEGORY_ID = cloudantSnippets.shareFile('workspace.bz2', VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) echo ':::::::::::::::::::: Use maven to run RecipeListParseTest in parallel ::::::::::::::::::::' int for (int i = 0; i < recipeFileList.size(); i++) { String recipeFileName = "${recipeFileList.get(i)}" branches["${recipeFileName}"] = { node('parsing') { sh 'rm -rf *' cloudantSnippets.getSharedFile('workspace.bz2', CATEGORY_ID, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) sh 'tar xjf workspace.bz2' echo ":::::::::::::::::::: ${recipeFileName} : generate settings.xml file with encrypted passwords to be used for accessing artifactory ::::::::::::::::::::" sh 'wget -N --quiet http://artifactory:8081/artifactory/plugins-release-local/com/ibm/devops/GenerateSecureSettings/1.0/GenerateSecureSettings-1.0-all.jar' def mvnHome = tool 'Local maven-3.2.3' env.PATH = "${mvnHome}/bin:${env.PATH}" env.SOME_PASSWORD = "$SOME_PASSWORD" sh "${tool 'local IBM Java 7'}/bin/java -jar GenerateSecureSettings-1.0-all.jar" env.SOME_PASSWORD = "not set" echo " :::::::::::::::::::: Running RecipeListParseTest with $recipeFileName ::::::::::::::::::::" sh "${tool 'Local maven-3.2.3'}/bin/mvn -s settings.xml -DrunSuite='**/*****.class' -DrecipeFile='${recipeFileName}' -DDB_SERVER=9.12.246.170 -Duima_datapath='resources' -DDB_LAZY=YES -DoutputDir='output/subset' -DJAVA_ARG_LINE='-Xmx5G -Xms1G -XX:MaxPermSize=256m -XX:HeapDumpPath=/dev/null' clean test -U | grep -v 'process.*start' | grep -v 'process.*end' | grep -v 'invoked.*returning' | tee ${recipeFileName}.mvn.log | grep '======' " echo "Upload JSON Output for ${recipeFileName}" sh "tar cjf target/classes/output/${recipeFileName}.json.bz2 target/classes/output/subset" sh "tar cjf ${recipeFileName}.mvn.log.bz2 ${recipeFileName}.mvn.log" env.CATEGORY_ID="${CATEGORY_ID}" cloudantSnippets.shareFile("target/classes/output/${recipeFileName}.json.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) echo "Upload ${recipeFileName}.mvn.log.bz2" cloudantSnippets.shareFile("${recipeFileName}.mvn.log.bz2", VCAP_SERVICE_TYPE, VCAP_SERVICE_NAME, VCAP_SERVICE_USERNAME,VCAP_SERVICE_PASSWORD) echo "Done parsing ${recipeFileName}" } } } } parallel branches node('A') { echo " :::::::::::::::::::: Retrieve all generated json into workspace ::::::::::::::::::::" for (int i = 0; i < recipeFileList.size(); i++) { String recipeFileName = "${recipeFileList.get(i)}" catchError { echo " :::::::::::::::::::: Retrieve json for $recipeFileName ::::::::::::::::::::" cloudantSnippets.getSharedFile("target/classes/output/${recipeFileName}.json.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD) cloudantSnippets.getSharedFile("${recipeFileName}.mvn.log.bz2", CATEGORY_ID, VCAP_SERVICE_USERNAME, VCAP_SERVICE_PASSWORD) sh "tar xjf target/classes/output/${recipeFileName}.json.bz2 " sh "tar xjf ${recipeFileName}.mvn.log.bz2 " } } }