Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55356

Some workflow jobs fail after restart on Java 11 server

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • workflow-job-plugin
    • Jenkins JDK 11 docker image and current plugins
      Multibranch Pipeline builds for git plugin, git client plugin, and platformlabeler plugin

    Description

      While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a

      jenkins-url/safeRestart
      

      will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

      Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

      07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
      Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
      Ready to run at Fri Dec 28 07:48:49 MST 2018
      07:48:49 Timeout set to expire in 17 min
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      07:48:50 Failed in branch windows-8-2.150.1
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
      

      and this (seems to fail on Windows and on Linux):

      08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
      Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
      ???
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
      Ready to run at Fri Dec 28 08:52:07 MST 2018
      08:52:07 Timeout set to expire in 54 min
      08:52:07 Timeout set to expire in 54 min
      08:52:07 Timeout set to expire in 54 min
      08:52:07 Timeout set to expire in 54 min
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] // timeout
      [Pipeline] }
      08:52:08 Failed in branch windows-8
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      08:52:08 Failed in branch windows-8-2.150.1
      08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
      [Pipeline] // node
      [Pipeline] }
      08:57:14 Failed in branch linux-8
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      08:57:14 Failed in branch linux-8-2.150.1
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
      

      and this (windows and linux):

      09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
      Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
      Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
      Ready to run at Fri Dec 28 09:13:51 MST 2018
      09:13:51 Timeout set to expire in 55 min
      09:13:51 Timeout set to expire in 55 min
      09:13:51 Timeout set to expire in 55 min
      09:13:51 Timeout set to expire in 55 min
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] // withEnv
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] // stage
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] // node
      [Pipeline] // timeout
      [Pipeline] }
      09:13:52 Failed in branch windows-8-2.150.1
      [Pipeline] }
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] }
      09:13:52 Failed in branch linux-8
      [Pipeline] }
      09:13:52 Failed in branch windows-8
      09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      09:18:58 Failed in branch linux-8-2.150.1
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
      

      The process never started message and the missing workspace message are visible in both the failed git client plugin builds and in the failed git plugin builds.

      The problem does not seem to repeat on a Java 8 environment, just on a Java 11 environment.
      The problem does seem to repeat less frequently on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure multiple times as well as the smaller computer. That failure was during a restart while the agents and the server were very busy.

      The Docker image includes all the plugins that were used in the failure case. I've dupilicated the failures on at least two different machines.

      Attachments

        Activity

          markewaite Mark Waite created issue -
          markewaite Mark Waite made changes -
          Field Original Value New Value
          Labels java11-compatibility
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          I don't know if the problem can be duplicated on a Java 8 environment. Will try that soon
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does not seem to repeat on a Java 8 environment, just on a Java 11 environment. More experiments to come.

          markewaite could you please also check the Jenkins logs to see if there's more there? Specifically I'm curious if this could actually be some kind of duplicate of JENKINS-55174.

          Also: could you test again with the fix provided (not yet merged) by dnusbaum in https://github.com/jenkinsci/workflow-support-plugin/pull/86?
          You can the Incremental release of this PR available at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/3.0-java11-alpha-2-rc689.90369f68d9e9/workflow-support-3.0-java11-alpha-2-rc689.90369f68d9e9.hpi

          Thank you!

          batmat Baptiste Mathus added a comment - markewaite could you please also check the Jenkins logs to see if there's more there? Specifically I'm curious if this could actually be some kind of duplicate of JENKINS-55174 . Also: could you test again with the fix provided (not yet merged) by dnusbaum in  https://github.com/jenkinsci/workflow-support-plugin/pull/86? You can the Incremental release of this PR available at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/3.0-java11-alpha-2-rc689.90369f68d9e9/workflow-support-3.0-java11-alpha-2-rc689.90369f68d9e9.hpi Thank you!
          markewaite Mark Waite added a comment -

          batmat thanks for asking!

          The details in this report are using the incremental build from the pull request that dnusbaum had provided. The results with that pull request are much better than without it. Without that pull request I see null pointer exception messages in the Jenkins logs on restart. The null pointer exceptions cause the jobs to fail and are described in JENKINS-55338. JENKINS-55338 is a duplicate of JENKINS-55174.

          There are no additional entries in the Jenkins log related to the problem as far as I can tell.

          I think PR-86 should be merged into the workflow support plugin for Java 11. It improves the results by removing the null pointer exception and moves the code onto a newer JBoss marshalling release.

          markewaite Mark Waite added a comment - batmat thanks for asking! The details in this report are using the incremental build from the pull request that dnusbaum had provided. The results with that pull request are much better than without it. Without that pull request I see null pointer exception messages in the Jenkins logs on restart. The null pointer exceptions cause the jobs to fail and are described in JENKINS-55338 . JENKINS-55338 is a duplicate of JENKINS-55174 . There are no additional entries in the Jenkins log related to the problem as far as I can tell. I think PR-86 should be merged into the workflow support plugin for Java 11. It improves the results by removing the null pointer exception and moves the code onto a newer JBoss marshalling release.
          dnusbaum Devin Nusbaum added a comment -

          It looks like the root failure in all cases is related to the "ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh" error message. It think that error message comes from workflow-durable-task-step, and happens if the workspace for a job is not a directory. Maybe this is a bug with the workspace-related changes in recent versions of branch-api?

          markewaite Can you add the full list of plugins installed in Jenkins and their versions to the description? It would also be great to see the Pipeline jobs that these log snippets come from.

          dnusbaum Devin Nusbaum added a comment - It looks like the root failure in all cases is related to the "ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh" error message. It think that error message comes from workflow-durable-task-step , and happens if the workspace for a job is not a directory. Maybe this is a bug with the workspace-related changes in recent versions of branch-api? markewaite Can you add the full list of plugins installed in Jenkins and their versions to the description? It would also be great to see the Pipeline jobs that these log snippets come from.
          batmat Baptiste Mathus added a comment - - edited

          vivek can you please plan analyzing this issue? Thanks!

          batmat Baptiste Mathus added a comment - - edited vivek can you please plan analyzing this issue? Thanks!
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does not seem to repeat on a Java 8 environment, just on a Java 11 environment. More experiments to come.
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *not* seem to repeat on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the passing computer has 32 GB RAM and a newer Intel i5 processor.

          More research. The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          markewaite Mark Waite added a comment -

          dnusbaum the link to the list of plugins is a list of plugin files committed to the git repository at the time I ran the test. The Pipeline job definitions are

          Both those Jenkinsfile examples are buildPlugin with a few options. The second example includes a long definition of some PCT / ATH code that is not executed.

          markewaite Mark Waite added a comment - dnusbaum the link to the list of plugins is a list of plugin files committed to the git repository at the time I ran the test. The Pipeline job definitions are Git client plugin Git plugin Both those Jenkinsfile examples are buildPlugin with a few options. The second example includes a long definition of some PCT / ATH code that is not executed.
          vivek Vivek Pandey added a comment -

          batmat yes, dnusbaum is already looking in to it.

          vivek Vivek Pandey added a comment - batmat yes, dnusbaum is already looking in to it.
          batmat Baptiste Mathus made changes -
          Assignee Devin Nusbaum [ dnusbaum ]
          batmat Baptiste Mathus made changes -
          Status Open [ 1 ] In Progress [ 3 ]

          Great thanks!

          batmat Baptiste Mathus added a comment - Great thanks!
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *not* seem to repeat on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the passing computer has 32 GB RAM and a newer Intel i5 processor.

          More research. The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *seem to repeat less frequently on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure only once. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          markewaite Mark Waite added a comment - - edited

          Failures seem to be more frequent when the master is heavily loaded. I've seen multiple failures on my 32 GB machine now. Will continue exploring to see if I can identify common failure patterns.

          When the 32 GB i5 machine failed, the message in the failing build log was similar to other Windows "workspace not found" messages:

          14:31:36 [INFO] ------------------------------------------------------------------------
          Resuming build at Wed Jan 02 14:33:19 MST 2019 after Jenkins restart
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ???
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ???
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ???
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ???
          Ready to run at Wed Jan 02 14:33:53 MST 2019
          14:33:53 Timeout set to expire in 45 min
          14:33:53 Timeout set to expire in 45 min
          14:33:53 Timeout set to expire in 45 min
          14:33:53 Timeout set to expire in 45 min
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // withEnv
          [Pipeline] // stage
          [Pipeline] stage
          [Pipeline] { (Archive (linux-8-2.150.1))
          [Pipeline] }
          [Pipeline] }
          [Pipeline] junit
          [Pipeline] // node
          14:33:54 Recording test results
          [Pipeline] // stage
          [Pipeline] }
          14:33:54 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          14:33:55 Failed in branch windows-8-2.150.1
          14:33:55 No test report files were found. Configuration error?
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          14:33:55 Failed in branch linux-8-2.150.1
          Resuming build at Wed Jan 02 14:44:08 MST 2019 after Jenkins restart
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ???
          Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: Waiting for next available executor on ‘mark-pc4-ssh’
          Ready to run at Wed Jan 02 14:44:29 MST 2019
          14:44:29 Timeout set to expire in 35 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          14:44:29 Failed in branch windows-8
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          
          GitHub has been notified of this commit’s build result
          
          Command Close created at
          	at hudson.remoting.Command.<init>(Command.java:68)
          	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1267)
          	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265)
          	at hudson.remoting.Channel.close(Channel.java:1438)
          	at hudson.remoting.Channel.close(Channel.java:1405)
          	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1272)
          Caused: hudson.remoting.Channel$OrderlyShutdown
          Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to aws-ubuntu-18-a
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          		at hudson.remoting.Request.call(Request.java:202)
          		at hudson.remoting.Channel.call(Channel.java:956)
          		at hudson.FilePath.act(FilePath.java:1072)
          		at hudson.FilePath.act(FilePath.java:1061)
          		at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:144)
          		at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
          		at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
          		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
          		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
          		at hudson.security.ACL.impersonate(ACL.java:290)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
          		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          		at java.base/java.lang.Thread.run(Thread.java:834)
          Also:   hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc3-ssh
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321)
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443)
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426)
          		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          		at java.base/java.lang.Thread.run(Thread.java:834)
          Also:   	Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to testing-a-mwaite
          			at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          			at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
          			at hudson.remoting.Channel.call(Channel.java:957)
          			at hudson.FilePath.act(FilePath.java:1072)
          			at hudson.FilePath.act(FilePath.java:1061)
          			at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
          			at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
          			at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
          			at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:50)
          			at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:23)
          			at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
          			at hudson.security.ACL.impersonate(ACL.java:290)
          			at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
          			at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          hudson.AbortException: No test report files were found. Configuration error?
          		at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:154)
          		at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
          		at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
          		at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          		at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          		at hudson.remoting.Request$2.run(Request.java:369)
          		at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          		at java.base/java.lang.Thread.run(Thread.java:834)
          Also:   hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc4-ssh
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321)
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443)
          		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426)
          		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          		at java.base/java.lang.Thread.run(Thread.java:834)
          Caused: hudson.remoting.RequestAbortedException
          	at hudson.remoting.Request.abort(Request.java:340)
          	at hudson.remoting.Channel.terminate(Channel.java:1040)
          	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1273)
          	at hudson.remoting.Channel$1.handle(Channel.java:565)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85)
          Finished: FAILURE
          
          markewaite Mark Waite added a comment - - edited Failures seem to be more frequent when the master is heavily loaded. I've seen multiple failures on my 32 GB machine now. Will continue exploring to see if I can identify common failure patterns. When the 32 GB i5 machine failed, the message in the failing build log was similar to other Windows "workspace not found" messages: 14:31:36 [INFO] ------------------------------------------------------------------------ Resuming build at Wed Jan 02 14:33:19 MST 2019 after Jenkins restart Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Ready to run at Wed Jan 02 14:33:53 MST 2019 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min 14:33:53 Timeout set to expire in 45 min [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // timeout [Pipeline] // withEnv [Pipeline] // stage [Pipeline] stage [Pipeline] { (Archive (linux-8-2.150.1)) [Pipeline] } [Pipeline] } [Pipeline] junit [Pipeline] // node 14:33:54 Recording test results [Pipeline] // stage [Pipeline] } 14:33:54 Failed in branch linux-8 [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:33:55 Failed in branch windows-8-2.150.1 14:33:55 No test report files were found. Configuration error? [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:33:55 Failed in branch linux-8-2.150.1 Resuming build at Wed Jan 02 14:44:08 MST 2019 after Jenkins restart Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: ??? Waiting to resume part of Git Plugin Folder » Git Branches - Jenkinsfile (GitHub) » add-tag-action-test #10: Waiting for next available executor on ‘mark-pc4-ssh’ Ready to run at Wed Jan 02 14:44:29 MST 2019 14:44:29 Timeout set to expire in 35 min [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } 14:44:29 Failed in branch windows-8 [Pipeline] // parallel [Pipeline] } [Pipeline] // timestamps [Pipeline] End of Pipeline GitHub has been notified of this commit’s build result Command Close created at at hudson.remoting.Command.<init>(Command.java:68) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1267) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265) at hudson.remoting.Channel.close(Channel.java:1438) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1272) Caused: hudson.remoting.Channel$OrderlyShutdown Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to aws-ubuntu-18-a at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.Channel.call(Channel.java:956) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:144) at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc3-ssh at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to testing-a-mwaite at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:50) at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:23) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) hudson.AbortException: No test report files were found. Configuration error? at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:154) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Also: hudson.AbortException: missing workspace C:\J\T\workspace\eline-github_add-tag-action-test on mark-pc4-ssh at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:321) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:443) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:426) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1040) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1273) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85) Finished: FAILURE
          vilacides Isa Vilacides made changes -
          Labels java11-compatibility java11-compatibility triaged
          markewaite Mark Waite added a comment -

          It seems that workflow-support 3.0 and 3.1 did not address this. I still see the following report in the Windows agent on my Java 11 server when I safeRestart the Jenkins server during Pipeline execution:

          [INFO] -------------------------------------------------------
          [INFO] Running InjectedTest
          [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.181 s - in InjectedTest
          [INFO] Running hudson.plugins.git.BranchTest
          [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.374 s - in hudson.plugins.git.BranchTest
          [INFO] Running hudson.plugins.git.GitAPIBadInitTest
          [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in hudson.plugins.git.GitAPIBadInitTest
          [INFO] Running hudson.plugins.git.GitExceptionTest
          [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.534 s - in hudson.plugins.git.GitExceptionTest
          [INFO] Running hudson.plugins.git.GitLockFailedExceptionTest
          [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in hudson.plugins.git.GitLockFailedExceptionTest
          [INFO] Running hudson.plugins.git.GitObjectTest
          [INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 s - in hudson.plugins.git.GitObjectTest
          [INFO] Running hudson.plugins.git.GitToolResolverTest
          [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.028 s - in hudson.plugins.git.GitToolResolverTest
          [INFO] Running hudson.plugins.git.GitToolTest
          [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.005 s - in hudson.plugins.git.GitToolTest
          [INFO] Running hudson.plugins.git.IndexEntryTest
          [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in hudson.plugins.git.IndexEntryTest
          [INFO] Running hudson.plugins.git.RevisionTest
          [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in hudson.plugins.git.RevisionTest
          [INFO] Running hudson.plugins.git.TagTest
          [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in hudson.plugins.git.TagTest
          [INFO] Running org.jenkinsci.plugins.gitclient.CliGitAPIImplTest
          missing workspace C:\J\T\workspace\it-client-pipeline-github_master on mark-pc3-ssh
          
          markewaite Mark Waite added a comment - It seems that workflow-support 3.0 and 3.1 did not address this. I still see the following report in the Windows agent on my Java 11 server when I safeRestart the Jenkins server during Pipeline execution: [INFO] ------------------------------------------------------- [INFO] Running InjectedTest [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.181 s - in InjectedTest [INFO] Running hudson.plugins.git.BranchTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.374 s - in hudson.plugins.git.BranchTest [INFO] Running hudson.plugins.git.GitAPIBadInitTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in hudson.plugins.git.GitAPIBadInitTest [INFO] Running hudson.plugins.git.GitExceptionTest [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.534 s - in hudson.plugins.git.GitExceptionTest [INFO] Running hudson.plugins.git.GitLockFailedExceptionTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in hudson.plugins.git.GitLockFailedExceptionTest [INFO] Running hudson.plugins.git.GitObjectTest [INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.297 s - in hudson.plugins.git.GitObjectTest [INFO] Running hudson.plugins.git.GitToolResolverTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.028 s - in hudson.plugins.git.GitToolResolverTest [INFO] Running hudson.plugins.git.GitToolTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.005 s - in hudson.plugins.git.GitToolTest [INFO] Running hudson.plugins.git.IndexEntryTest [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in hudson.plugins.git.IndexEntryTest [INFO] Running hudson.plugins.git.RevisionTest [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in hudson.plugins.git.RevisionTest [INFO] Running hudson.plugins.git.TagTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in hudson.plugins.git.TagTest [INFO] Running org.jenkinsci.plugins.gitclient.CliGitAPIImplTest missing workspace C:\J\T\workspace\it-client-pipeline-github_master on mark-pc3-ssh
          dnusbaum Devin Nusbaum added a comment -

          Yes, there are no changes in any recent Pipeline plugin release I am aware of that would explain or fix this issue. I investigated a few weeks ago but was not able to create a reproduction case. The error message comes from workflow-durable-task-step. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes).

          dnusbaum Devin Nusbaum added a comment - Yes, there are no changes in any recent Pipeline plugin release I am aware of that would explain or fix this issue. I investigated a few weeks ago but was not able to create a reproduction case. The error message comes from workflow-durable-task-step . It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes).
          markewaite Mark Waite added a comment -

          That directory exists and looks like a typical workspace.

          it does contain a directory named .@tmp which was a surprise for me. That directory contains 3 files that with names that start with 'jenkins-gitclient-permission'. I assume some form of temporary file, but don't recognize them as anything specific.

          markewaite Mark Waite added a comment - That directory exists and looks like a typical workspace. it does contain a directory named .@tmp which was a surprise for me. That directory contains 3 files that with names that start with 'jenkins-gitclient-permission'. I assume some form of temporary file, but don't recognize them as anything specific.
          batmat Baptiste Mathus made changes -
          Labels java11-compatibility triaged java11-compatibility scrub triaged
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be windows specific):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *seem to repeat less frequently on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure only once. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *seem to repeat less frequently on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure only once. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does *seem to repeat less frequently on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure only once. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does seem to repeat *less frequently* on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure multiple times as well as the smaller computer. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          markewaite Mark Waite made changes -
          Description While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does seem to repeat *less frequently* on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure multiple times as well as the smaller computer. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. However, I haven't yet been able to duplicate the problem on any other machine. More investigation soon.
          While running Java 11 based Jenkins in a docker container using a pre-release of the workflow support plugin which includes the fix for the null pointer exception, a
          {code}
          jenkins-url/safeRestart
          {code}
           will cause several of the Pipeline jobs that were running to fail when Jenkins tries to resume the jobs.

          Build log output from the failed builds has included messages like (seems to be more common in windows, but visible in Linux agents as well):

          {noformat}
          07:40:50 [INFO] Running org.jenkinsci.plugins.gitclient.PushTest
          Resuming build at Fri Dec 28 07:48:06 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #6: Waiting for next available executor on ‘coleen-pc2-ssh’
          Ready to run at Fri Dec 28 07:48:49 MST 2018
          07:48:49 Timeout set to expire in 17 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          07:48:50 Failed in branch windows-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (seems to fail on Windows and on Linux):

          {noformat}
          08:48:33 [INFO] --------------------------------[ hpi ]---------------------------------
          Resuming build at Fri Dec 28 08:51:46 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #7: ???
          Ready to run at Fri Dec 28 08:52:07 MST 2018
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          08:52:07 Timeout set to expire in 54 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          08:52:08 Failed in branch windows-8
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:52:08 Failed in branch windows-8-2.150.1
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-3af9d572
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          08:57:14 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/der_git-client-pipeline_beta-3.0@tmp/durable-1ba13955
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          08:57:14 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on coleen-pc2-ssh
          {noformat}

          and this (windows and linux):

          {noformat}
          09:12:13 [INFO] --- maven-help-plugin:3.1.1:evaluate (default-cli) @ git-client ---
          Resuming build at Fri Dec 28 09:13:35 MST 2018 after Jenkins restart
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Waiting to resume part of Git Client Plugin Folder » Git Client Branches - Jenkinsfile » beta-3.0 #8: ???
          Ready to run at Fri Dec 28 09:13:51 MST 2018
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          09:13:51 Timeout set to expire in 55 min
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] // withEnv
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // stage
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] // node
          [Pipeline] // timeout
          [Pipeline] }
          09:13:52 Failed in branch windows-8-2.150.1
          [Pipeline] }
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] // node
          [Pipeline] }
          09:13:52 Failed in branch linux-8
          [Pipeline] }
          09:13:52 Failed in branch windows-8
          09:18:58 process apparently never started in /home/mwaite/testing-a.markwaite.net-agent/workspace/r_git-client-pipeline_beta-3.0_2@tmp/durable-9d0a69fe
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // timeout
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] }
          09:18:58 Failed in branch linux-8-2.150.1
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] End of Pipeline
          ERROR: missing workspace C:\J\S\workspace\der_git-client-pipeline_beta-3.0 on mark-pc4-ssh
          {noformat}

          The {{process never started}} message and the {{missing workspace}} message are visible in both the failed git client plugin builds and in the failed git plugin builds.

          The problem does *not* seem to repeat on a Java 8 environment, just on a Java 11 environment.
          The problem does seem to repeat *less frequently* on a Java 11 environment running on a larger computer. The failing computer has 8 GB RAM with an older Intel i5 processor, while the less frequently failing computer has 32 GB RAM and a newer Intel i5 processor. The 32 GB machine has shown the failure multiple times as well as the smaller computer. That failure was during a restart while the agents and the server were very busy.

          The [Docker image|https://github.com/MarkEWaite/docker-lfs/tree/30517c315d6dc052e3f88a749834891bdc7c5725/ref/plugins] includes all the plugins that were used in the failure case. I've dupilicated the failures on at least two different machines.

          > repeat less frequently on a Java 11 environment running on a larger computer

          Triggered a thought markewaite: would you be able to run that test inside a Docker container with memory constraints? I can help with that if that's something you've never played with before. But basically passing like -m 1G will limit the memory available inside the container to 1GB of RAM, which is possibly going to trigger that behavior more immediately, if that is actually related to some memory settings .

          Cheers!

          batmat Baptiste Mathus added a comment - > repeat less frequently on a Java 11 environment running on a larger computer Triggered a thought markewaite : would you be able to run that test inside a Docker container with memory constraints? I can help with that if that's something you've never played with before. But basically passing like -m 1G will limit the memory available inside the container to 1GB of RAM, which is possibly going to trigger that behavior more immediately, if that is actually related to some memory settings . Cheers!
          dnusbaum Devin Nusbaum added a comment - - edited

          The error message comes from workflow-durable-task-step. It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes).

          Another thing to point out is that FilePath#isDirectory will return false in some failure cases (see the Javadoc for File#isDirectory). Perhaps we should update this line to use NIO methods so it throws exceptions instead of returning false in some cases (another case of JENKINS-47324).

          Edit: I filed https://github.com/jenkinsci/jenkins/pull/3864 for that issue.

          Double edit: Also to clarify, if FilePath#isDirectory had thrown an exception, then this code would have been called, which would have caused Pipeline to attempt to connect again rather than aborting the build immediately. That issue wouldn't really explain why we are seeing this on Java 11 and not Java 8, but seems worth investigating.

          dnusbaum Devin Nusbaum added a comment - - edited The error message comes from workflow-durable-task-step . It would be interesting to check whether the directory mentioned in the error actually exists on the agent, or if it is missing, or exists but has a different name (perhaps a randomized suffix or `-` instead of `_` or something, which could be related to recent branch-api changes). Another thing to point out is that FilePath#isDirectory will return false in some failure cases (see the Javadoc for File#isDirectory ). Perhaps we should update this line to use NIO methods so it throws exceptions instead of returning false in some cases (another case of JENKINS-47324 ). Edit: I filed https://github.com/jenkinsci/jenkins/pull/3864  for that issue. Double edit: Also to clarify, if FilePath#isDirectory had thrown an exception, then this code would have been called, which would have caused Pipeline to attempt to connect again rather than aborting the build immediately. That issue wouldn't really explain why we are seeing this on Java 11 and not Java 8, but seems worth investigating.
          basil Basil Crow added a comment -

          I experienced this failure mode twice on January 16, two weeks after upgrading Jenkins from 2.138.1 LTS (with workflow-job 2.25, workflow-cps 2.54, and workflow-durable-task-step 2.21) to 2.150.1 LTS (with workflow-job 2.31, workflow-cps 2.61, and workflow-durable-task-step 2.27). I am not running Java 11. The job has been running daily and has only failed twice with this failure mode, so the error is transient.

          ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod on scale-dc2
          ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod@2 on dc3
          

          When this error occurred on the 16th, I logged into these machines and checked the given directories on the command line. In both cases the directories existed. So I suspect there may have been some transient I/O error at the time.

          basil Basil Crow added a comment - I experienced this failure mode twice on January 16, two weeks after upgrading Jenkins from 2.138.1 LTS (with workflow-job 2.25, workflow-cps 2.54, and workflow-durable-task-step 2.21) to 2.150.1 LTS (with workflow-job 2.31, workflow-cps 2.61, and workflow-durable-task-step 2.27). I am not running Java 11. The job has been running daily and has only failed twice with this failure mode, so the error is transient. ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod on scale-dc2 ERROR: missing workspace /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/master/sync-ova-into-dcod@2 on dc3 When this error occurred on the 16th, I logged into these machines and checked the given directories on the command line. In both cases the directories existed. So I suspect there may have been some transient I/O error at the time.
          oleg_nenashev Oleg Nenashev made changes -
          Epic Link JENKINS-40689 [ 177392 ]
          oleg_nenashev Oleg Nenashev made changes -
          Labels java11-compatibility scrub triaged java11-compatibility triaged
          oleg_nenashev Oleg Nenashev added a comment -

          dnusbaumwhat is the status here? We are about to proceed with Java 11 GA in Jenkins. I do not think it is a blocker, but it would be nice to get your feedback

          oleg_nenashev Oleg Nenashev added a comment - dnusbaum what is the status here? We are about to proceed with Java 11 GA in Jenkins. I do not think it is a blocker, but it would be nice to get your feedback
          dnusbaum Devin Nusbaum added a comment -

          oleg_nenashev Unchanged from my perspective. Given that basil mentioned that they have seen the issue on Java 8, it seems like this is not something specific to Java 11. I closed the PR I mentioned in this comment because I was not able to perform the testing necessary to feel confident about the change, and it wasn't clear to me what kinds of failures would have been exposed by switching to NIO. It would probably be safe to reopen it and change but change the behavior to return false if the directory does not exist to be much closer to the original behavior, but I'm not sure what the benefit would be. If anyone is able to come up with a self-contained and consistent reproduction case, then I would be more than happy to take a look, but without any other ideas I am just grasping at straws for now.

          dnusbaum Devin Nusbaum added a comment - oleg_nenashev Unchanged from my perspective. Given that basil mentioned that they have seen the issue on Java 8, it seems like this is not something specific to Java 11. I closed the PR I mentioned in this comment because I was not able to perform the testing necessary to feel confident about the change, and it wasn't clear to me what kinds of failures would have been exposed by switching to NIO. It would probably be safe to reopen it and change but change the behavior to return false if the directory does not exist to be much closer to the original behavior, but I'm not sure what the benefit would be. If anyone is able to come up with a self-contained and consistent reproduction case, then I would be more than happy to take a look, but without any other ideas I am just grasping at straws for now.
          markewaite Mark Waite made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          oleg_nenashev Oleg Nenashev added a comment -

          Platform SIG meeting: As far as markewaite concerned, it is resolved. 

          oleg_nenashev Oleg Nenashev added a comment - Platform SIG meeting: As far as markewaite concerned, it is resolved. 
          oleg_nenashev Oleg Nenashev made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          oleg_nenashev Oleg Nenashev made changes -
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
          markewaite Mark Waite made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            dnusbaum Devin Nusbaum
            markewaite Mark Waite
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: