Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42692

When git fetch fails in Scan Multibranch Pipeline fails, the job never stops

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • Jenkins 2.32.2 LTS
      Windows Server 2012 R2
      BitBucket Plugin 1.1.5
      Blue Ocean Beta 1.0.0-b25
      Pipeline 2.5
      Pipeline Multibranch 2.14

      When running the `Scan Multibranch Pipeline` step of my multibranch pipeline (where I believe it fetches the branches to determine which contain a jenkinsfile and have not yet been run) The git fetch command is timing out. (This is probably a credentials or git plugin issue, and not the cause of this bug).

      The bug is that when I get the `ERROR: Timeout after 10 minutes` there is then an `ERROR: Failed to join a process` error, and that seems to lock the Build Executor threads. Preventing Jenkins from restarting even with the /restart url, and instead forcing me to kill the Java process before having it happen all over again.

       From /job/MYPipeline/indexing/console

      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      Started by timer
      [Sat Mar 11 12:14:25 EST 2017] Starting branch indexing...
      Setting origin to https://USER@bitbucket.org/TEAM/PROJECT.git
       > D:\Git\cmd\git.exe config remote.origin.url https://USER@bitbucket.org/TEAM/PROJECT.git # timeout=10
      Fetching & pruning origin...
      Fetching upstream changes from origin
       > D:\Git\cmd\git.exe --version # timeout=10
      using GIT_ASKPASS to set credentials MY_CREDENTIALS
       > D:\Git\cmd\git.exe fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
      ERROR: Timeout after 10 minutes
      ERROR: Failed to join a process
      org.jvnet.winp.WinpException: Failed to read RT_USER_PROCESS_PARAMETERS error=299 at envvar-cmdline.cpp:151
      	at org.jvnet.winp.Native.getCmdLineAndEnvVars(Native Method)
      	at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:126)
      	at org.jvnet.winp.WinProcess.getCommandLine(WinProcess.java:102)
      	at hudson.util.ProcessTree$Windows$1.getArguments(ProcessTree.java:444)
      	at hudson.plugins.msbuild.MsBuildKillingVeto.vetoProcessKilling(MsBuildKillingVeto.java:56)
      	at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:242)
      	at hudson.util.ProcessTree$Windows$1.killRecursively(ProcessTree.java:425)
      	at hudson.util.ProcessTree.killAll(ProcessTree.java:145)
      	at hudson.Proc$LocalProc.destroy(Proc.java:380)
      	at hudson.Proc$LocalProc.kill(Proc.java:372)
      	at hudson.Proc$1.run(Proc.java:159)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)

      And from the other failing task:

      Started by timer
      [Sat Mar 11 11:59:00 EST 2017] Starting branch indexing...
       > D:\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=10
      Setting origin to https://USER@bitbucket.org/TEAM/PROJECT.git
       > D:\Git\cmd\git.exe config remote.origin.url https://USER@bitbucket.org/TEAM/PROJECT.git # timeout=10
      Fetching & pruning origin...
      Fetching upstream changes from origin
       > D:\Git\cmd\git.exe --version # timeout=10
      using GIT_ASKPASS to set credentials MyCredentials
       > D:\Git\cmd\git.exe fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
      ERROR: Timeout after 10 minutes
      ERROR: Failed to join a process
      org.jvnet.winp.WinpException: Failed to read RT_USER_PROCESS_PARAMETERS error=299 at envvar-cmdline.cpp:151
      	at org.jvnet.winp.Native.getCmdLineAndEnvVars(Native Method)
      	at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:126)
      	at org.jvnet.winp.WinProcess.getCommandLine(WinProcess.java:102)
      	at hudson.util.ProcessTree$Windows$1.getArguments(ProcessTree.java:444)
      	at hudson.plugins.msbuild.MsBuildKillingVeto.vetoProcessKilling(MsBuildKillingVeto.java:56)
      	at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:242)
      	at hudson.util.ProcessTree$Windows$1.killRecursively(ProcessTree.java:425)
      	at hudson.util.ProcessTree.killAll(ProcessTree.java:145)
      	at hudson.Proc$LocalProc.destroy(Proc.java:380)
      	at hudson.Proc$LocalProc.kill(Proc.java:372)
      	at hudson.Proc$1.run(Proc.java:159)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Setting origin to https://USER@bitbucket.org/TEAM/PROJECT.git
       > D:\Git\cmd\git.exe config remote.origin.url https://USER@bitbucket.org/TEAM/PROJECT.git # timeout=10
      Fetching & pruning origin...
      Fetching upstream changes from origin
       > D:\Git\cmd\git.exe --version # timeout=10
      using GIT_ASKPASS to set credentials MyCredentials
       > D:\Git\cmd\git.exe fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
      ERROR: Timeout after 10 minutes
      ERROR: Failed to join a process
      org.jvnet.winp.WinpException: Failed to read RT_USER_PROCESS_PARAMETERS error=299 at envvar-cmdline.cpp:151
      	at org.jvnet.winp.Native.getCmdLineAndEnvVars(Native Method)
      	at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:126)
      	at org.jvnet.winp.WinProcess.getCommandLine(WinProcess.java:102)
      	at hudson.util.ProcessTree$Windows$1.getArguments(ProcessTree.java:444)
      	at hudson.plugins.msbuild.MsBuildKillingVeto.vetoProcessKilling(MsBuildKillingVeto.java:56)
      	at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:242)
      	at hudson.util.ProcessTree$Windows$1.killRecursively(ProcessTree.java:425)
      	at hudson.util.ProcessTree.killAll(ProcessTree.java:145)
      	at hudson.Proc$LocalProc.destroy(Proc.java:380)
      	at hudson.Proc$LocalProc.kill(Proc.java:372)
      	at hudson.Proc$1.run(Proc.java:159)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)

       

      When I went in to check today the job had been running for 4 days 4 hours.

          [JENKINS-42692] When git fetch fails in Scan Multibranch Pipeline fails, the job never stops

          Oleg Nenashev added a comment -

          It is a duplicate of JENKINS-30782.
          Maybe it is possible to workaround it in plugins, but I doubt

          Oleg Nenashev added a comment - It is a duplicate of JENKINS-30782 . Maybe it is possible to workaround it in plugins, but I doubt

          Sean MacKay added a comment -

          You appear to be correct. The job not terminating correctly appears to be the crux of the issue.

           

          For completeness sake, the cause of my timeout above was the Git Credential Manager for windows attempting to pop up a username and password dialogue box, but being unable to do so as jenkins runs as the LOCAL_SYSTEM account on windows which does not have desktop access (even with it granted in the services config apparently). 

          In the past I'd gotten around this popup by using a tool to spawn a cmd.exe process as LOCAL_SYSTEM but accessible by myself, then made the git checkout request and handled the dialog myself so it would be saved for Jenkins to use. However, it seems that expires. I removed git and reinstalled without the Git Credential Manager and the Jenkins credential config seems to work. 

          So the error causing this error to occur has been solved for me, but the Jenkins job not ending correctly after an error is still a frustrating experience that can happen.

          Sean MacKay added a comment - You appear to be correct. The job not terminating correctly appears to be the crux of the issue.   For completeness sake, the cause of my timeout above was the Git Credential Manager for windows attempting to pop up a username and password dialogue box, but being unable to do so as jenkins runs as the LOCAL_SYSTEM account on windows which does not have desktop access (even with it granted in the services config apparently).  In the past I'd gotten around this popup by using a tool to spawn a cmd.exe process as LOCAL_SYSTEM but accessible by myself, then made the git checkout request and handled the dialog myself so it would be saved for Jenkins to use. However, it seems that expires. I removed git and reinstalled without the Git Credential Manager and the Jenkins credential config seems to work.  So the error causing this error to occur has been solved for me, but the Jenkins job not ending correctly after an error is still a frustrating experience that can happen.

          Sean MacKay added a comment -

          I'm going to downgrade to minor because solving the underlying issrom preventing the use of Jenkins.

          Sean MacKay added a comment - I'm going to downgrade to minor because solving the underlying issrom preventing the use of Jenkins.

            Unassigned Unassigned
            ironsean Sean MacKay
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: