Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23743

Failure on Git Polling Log when using Linux Master and Windows Slave

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Environment:
    • Similar Issues:

      Description

      I have a Master and Two slaves. The Linux Slave is behaving very well.

      The problem is only related to the Windows Slave. It works nicely, till the problem happens. The only way to fix it is restarting Jenkins. Its a real problem, and affecting my teamwork. People ask me why the build was not triggered, and I then ask them to commit something else, after I quickly reboot Jenkins.

      While the error is happening, I get:

      Git Polling Log
      Started on Jul 3, 2014 3:10:24 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision 8ae038f280a0faa58243ec1df7af64d961a951c8 (origin/master)
      using GIT_SSH to set credentials
      > C:\git\cmd\git.exe ls-remote -h git@gitlab:group/repo.git master
      FATAL: hudson.plugins.git.GitException: Error performing command: C:\git\cmd\git.exe ls-remote -h git@gitlab:group/repo.git master
      hudson.util.IOException2: hudson.plugins.git.GitException: Error performing command: C:\git\cmd\git.exe ls-remote -h git@gitlab:group/repo.git master
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:459)
      at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
      at hudson.scm.SCM.poll(SCM.java:373)
      at hudson.model.AbstractProject._poll(AbstractProject.java:1584)
      at hudson.model.AbstractProject.poll(AbstractProject.java:1493)
      at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:462)
      at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:491)
      at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:679)
      Caused by: hudson.plugins.git.GitException: Error performing command: C:\git\cmd\git.exe ls-remote -h git@gitlab:group/repo.git master
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1332)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1110)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1101)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getHeadRev(CliGitAPIImpl.java:1788)
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:492)
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:457)
      ... 13 more
      Caused by: java.io.IOException: Cannot run program "C:\git\cmd\git.exe": java.io.IOException: error=2, No such file or directory
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
      at hudson.Proc$LocalProc.<init>(Proc.java:244)
      at hudson.Proc$LocalProc.<init>(Proc.java:216)
      at hudson.Launcher$LocalLauncher.launch(Launcher.java:773)
      at hudson.Launcher$ProcStarter.start(Launcher.java:353)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1321)
      ... 19 more
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
      at java.lang.ProcessImpl.start(ProcessImpl.java:81)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
      ... 24 more
      Done. Took 68 ms
      No changes

      I know, it looks like git.exe is not there. But it is, really is. After restarting Jenkins, this happens (on another repo I was able to see the same error, and not I can push something).

      Started on Jul 3, 2014 4:04:40 PM
      Polling SCM changes on WINMACHINE
      Using strategy: Default
      [poll] Last Built Revision: Revision 62e80167e7067ede25727c3cb96885551694c860 (origin/master)
      > C:\git\cmd\git.exe rev-parse --is-inside-work-tree
      Fetching changes from the remote Git repositories
      > C:\git\cmd\git.exe config remote.origin.url git@gitlab:tools/repo.git
      Fetching upstream changes from git@gitlab:tools/repo.git
      > C:\git\cmd\git.exe --version
      using GIT_SSH to set credentials
      > C:\git\cmd\git.exe fetch --tags --progress git@gitlab:tools/repo.git +refs/heads/:refs/remotes/origin/
      Polling for changes in
      Seen branch in repository origin/master
      Seen 1 remote branch
      > C:\git\cmd\git.exe log --full-history --no-abbrev --format=raw -M -m --raw 62e80167e7067ede25727c3cb96885551694c860..bb82f97faa7e27ba53c600050c8bef66df7fe5d0
      Done. Took 18 sec
      Changes found

      Interesting is that now, right below the date, it says:
      "Polling SCM changes on WINMACHINE"
      This was not available on previous information...
      It cannot be trying to go to the wrong machine... or "no-machine"? Its odd...

      And I am using "branches to build" = */master

      Another interesting thing... when not works, I can see this line:
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
      Why I got an UNUXProcess error, if I am running it on a WinMachine? Does it make sense?

      As per the checklist:

      • I was not running git --version as build step, but was using git log. I already changed my pre-build script, which affects all jobs. Cannot force the error to happen, so I need to wait will next time the problem happen
      • All Projects are restricted to run or on Linux or on Windows Slave machines. In this case, it affects only projects tied to Windows Slave.
      • My GitPolling is configured as @yearly, since my GitLab installation does the trigger to the polling.
      • I am not forcing pulling. I could not understood why I should use this, and on its description, we can read: "If this option is selected, polling will require a workspace and might trigger unwanted builds (see JENKINS-10131)."

      The only fix I can do is rebooting Jenkins. Then everything starts to work again, for a while.

      Lovato

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          You might try a work around of adding the "Force polling using workspace" as an "Additional behaviour" in your git SCM definition of the windows job.

          If the poll runs on the Linux node (the master) but somehow decides it needs to use the Windows git configuration to poll the master, that might cause the Linux master to attempt to execute git from the Windows git location, instead of the Linux git location. That would be a bug, and I'd be surprised at that bug, since there are many installations of Jenkins git which use a wide mix of Linux and Windows slaves.

          I'm not aware of such a bug, but if the "Force polling using workspace" resolves the problem for you, it may hint that is an area for more investigation.

          Show
          markewaite Mark Waite added a comment - You might try a work around of adding the "Force polling using workspace" as an "Additional behaviour" in your git SCM definition of the windows job. If the poll runs on the Linux node (the master) but somehow decides it needs to use the Windows git configuration to poll the master, that might cause the Linux master to attempt to execute git from the Windows git location, instead of the Linux git location. That would be a bug, and I'd be surprised at that bug, since there are many installations of Jenkins git which use a wide mix of Linux and Windows slaves. I'm not aware of such a bug, but if the "Force polling using workspace" resolves the problem for you, it may hint that is an area for more investigation.
          Hide
          maglovato Marco Lovato added a comment -

          Ok. Thank you.
          People here changed the configuration w/o being aware the real problem, and added git (or git.exe) to the path, because they thought that this was the problem, removing absolute path from git configuration on all nodes.

          That saying, scenario changed. I will wait till the issue happens again, reboot jenkins and set this flag.

          Will post here as soon as it happens.

          Show
          maglovato Marco Lovato added a comment - Ok. Thank you. People here changed the configuration w/o being aware the real problem, and added git (or git.exe) to the path, because they thought that this was the problem, removing absolute path from git configuration on all nodes. That saying, scenario changed. I will wait till the issue happens again, reboot jenkins and set this flag. Will post here as soon as it happens.
          Hide
          cdoucette Charles Doucette added a comment -

          FYI - When I used "Force polling using workspace - my problem was resolved".
          I see that this issue is resolved.
          What version of the plugin will include this fix?

          Show
          cdoucette Charles Doucette added a comment - FYI - When I used "Force polling using workspace - my problem was resolved". I see that this issue is resolved. What version of the plugin will include this fix?
          Hide
          markewaite Mark Waite added a comment -

          No change was made to the plugin to resolve this issue. Marco Lovato reported that someone had changed the configuration of his system. Once he changed the configuration back to the original, the problem was resolved.

          If you have a case which shows this type of problem and can provide steps which will duplicate the problem, please open a new bug report with those steps described in detail.

          Show
          markewaite Mark Waite added a comment - No change was made to the plugin to resolve this issue. Marco Lovato reported that someone had changed the configuration of his system. Once he changed the configuration back to the original, the problem was resolved. If you have a case which shows this type of problem and can provide steps which will duplicate the problem, please open a new bug report with those steps described in detail.
          Hide
          peter_ricardo peter ricardo added a comment - - edited

          I have the same problem. The "Force polling using workspace" solution seems to solve the problem, but it actually disables checking and now runs the JOB independent of change in SCM. In the "scmPollLog" log you realize that Jenkins ignores whether or not there was a change. for the execution of the JOB.

          This scenario really only occurs on my "Windows" slaves. On "Linux" slaves this problem does not occur.

          Below is an account of the "scmPollLog" of the Linux environment and Windows environment respectively:

          [Linux Slave]

          Started on Sep 3, 2019 8:04:00 PM
          no baseline polling in / opt / programs / jenkins / home / jobs / DEVOPS / jobs / example-pipeline-build-once / jobs / single-job / jobs / 0-job-pipeline / workspace @ libs / pipeline-lego on
          Using strategy: Default
          [poll] Last Built Revision: Revision a789961c69b460065cdf02969c945cb40b1cdc81 (refs / remotes / origin / master)
          using credential 7701291b-c765-4ebc-a41a-a2150119532d
           > / usr / bin / git --version # timeout = 10
          using GIT_SSH to set credentials git.serpro Publishing User (Global Key)
           > / usr / bin / git ls-remote -h git@git.serpro: devops / aicat / delivery-continue-example-jsf.git # timeout = 10
          Found 4 remote heads on git@git.serpro: devops / aicat / delivery-continue-example-jsf.git
          [poll] Latest remote head revision on refs / heads / master is: a789961c69b460065cdf02969c945cb40b1cdc81 - already built by 168
          Done Took 0.64 sec
          No changes

          -------------
          [Windows slave]

          Started on Sep 3, 2019 8:06:00 PM
          no baseline polling in / opt / programs / jenkins / home / jobs / DEVOPS / jobs / example-pipeline-build-once / jobs / single-job / jobs / 0-job-pipeline / workspace @ libs / pipeline-lego on
          Using strategy: Default
          [poll] Last Built Revision: Revision a789961c69b460065cdf02969c945cb40b1cdc81 (refs / remotes / origin / master)
          using credential 7701291b-c765-4ebc-a41a-a2150119532d
           > C: \ Program Files \ Git \ cmd \ git.exe --version # timeout = 10
          using GIT_SSH to set credentials git.serpro Publishing User (Global Key)
           > C: \ Program Files \ Git \ cmd \ git.exe ls-remote -h git@git.serpro: devops / aicat / delivery-continue-example-jsf.git # timeout = 10
          ERROR: polling failed in E: \ Jenkins2 \ Slaves \ workspace \ DEVOPS \ example-pipeline-build-once \ single-job \ 0-job-pipeline on windows_teste_spo
          java.io.IOException: error = 2, No such file or directory
          at java.lang.UNIXProcess.forkAndExec (Native Method)
          at java.lang.UNIXProcess. <init> (UNIXProcess.java:247)
          at java.lang.ProcessImpl.start (ProcessImpl.java:134)
          at java.lang.ProcessBuilder.start (ProcessBuilder.java:1029)
          Caused: java.io.IOException: Cannot run program "C: \ Program Files \ Git \ cmd \ git.exe": error = 2, No such file or directory
          at java.lang.ProcessBuilder.start (ProcessBuilder.java:1048)
          at hudson.Proc $ LocalProc. <init> (Proc.java:250)
          at hudson.Proc $ LocalProc. <init> (Proc.java:219)
          at hudson.Launcher $ LocalLauncher.launch (Launcher.java:937)
          at hudson.Launcher $ ProcStarter.start (Launcher.java:455)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn (CliGitAPIImpl.java:2038)
          Caused: hudson.plugins.git.GitException: Error performing command: C: \ Program Files \ Git \ cmd \ git.exe ls-remote -h git@git.serpro: devops / aicat / delivery-continuous-example-jsf. git
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn (CliGitAPIImpl.java:2049)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1761)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1666)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1657)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getHeadRev (CliGitAPIImpl.java:2824)
          at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl (GitSCM.java:699)
          at hudson.plugins.git.GitSCM.compareRemoteRevisionWith (GitSCM.java:658)
          Caused: java.io.IOException
          at hudson.plugins.git.GitSCM.compareRemoteRevisionWith (GitSCM.java:660)
          at org.jenkinsci.plugins.workflow.job.WorkflowJob.poll (WorkflowJob.java:617)
          at hudson.triggers.SCMTrigger $ Runner.runPolling (SCMTrigger.java:603)
          at hudson.triggers.SCMTrigger $ Runner.run (SCMTrigger.java:649)
          at hudson.util.SequentialExecutionQueue $ QueueEntry.run (SequentialExecutionQueue.java:119)
          at java.util.concurrent.Executors $ RunnableAdapter.call (Executors.java:511)
          at java.util.concurrent.FutureTask.run (FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java:624)
          at java.lang.Thread.run (Thread.java:748)
          Done Took 32 ms
          No changes
          Page generated: 09/03/2019 8:06 pm BRTJenkins ver. 2,176.2

          Show
          peter_ricardo peter ricardo added a comment - - edited I have the same problem. The "Force polling using workspace" solution seems to solve the problem, but it actually disables checking and now runs the JOB independent of change in SCM. In the "scmPollLog" log you realize that Jenkins ignores whether or not there was a change. for the execution of the JOB. This scenario really only occurs on my "Windows" slaves. On "Linux" slaves this problem does not occur. Below is an account of the "scmPollLog" of the Linux environment and Windows environment respectively: [Linux Slave] Started on Sep 3, 2019 8:04:00 PM no baseline polling in / opt / programs / jenkins / home / jobs / DEVOPS / jobs / example-pipeline-build-once / jobs / single-job / jobs / 0-job-pipeline / workspace @ libs / pipeline-lego on Using strategy: Default [poll] Last Built Revision: Revision a789961c69b460065cdf02969c945cb40b1cdc81 (refs / remotes / origin / master) using credential 7701291b-c765-4ebc-a41a-a2150119532d  > / usr / bin / git --version # timeout = 10 using GIT_SSH to set credentials git.serpro Publishing User (Global Key)  > / usr / bin / git ls-remote -h git@git.serpro: devops / aicat / delivery-continue-example-jsf.git # timeout = 10 Found 4 remote heads on git@git.serpro: devops / aicat / delivery-continue-example-jsf.git [poll] Latest remote head revision on refs / heads / master is: a789961c69b460065cdf02969c945cb40b1cdc81 - already built by 168 Done Took 0.64 sec No changes ------------- [Windows slave] Started on Sep 3, 2019 8:06:00 PM no baseline polling in / opt / programs / jenkins / home / jobs / DEVOPS / jobs / example-pipeline-build-once / jobs / single-job / jobs / 0-job-pipeline / workspace @ libs / pipeline-lego on Using strategy: Default [poll] Last Built Revision: Revision a789961c69b460065cdf02969c945cb40b1cdc81 (refs / remotes / origin / master) using credential 7701291b-c765-4ebc-a41a-a2150119532d  > C: \ Program Files \ Git \ cmd \ git.exe --version # timeout = 10 using GIT_SSH to set credentials git.serpro Publishing User (Global Key)  > C: \ Program Files \ Git \ cmd \ git.exe ls-remote -h git@git.serpro: devops / aicat / delivery-continue-example-jsf.git # timeout = 10 ERROR: polling failed in E: \ Jenkins2 \ Slaves \ workspace \ DEVOPS \ example-pipeline-build-once \ single-job \ 0-job-pipeline on windows_teste_spo java.io.IOException: error = 2, No such file or directory at java.lang.UNIXProcess.forkAndExec (Native Method) at java.lang.UNIXProcess. <init> (UNIXProcess.java:247) at java.lang.ProcessImpl.start (ProcessImpl.java:134) at java.lang.ProcessBuilder.start (ProcessBuilder.java:1029) Caused: java.io.IOException: Cannot run program "C: \ Program Files \ Git \ cmd \ git.exe": error = 2, No such file or directory at java.lang.ProcessBuilder.start (ProcessBuilder.java:1048) at hudson.Proc $ LocalProc. <init> (Proc.java:250) at hudson.Proc $ LocalProc. <init> (Proc.java:219) at hudson.Launcher $ LocalLauncher.launch (Launcher.java:937) at hudson.Launcher $ ProcStarter.start (Launcher.java:455) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn (CliGitAPIImpl.java:2038) Caused: hudson.plugins.git.GitException: Error performing command: C: \ Program Files \ Git \ cmd \ git.exe ls-remote -h git@git.serpro: devops / aicat / delivery-continuous-example-jsf. git at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn (CliGitAPIImpl.java:2049) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1761) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1666) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials (CliGitAPIImpl.java:1657) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getHeadRev (CliGitAPIImpl.java:2824) at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl (GitSCM.java:699) at hudson.plugins.git.GitSCM.compareRemoteRevisionWith (GitSCM.java:658) Caused: java.io.IOException at hudson.plugins.git.GitSCM.compareRemoteRevisionWith (GitSCM.java:660) at org.jenkinsci.plugins.workflow.job.WorkflowJob.poll (WorkflowJob.java:617) at hudson.triggers.SCMTrigger $ Runner.runPolling (SCMTrigger.java:603) at hudson.triggers.SCMTrigger $ Runner.run (SCMTrigger.java:649) at hudson.util.SequentialExecutionQueue $ QueueEntry.run (SequentialExecutionQueue.java:119) at java.util.concurrent.Executors $ RunnableAdapter.call (Executors.java:511) at java.util.concurrent.FutureTask.run (FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java:624) at java.lang.Thread.run (Thread.java:748) Done Took 32 ms No changes Page generated: 09/03/2019 8:06 pm BRTJenkins ver. 2,176.2
          Hide
          markewaite Mark Waite added a comment -

          You might check for a possible file descriptor leak from code being called on the agent. Perform a disconnect of the agent, then connect it again.

          If that resolves it, then you have the option of doing a disconnect / reconnect as a workaround.

          If that does not resolve it, you could try killing and restarting the agent process on the agent computer.

          Show
          markewaite Mark Waite added a comment - You might check for a possible file descriptor leak from code being called on the agent. Perform a disconnect of the agent, then connect it again. If that resolves it, then you have the option of doing a disconnect / reconnect as a workaround. If that does not resolve it, you could try killing and restarting the agent process on the agent computer.
          Hide
          peter_ricardo peter ricardo added a comment -

          Thanks for the tip, I will try here and notice about the result ...

          Show
          peter_ricardo peter ricardo added a comment - Thanks for the tip, I will try here and notice about the result ...
          Hide
          peter_ricardo peter ricardo added a comment -

          The issue has been resolved by updating the client version 'git' on the slave. It went back to work after this update.

          Thanks!

          Show
          peter_ricardo peter ricardo added a comment - The issue has been resolved by updating the client version 'git' on the slave. It went back to work after this update. Thanks!

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            maglovato Marco Lovato
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: