This is most likely not a bug in the git plugin or the git client plugin, but a configuration error on either the agent or the master.

That message usually happens when a Windows agent has defined the git location incorrectly. The preferred way to define the location of git (for Windows and for Linux) is "git" and then allow the operating system path configuration on the agent to find command line git using the operating system path.

Do not define the path to the Windows git executable as "C:\Program Files\GIt\git-cmd.exe" or "C:\Program Files\Git\git-bash.exe", since those are two Windows specific programs which don't work with the git plugin. If you're Windows specific, use either "git" (preferred) or "git.exe" or "git.cmd".

If this is not windows, then you may have exhausted open files or some other system level resource limit on your agent.

Another possible cause on Linux might be that the file system is mounted with "noexec". That seems quite unlikely, but might cause that symptom.

Hi Mark,

Thanks for your inputs.

Here the scenario is when we create new job i am able to see the list of branches and tags when using GIT parameter after the build completes and if i again try to Build With Parameter i am getting above error.

Any advice ?

Thanks
Senthil.

spalani I think I've been able to duplicate the problem, with some further guessing.

The list of tags and branches in the parameterized plugin pick list includes "ssh not found" if I attempt my git parameter plugin based job on a Windows agent and run it twice. So long as I run the job on a Linux agent (any Linux agent, not just limited to the master agent), then it is well behaved. Once I've executed on a Windows agent, it reports "ssh not found" and refuses to show the list of branches/tags.

In my case, I used a URL of the form ssh://hostname:port/path/to/repo.git, though it fails the same way when I use a URL of the form user@hostname:/path/to/repo.git.

Note that all those failures require a Windows machine. They don't seem to happen until a Windows machine is involved. I have a multi-configuration job which uses the same technique to copy files to all my machines (Windows and Linux) and it works correctly. I assume it works correctly because the initial evaluation of the git parameters happens on a Linux machine. When I switch and force that initial evaluation of git parameters to a Windows machine, then the multi-configuration job shows the same failure.

I believe this will need further investigation by the maintainer of the git parameter plugin. The issue may ultimately be in the git client plugin, but I would rather have the git parameter plugin maintainer (or you) perform that investigation.

Alternatives to avoid the problem include:

• Use a string or a list of choices rather than a git parameter to select the branch name
• Run the job on a Linux machine
• Make the job a multi-configuration job that starts on a linux machine, then defines the name of the windows agent to execute the job

Hi,
I will take a look at this on my return from holidays (mid-August)
Thanks markewaite for the description of the problem.

Regards
Boguslaw

Hi markewaite

I have tested it again as per your investigation.

I am able to get the branch and tags list if i choose the build PC slave as Linux.

But when i select the node as windows not able to list branch and tags.

Thanks
Senthil

Hi klimas7/markewaite

Pleas ecan i know any is this fixed in any future releases plugins.

Thanks
Senthil

spalani I can confirm that it is not fixed in the latest versions of the git client plugin, git plugin, and git parameter plugin. My test infrastructure was still showing this failure yesterday.

When the bug is fixed, this bug report will be updated.

You're welcome to assist with investigating and fixing the bug. We'd love to have more developers on Jenkins plugins, especially on bugs that are of interest to them.

Jenkins ver. 2.19.3

Git plugin 2.5.2
Git Changelog 1.49
Git Parameter Plug-In 0.8.1
Git client plugin 2.5.0

Slave is on Windows Server.

[ *** Release **QA ] Download tags from the repository failed hudson.plugins.git.GitException: Command "git ls-remote -t git@bitbucket.org:*/*.git qa" returned status code 128: stdout: stderr: /tmp/ssh8997135691813480349.sh: 6: /tmp/ssh8997135691813480349.sh: ssh: not found fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1643) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1561) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1552) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2729) at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.getTag(GitParameterDefinition.java:332) at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.generateContents(GitParameterDefinition.java:284) at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition$DescriptorImpl.doFillValueItems(GitParameterDefinition.java:558) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

hi klimas7 , can you kindly provide any updates ?

Could you please provide an update on when we can expect a fix for this?

Sorry but I didn't have enough time to investigate deeper this issue.
I will try do this in this week.
(But I think, this is a configuration issue)

I am almost certain that it is not since the same configuration works if you put it into a multi-config project and a Linux slave is checking out first, before the job gets handed off to a Windows slave.

I created very simple test environments.

• Linux, where I installed clean (default plugins) Jenkins LTS 2.73.2 and git-parameter-plugin
  • git 3.6.4
  • git-client 2.6.0
  • git-parameter 0.9.0
• Windows. Important! I installed git on this Windows
• I used repository on GitHub https://github.com/klimas7/Learn

Slave configuration on my Jenkins

Next I generated ssh key and I configured credentials on Jenkins and add this key on my github account.

On that prepared environment I crated first test job (Test_B). This job only builds my github project on windows slave.

And I got output

Next I created job (Test_2B) which have a git-parameter plugin.

And when I first run build I got

Can anyone replicate this problem using the similar configuration and repository on github?

Please build multiple times. The first time it works but not if you try subsequently.

I had consistent failures from a test job that I created in my regression test repository. Unfortunately, today that test job is successful. I confirmed that it builds multiple times successfully. I've seen the problem before, but don't have the specific conditions which will cause it to appear.

Hi marcellus I builds this jobs a few time and everything is ok

klimas7 I ran another test and found a case where I think I can see it consistently.

I have a matrix job using an elastic axis definition which distributes sample data to all my agents.  As a matrix job, it starts execution on a flyweight executor, then launches jobs on each of the agents which match the selected labels.

My default definition of that job allows the flyweight executor to run on any agent.  If after a successful run, I redefine that job to limit the agent for the flyweight executor to "windows", then the job execution reports:

Command "git ls-remote -h git@github.com:MarkEWaite/auth-data.git" returned status code 128:
stdout: 
stderr: /tmp/ssh5538182807084448882.sh: 6: /tmp/ssh5538182807084448882.sh: ssh: not found
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Please look at the Log

Odd things about that job (prior definition attached to the bug report as config-no-flyweight.xml , failing definition attached as config-windows-flyweight.xml ):

1. Multiple preceding successful runs of the job without limiting the flyweight executor label
2. Job is parameterized with git branch
3. Job typically is started by an upstream job with the parameter value passed from the upstream job
4. Works great when started by an upstream job, whether or not the flyweight executor label was set

klimas7 Could you please use the following settings? Maybe this triggers the behavior.

I'm having the same issue but noticed there is no activity since last November. Did everyone figure out a workaround for this?

Can also confirm, having an identical issue. Seems to revolve around Git parameter plugin.

Weirdly enough I've tracked the git ls-remote process as well as temporary sh file that triggers the ssh, and it's all executed on the Jenkins main, not sure why it happens only when the actual build is supposed to take place on a Windows agent (we have two, with different configurations and both suddenly started having the issue).

Also not sure why there is an ssh command executing `git-upload-pack` right after the ls-remote action only when it's being executed via parameter plugin.

Git Plugin -->3.3.0
Git Parameter Plugin-->0.8.0

klimas7

We have a Bitbucket Server as git server instead of using github. But in order to reproduce the issue I requested to grant our build machines access to github.com so I can try to reproduce the issue with your configuration. Hopefully this will help us find the culprit.

Most likely, change made in the JENKINS-40232, caused that the plugin does not behave deterministically. 

Befor version 0.8.0 plugin use repository which featch in job workspace (when the repository not existsed in workspace, plugin did clone to workspace). After version 0.8.0 plugin uses git ls-remote to download branch or tag.

Currently  I have a little more time and I will try investigate deeper this issue. 

Hi all,

To get a list of branches or tags, the plugin creates a GitClient object and sets the workspace as null,
in this case, the GitUtils.workspaceToNode method returns the Jenkins active instance (master).
Due to this, all operations that the plugin performs while downloading branch / tags
are performed on the main jenkins node. Finally, to got list branch / tag the plugin used method GitClient.getRemoteReferences nothing else.
With the above in mind, I am inclined to the fact that it is a configuration problem.
I understand that each of these jobs works correctly on the master node?

Building of our project on the master is not possible due to the fact that it is a .NET 3.5 application which is only functional on a windows slave. However there is a rather ugly workaround that makes use of the multi configuration job type. It is configured to check out the branch on the master first and then hands the build action down to the windows slave. In this scenario it is working.

If a configuration error causes this it would be good to know what the error is supposed to be so we can check and verify it.

Maybe I didn't express myself precisely, I meant it:

I understand that each of these jobs (SCM) works fetch/clone correctly on the master node? (the plugin does not need to build a project)

To point this part of my previous answer out: yes they do fetch/clone correctly on the master.

Hi,

I can't reproduce this issue, but I found code smells, which can provide to unexpected behavior. commit: f85678c3 I changed  it in release 0.9.10.

Additional I changed error handling, please look on wiki page

If this issue steal occurred in your environment after update the plugin. Please add more information which I can use to reproduce this issue and reopen this issue. 

Release 0.9.10

I still have the bug.

However, the error message id not displayed the same way.

Remember that the first time you create a build in a job, this does work. It's only from the second build that the issue occurs.

Prior 0.9.10:

After 0.9.10:

delovan the git plugin uses command line git to clone the repository. Command line git requires a script file that provides the private key for authentication to ssh repositories. That script file invokes the command ssh to provide authenticated transport between command line git and your git repository. That script can't find the ssh command on the path.

That could be because the /tmp directory on the master is mounted with the noexec option. Refer to stackexchange for instructions to check the file system mount options.

That could be because the PATH environment variable has been damaged or altered so that ssh is not on the PATH when running from the Jenkins master. The Jenkins "System Information" page will show you the value of the PATH environment variable on that Jenkins server.

That could be because the ssh command has been removed from the master. Seems unlikely, but possible. Check for /usr/bin/ssh or /bin/ssh to confirm that you have an ssh command installed in the environment where Jenkins is running.

The git plugin uses command line git to clone https repositories as well, but cloning an https repository uses a username / password instead of a private key. Cloning an https repository does not use ssh, while cloning an ssh based repository does use the ssh command.

Hello markewaite and thank you for your response. 

I also forget to mention that I run Jenkins under it's docker version.

Therefore, the /tmp directory with noexec is not affected by that

I did check the PATH variable, it's configured with _/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin_ so ssh is in it.

When I docker exec into the docker container, I can launch ssh commands without issues.

Is there a way to preserve the temporary bash scripts in _/_tmp that are used to launch the git commands? That way, it would be far easier to troubleshoot the issue.

There isn't a way to preserve the temporary bash scripts. That's intentional, since they contain credentials. We want the credential information to be removed from the disc as soon as possible.

You could build a local copy of the git client plugin and remove the code in the finally block that deletes those temporary files.

I had the same problem, and changing from SSH to HTTP did not work. I got the error below when trying HTTP:

I managed to "fix" the problem by creating a new project cloning the existing one. The new one worked fine. I did not change any config at all. Thus I suppose this is really some sort of problem in the Git plugin. Not a user config problem.

robsonhermes Are you sure it's durable ?

With SSH, cloning the project was a workaround but only works for the very first build. Starting the 2nd build, it failed.

Oh damn it. You are right, now I have failure starting from the 2nd attempt.

robsonhermes, I suggest you do as markewaite suggested for keeping tmp files. I've done it and surprisingly I can't reproduce the bug anymore, even though I switch back to the original plugin.

For info, after you build the plugin, just copy the plugin files over the original plugin (keep a backup) and restart jenkins.

With theses files, you will be able to run them directly in bash and have more debug info.

Related: https://issues.jenkins-ci.org/browse/JENKINS-56558

In our case "/tmp" is not mounted with "noexec".

I don't understand why it even tries to execute it on the master machine instead of the slave machine. It clearly should run on the slave machine.