Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54884

ssh-slaves or trilead-api stops all git over ssh (git+ssh) from working

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.138.3
      ssh-slaves 1.29.1
      trillead-api 1.01 (? dunno; erased)
    • Similar Issues:
    • Released As:
      ssh-slaves-1.29.2

      Description

      With those two plugins installed, all git+ssh stopped working, even if it had nothing to do with slave nodes.

       > git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from ssh://git@git:7999/proj/jenkins-backups.git
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
      	at hudson.scm.SCM.checkout(SCM.java:504)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
      	at hudson.model.Run.execute(Run.java:1819)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: Permission denied (publickey).
      fatal: Could not read from remote repository.Please make sure you have the correct access rights
      and the repository exists.	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
      	... 11 more
      ERROR: Error fetching remote repo 'origin'
       

      This would happen when pipeline branches were checked as well:

      Started by timer
      [Tue Nov 27 06:36:00 EST 2018] Starting branch indexing...
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git ls-remote ssh://git@git:7999/proj/opencpi.git # timeout=10
      ERROR: [Tue Nov 27 06:36:00 EST 2018] Could not update folder level actions from source e9d3dadd-64f7-458c-b3ba-98edd9652a8c
      hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
      stdout: 
      stderr: Permission denied (publickey).
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
      	at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
      	at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
      	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
      	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
      	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
      	at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      [Tue Nov 27 06:36:00 EST 2018] Finished branch indexing. Indexing took 0.16 sec
      FATAL: Failed to recompute children of Job_1
      hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
      stdout: 
      stderr: Permission denied (publickey).
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
      	at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
      	at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
      	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
      	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
      	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
      	at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      I verified with the server that the "Jenkins" user has permissions normally:

       # From Jenkins Server as user jenkins
      $ ssh -p 7999 git whoami
      opencpi_jenkins
      $ git fetch # works
      # From Jenkins' "Run buildstep before SCM runs"
      $ /bin/sh -xe /tmp/jenkins3916373384688410155.sh
      + ssh -p 7999 git whoami
      opencpi_jenkins
      Success build forhudson.tasks.Shell@18819daa
      

      Reverting to ssh-slaves 1.28.1 (and uninstalling trilead-api) fixed it.

      Last Tuesday, I updated a bunch of plugins as a batch and couldn't even get a page to render. Since I was only at work for a few hours, I simply rolled back all plugins and let it be. Just mentioning that because I see other tickets that indicate these plugins may have been the problem then as well.

        Attachments

          Issue Links

            Activity

            aarondmarasco_vsi Aaron D. Marasco created issue -
            aarondmarasco_vsi Aaron D. Marasco made changes -
            Field Original Value New Value
            Environment ssh-slaves 1.29.1
            trillead-api 1.01 (? dunno; erased)
            Jenkins ver. 2.138.3
            ssh-slaves 1.29.1
            trillead-api 1.01 (? dunno; erased)
            aarondmarasco_vsi Aaron D. Marasco made changes -
            Description With those two plugins installed, all git+ssh stopped working, even if it had nothing to do with slave nodes.
            {code:java}
             > git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*
            ERROR: Error fetching remote repo 'origin'
            hudson.plugins.git.GitException: Failed to fetch from ssh://git@git:7999/proj/jenkins-backups.git
            at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
            at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
            at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
            at hudson.scm.SCM.checkout(SCM.java:504)
            at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
            at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
            at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
            at hudson.model.Run.execute(Run.java:1819)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.Please make sure you have the correct access rights
            and the repository exists. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
            at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
            ... 11 more
            ERROR: Error fetching remote repo 'origin'
             {code}

            This would happen when pipeline branches were checked as well. Reverting to ssh-slaves 1.28.1 (and uninstalling trilead-api) fixed it.

            {code}
            Started by timer
            [Tue Nov 27 06:36:00 EST 2018] Starting branch indexing...
             > git --version # timeout=10
            using GIT_SSH to set credentials
             > git ls-remote ssh://git@git:7999/proj/opencpi.git # timeout=10
            ERROR: [Tue Nov 27 06:36:00 EST 2018] Could not update folder level actions from source e9d3dadd-64f7-458c-b3ba-98edd9652a8c
            hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.

            Please make sure you have the correct access rights
            and the repository exists.

            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
            at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
            at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            [Tue Nov 27 06:36:00 EST 2018] Finished branch indexing. Indexing took 0.16 sec
            FATAL: Failed to recompute children of Job_1
            hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.

            Please make sure you have the correct access rights
            and the repository exists.

            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
            at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
            at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            Finished: FAILURE
            {code}

            I verified with the server that the "Jenkins" user has permissions normally:
            {code}
            # From Jenkins Server as user jenkins
            $ ssh -p 7999 git whoami
            opencpi_jenkins
            $ git fetch # works
            # From Jenkins' "Run buildstep before SCM runs"
            $ /bin/sh -xe /tmp/jenkins3916373384688410155.sh
            + ssh -p 7999 git whoami
            opencpi_jenkins
            Success build forhudson.tasks.Shell@18819daa
            {code}
            With those two plugins installed, all git+ssh stopped working, even if it had nothing to do with slave nodes.
            {code:java}
             > git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*
            ERROR: Error fetching remote repo 'origin'
            hudson.plugins.git.GitException: Failed to fetch from ssh://git@git:7999/proj/jenkins-backups.git
            at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
            at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
            at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
            at hudson.scm.SCM.checkout(SCM.java:504)
            at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
            at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
            at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
            at hudson.model.Run.execute(Run.java:1819)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://git@git:7999/proj/jenkins-backups.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.Please make sure you have the correct access rights
            and the repository exists. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
            at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
            ... 11 more
            ERROR: Error fetching remote repo 'origin'
             {code}
            This would happen when pipeline branches were checked as well:
            {code:java}
            Started by timer
            [Tue Nov 27 06:36:00 EST 2018] Starting branch indexing...
             > git --version # timeout=10
            using GIT_SSH to set credentials
             > git ls-remote ssh://git@git:7999/proj/opencpi.git # timeout=10
            ERROR: [Tue Nov 27 06:36:00 EST 2018] Could not update folder level actions from source e9d3dadd-64f7-458c-b3ba-98edd9652a8c
            hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.

            Please make sure you have the correct access rights
            and the repository exists.

            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
            at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
            at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            [Tue Nov 27 06:36:00 EST 2018] Finished branch indexing. Indexing took 0.16 sec
            FATAL: Failed to recompute children of Job_1
            hudson.plugins.git.GitException: Command "git ls-remote ssh://git@git:7999/proj/opencpi.git" returned status code 128:
            stdout:
            stderr: Permission denied (publickey).
            fatal: Could not read from remote repository.

            Please make sure you have the correct access rights
            and the repository exists.

            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1640)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1631)
            at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2851)
            at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1113)
            at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:765)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:591)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:429)
            Finished: FAILURE
            {code}
            I verified with the server that the "Jenkins" user has permissions normally:
            {code:java}
             # From Jenkins Server as user jenkins
            $ ssh -p 7999 git whoami
            opencpi_jenkins
            $ git fetch # works
            # From Jenkins' "Run buildstep before SCM runs"
            $ /bin/sh -xe /tmp/jenkins3916373384688410155.sh
            + ssh -p 7999 git whoami
            opencpi_jenkins
            Success build forhudson.tasks.Shell@18819daa
            {code}
            Reverting to ssh-slaves 1.28.1 (and uninstalling trilead-api) fixed it.

            Last Tuesday, I updated a bunch of plugins as a batch and couldn't even get a page to render. Since I was only at work for a few hours, I simply rolled back all plugins and let it be. Just mentioning that because I see other tickets that indicate these plugins may have been the problem then as well.
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Link This issue is related to JENKINS-53810 [ JENKINS-53810 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Link This issue is related to JENKINS-53810 [ JENKINS-53810 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Link This issue is related to JENKINS-54746 [ JENKINS-54746 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Remote Link This issue links to "PR (Web Link)" [ 22131 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            ifernandezcalvo Ivan Fernandez Calvo made changes -
            Released As ssh-slaves-1.29.2
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]

              People

              Assignee:
              ifernandezcalvo Ivan Fernandez Calvo
              Reporter:
              aarondmarasco_vsi Aaron D. Marasco
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: