Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62837

Use SSH credentials stored in .ssh directory as well as those specified in Jenkins

      Hoping that someone feels a bit of ownership to this plugin  I just can't get this plugin to work. I have now tried in vain for several days without luck. 

      (I also see what is described here: https://issues.jenkins.io/browse/JENKINS-62463)

       

      Installation works fine. Have setup the Bitbucket server in the System Configuration part of Jenkins with appropriate creds, tested communication and it states:  "Jenkins can connect with Bitbucket Server."

      Then I created a VERY basic none-pipeline freestyle job in Jenkins with just an hello world printout. 

      In the SCM section I add the new standalone "Bitbucket Server" and it kindly helps me with the values to fill in so there is a connection towards the Bitbucket server. 

      Project is TEST, "Repository name" is testing, "Branches to Build" is set to refs/heads/develop since we use that branch by default. It all seems to be fine.

      I also tick the "Bitbucket Server trigger build after push" checkbox to automatically create the webhook in Bitbucket.

      And finally, I add a bash and an "echo Hello world" Thats it.

       

      In the feature description it states:

      "Automatic webhook creation in a Bitbucket Server repo when a Jenkins job is saved"

      so I save the new job and the following is printed out in the Jenkins log:

      2020-06-27 06:11:04.886+0000 [id=4528]  INFO    c.a.b.j.i.t.r.BitbucketWebhookHandler#process: New Webhook registered - {"id":5,"name":"14a8acf34ba3061d5ddbac415ce525d671136238","url":"https://jenkins.work.ci/bitbucket-server-webhook/trigger","events":[repo:refs_changed],"active":true}
      2020-06-27 06:11:04.886+0000 [id=4528]  INFO    c.a.b.j.i.t.BitbucketWebhookTriggerImpl$BitbucketWebhookTriggerDescriptor#registerWebhook: Webhook returned -{"id":5,"name":"14a8acf34ba3061d5ddbac415ce525d671136238","url":"https://jenkins.work.ci/bitbucket-server-webhook/trigger","events":[repo:refs_changed],"active":true}

      FINE!

      Then I make a simple change in one of the files in the TEST/testing-repo to start the procedure. Now I can see some action but not what I wanted. In the Jenkins master log I can see: 

      2020-06-27 06:37:07.471+0000 [id=7935]  SEVERE  c.a.b.j.i.t.BitbucketTriggerWorker#run: Failed to trigger job hudson.model.FreeStyleProject@4f05787a[TEST-JOB-WEBHOOK-TRIGGERED-JOB] because an error occurred while writing the polling log to C:\Jenkins\jobs\TEST-JOB-WEBHOOK-TRIGGERED-JOB\bitbucket-webhook-trigger.logjava.lang.IllegalArgumentException: Git repository URL 1 is an empty string in job definition. Checkout requires a valid repository URL        
        at hudson.plugins.git.GitSCM.buildEnvironment(GitSCM.java:1409)        
        at com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCM.buildEnvironment(BitbucketSCM.java:180)
        at hudson.scm.SCM.buildEnvVars(SCM.java:554)      
        at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:874)      
        at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:497)      
        at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:709)      
        at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:674)      
        at com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCM.compareRemoteRevisionWith(BitbucketSCM.java:214)
        at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:401)      
        at hudson.scm.SCM.poll(SCM.java:418)       
        at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1410)      
        at hudson.model.AbstractProject._poll(AbstractProject.java:1380)      
        at hudson.model.AbstractProject.poll(AbstractProject.java:1291)       
        at com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketTriggerWorker.run(BitbucketTriggerWorker.java:70)
        at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)      
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)      
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)      
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)      
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)      
        at java.lang.Thread.run(Thread.java:748)

       

      On top of that, I get several of these: (These jobs are deactivated)

      2020-06-27 06:37:09.619+0000 [id=7667]  WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID b7ad95da-69c4-4d3a-b4c8-3d49419d6d78java.io.IOException: TEST-JOB-MULTI-REPO-pmp is not buildable!      
        at jenkins.model.ParameterizedJobMixIn.doBuildWithParameters(ParameterizedJobMixIn.java:233)      
        at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuildWithParameters(ParameterizedJobMixIn.java:416)  
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)      
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
      ...
      
      2020-06-27 06:37:09.697+0000 [id=7668]  WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 835b22c2-5eac-47b9-9508-867934906c99java.io.IOException: TEST-JOB-MULTI-REPO-pmp is not buildable!      
        at jenkins.model.ParameterizedJobMixIn.doBuildWithParameters(ParameterizedJobMixIn.java:233)      
        at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuildWithParameters(ParameterizedJobMixIn.java:416) 
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      ...
      
      2020-06-27 06:37:09.898+0000 [id=7363]  WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 64d7479d-c512-4759-8f2e-32615ea30c12java.io.IOException: TEST-JOB-MULTI-REPO-pmp is not buildable!      
        at jenkins.model.ParameterizedJobMixIn.doBuildWithParameters(ParameterizedJobMixIn.java:233)      
        at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuildWithParameters(ParameterizedJobMixIn.java:416)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      ...
      2020-06-27 06:37:10.236+0000 [id=5185]  WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 3c2674de-02a7-451c-8cac-0aaf9f7d95edjava.io.IOException: TEST-JOB-RHELBS01-PRINT-NOTIFIER-VARS-pmp is not buildable!      
        at jenkins.model.ParameterizedJobMixIn.doBuildWithParameters(ParameterizedJobMixIn.java:233)      
        at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuildWithParameters(ParameterizedJobMixIn.java:416)  
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      ...

      And more......

       

      Now then, I'm assuming that the description that comes with the plugin is correct so I have only added the BitBucket Server in the SCM configuration of the job. No multiple SCMs. Just the one Bitbucket Server. Is that the way it is to be setup?I ask these questions since I get a slightly different behaviour if I set it up using "Multiple SCMs", One for bitbucket and another for Git pointing to the same project, repository and refspec.Doing it this way makes the "java.lang.IllegalArgumentException" for the TEST-JOB-WEBHOOK-TRIGGERED-JOB disappear. 

      In none of all the attempts I have done have the push in the bitbucket triggered the TEST-JOB-WEBHOOK-TRIGGERED-JOB to execute.

      I also see these printout every now and then. Don't know if they are related.

      2020-06-27 06:45:51.825+0000 [id=8642]  INFO    c.a.b.j.i.h.HttpRequestExecutorImpl#handleError: Bitbucket - did not accept the request
      2020-06-27 06:50:35.884+0000 [id=8641]  INFO    c.a.b.j.i.h.HttpRequestExecutorImpl#handleError: Bitbucket - did not accept the request
      2020-06-27 06:50:35.890+0000 [id=7286]  INFO    c.a.b.j.i.h.HttpRequestExecutorImpl#handleError: Bitbucket - did not accept the request
      2020-06-27 06:50:35.893+0000 [id=8639]  INFO    c.a.b.j.i.h.HttpRequestExecutorImpl#handleError: Bitbucket - did not accept the request
       

       

      Final piece of info i see in the Bitbucket:

      -------------------------------------------

      Active Name URL Last response Actions
      ACTIVE 14a8acf34ba3061d5ddbac415ce525d671136238 https://jenkins.work.ci/bitbucket-server-webhook/trigger 200 
      
      The last success and failure for each event are recorded in the table below (up to the last 30 days).
      Event type      Last success       Last failure Successful calls
      Repository refs updated 7 mins ago Never failed 1/1 (100%)

      Request details

      Event type:repo:refs_changed
      URL endpoint:https://jenkins.work.ci/bitbucket-server-webhook/trigger
      
      Headers
      X-Request-Id: 3f0ec2dc-af69-4564-9634-e3a205571f4b
      Content-Type: application/json; charset=utf-8
      X-Event-Key: repo:refs_changed
      Body{
        "eventKey":"repo:refs_changed",
        "date":"2020-06-27T08:52:16+0200",
        "actor{
          "name":"tester",
          "emailAddress":"tester@tester.com",
          "id":52,
          "displayName":"Tester",
          "active":true,
          "slug":"tester",
          "type":"NORMAL",
          "links":{"self":[{"href":"https://jenkins.work.ci/users/tester"}]}
        },
        "repository":{
          "slug":"testing",
          "id":126,
          "name":"testing",
          "scmId":"git",
          "state":"AVAILABLE",
          "statusMessage":"Available",
          "forkable":false,
          "project":{
              "key":"TEST",
              "id":122,
              "name":"TEST",
              "description":"Common project",
              "public":false,
              "type":"NORMAL",
              "links":{"self":[{"href":"https://jenkins.work.ci/projects/TEST"}]}
          },
          "public":false,
          "links":{
              "clone":[{"href":"ssh://git@bitbucket.work.ci:7999/test/testing.git","name":"ssh"}],
              "self":[{"href":"https://bitucket.work.ci/projects/TEST/repos/testing/browse"}]
          }
        },
        "changes":[{"ref": {"id":"refs/heads/develop","displayId":"develop","type":"BRANCH"},
            "refId":"refs/heads/develop",
            "fromHash":"a22cfb503f4bef9687a4d091ffa6f05711f41922",
            "toHash":"9011a9359e008b8cfb98decbe19d37a52ded4925",
            "type":"UPDATE"}]
      }

      Response details

      HTTP status:200
      Duration45msHeadersDate: Sat, 27 Jun 2020 06:52:16 GMT
      Server: Jetty(9.4.27.v20200227)
      Set-Cookie: ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Secure; HttpOnly
      X-Content-Type-Options: nosniff
      Via: 1.1 localhost (Apache-HttpClient/4.5.5 (cache))
      Content-Length: 0
      Expires: Thu, 01 Jan 1970 00:00:00 GMT
       

      Note that we only allow access to bitbucket through SSH. We have disabled SCM connections to Bitbucket over HTTP(S). 

      So, what am I doing wrong here?

      Regards

      Kent Granstrom

       

          [JENKINS-62837] Use SSH credentials stored in .ssh directory as well as those specified in Jenkins

          Hi Kent, thanks for raising this ticket.

          From what I can read, it looks like you're experiencing two separate issues:

          The ParameterizedJob not being buildable is also occurring, which I'm not certain is a separate issue, but I believe both are likely being caused because you are attempting to build using SSh credentials. Unfortunately, the current version of the plugin does not support clone by SSh.

          The feature is being actively worked on, although it has been delayed we hope to release it as soon as possible- you can watch for updates on this issue: https://issues.jenkins-ci.org/browse/JENKINS-60492

          What has me confused is right now we have no method of even providing SSh credentials to the Bitbucket SCM. Can you explain how you are currently setting up your build? I wonder if there may be other issues at play here.

          Thanks,

          Martin

          Martin Henschke added a comment - Hi Kent, thanks for raising this ticket. From what I can read, it looks like you're experiencing two separate issues: Your pipeline job is not being triggered on push, even though a webhook is being sent ( https://issues.jenkins.io/browse/JENKINS-62463 ) Your builds are failing when run because Bitbucket is failing to accept the request. The ParameterizedJob not being buildable is also occurring, which I'm not certain is a separate issue, but I believe both are likely being caused because you are attempting to build using SSh credentials. Unfortunately, the current version of the plugin does not support clone by SSh. The feature is being actively worked on, although it has been delayed we hope to release it as soon as possible- you can watch for updates on this issue: https://issues.jenkins-ci.org/browse/JENKINS-60492 What has me confused is right now we have no method of even providing SSh credentials to the Bitbucket SCM. Can you explain how you are currently setting up your build? I wonder if there may be other issues at play here. Thanks, Martin

          Kent Granström added a comment - - edited

          Thanks for replying mhenschke_atlassian

          Yes, there are at least 2 issues, perhaps more, and the one you are referring to is similar to one of the things I see even though that issue points towards a pipeline build. 

          It appears as it is not build-job-type-specific since I have tested this on both classic and pipeline jobs and I get the same result on both type of jobs.

          With regards to the failing. I am not sure if it actually fails even though it states "Bitbucket - did not accept the request". Looking at the printouts with some debugging switched on it appears that it has to do with one of the REST-calls where it checks mirrors requiring a Bitbucket Data Center license, and we have a Bitbucket Server license so I guess the reply on that call is interpreted in this way. Anyway, this is me guessing.

           

          Bitbucket - call successful
          jul 04, 2020 9:17:43 FM FINE com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl
          Bitbucket - call successful
          jul 04, 2020 9:17:43 FM INFO com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl handleError
          Bitbucket - did not accept the request
          jul 04, 2020 9:17:43 FM FINE com.atlassian.bitbucket.jenkins.internal.scm.BitbucketMirrorHandler
          Failed to retrieve mirroring information for project TEST and repo testing
          com.atlassian.bitbucket.jenkins.internal.client.exception.BadRequestException: - response: 409 with body: '{"errors":[{"context":null,"message":"Mirroring requires a Bitbucket Data Center license.","exceptionName":"com.atlassian.bitbucket.mirroring.upstream.MirroringDisabledException"}]}'
               at  com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.handleError( HttpRequestExecutorImpl.java:126)
               at com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.executeRequest (HttpRequestExecutorImpl.java:81)
               at com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.executeGet( HttpRequestExecutorImpl.java:50)
               at com.atlassian.bitbucket.jenkins.internal.client.BitbucketRequestExecutor.makeGetRequest( BitbucketRequestExecutor.java:159)
           at com.atlassian.bitbucket.jenkins.internal.client.BitbucketRequestExecutor.makeGetRequest( BitbucketRequestExecutor.java:87)
          ...

          With regards to the "how do we do it"...

          Nothing special in fact. Using the Git SCM we just supply the "Repository URL" with an SSH URL like: ssh://git@bitbucket.work.ci/TEST/testing.git

          The credentials are basically not necessary for this since we're using the ssh/config to specify the id_rsa and the host is a known host. The creds are used for GUI logon only.

          // code placeholder
          bash-4.2$ more .ssh/config
          # BitBucket
          Host bitbucket.work.ci
          User git
          Hostname bitbucket.work.ci
          Port 7999
          IdentityFile ~/.ssh/id_rsa

          With the Bitbucket server plugin there is no way of specifying the ssh URL so I'm assuming that the path is derived from the "Bitbucket Server Instance", "Project Name" and "Repository name". If that is internally used for a clone of a [https://|https:].... URL, it will fail since we don't allow that. SSH only!

          Let me know if this is sufficient to make progress on this issue. Just to let you know that I'd be happy test it out any solution in case you need it.

          Finally, yes, I tested the "Test Connection" and it works.

          Regards

          Kent

          Kent Granström added a comment - - edited Thanks for replying mhenschke_atlassian .  Yes, there are at least 2 issues, perhaps more, and the one you are referring to is similar to one of the things I see even though that issue points towards a pipeline build.  It appears as it is not build-job-type-specific since I have tested this on both classic and pipeline jobs and I get the same result on both type of jobs. With regards to the failing. I am not sure if it actually fails even though it states "Bitbucket - did not accept the request". Looking at the printouts with some debugging switched on it appears that it has to do with one of the REST-calls where it checks mirrors requiring a Bitbucket Data Center license, and we have a Bitbucket Server license so I guess the reply on that call is interpreted in this way. Anyway, this is me guessing.   Bitbucket - call successful jul 04, 2020 9:17:43 FM FINE com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl Bitbucket - call successful jul 04, 2020 9:17:43 FM INFO com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl handleError Bitbucket - did not accept the request jul 04, 2020 9:17:43 FM FINE com.atlassian.bitbucket.jenkins.internal.scm.BitbucketMirrorHandler Failed to retrieve mirroring information for project TEST and repo testing com.atlassian.bitbucket.jenkins.internal.client.exception.BadRequestException: - response: 409 with body: '{"errors":[{"context":null,"message":"Mirroring requires a Bitbucket Data Center license.","exceptionName":"com.atlassian.bitbucket.mirroring.upstream.MirroringDisabledException"}]}' at com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.handleError( HttpRequestExecutorImpl.java:126) at com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.executeRequest (HttpRequestExecutorImpl.java:81) at com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl.executeGet( HttpRequestExecutorImpl.java:50) at com.atlassian.bitbucket.jenkins.internal.client.BitbucketRequestExecutor.makeGetRequest( BitbucketRequestExecutor.java:159) at com.atlassian.bitbucket.jenkins.internal.client.BitbucketRequestExecutor.makeGetRequest( BitbucketRequestExecutor.java:87) ... With regards to the "how do we do it"... Nothing special in fact. Using the Git SCM we just supply the "Repository URL" with an SSH URL like: ssh://git@bitbucket.work.ci/TEST/testing.git The credentials are basically not necessary for this since we're using the ssh/config to specify the id_rsa and the host is a known host. The creds are used for GUI logon only. // code placeholder bash-4.2$ more .ssh/config # BitBucket Host bitbucket.work.ci User git Hostname bitbucket.work.ci Port 7999 IdentityFile ~/.ssh/id_rsa With the Bitbucket server plugin there is no way of specifying the ssh URL so I'm assuming that the path is derived from the "Bitbucket Server Instance", "Project Name" and "Repository name". If that is internally used for a clone of a [https://|https:] .... URL, it will fail since we don't allow that. SSH only! Let me know if this is sufficient to make progress on this issue. Just to let you know that I'd be happy test it out any solution in case you need it. Finally, yes, I tested the "Test Connection" and it works. Regards Kent

          Hi Kent,

          Thanks for clarifying. To answer your question, yes we derive the clone URL from the provided HTTP address in the server config and the project and repo name. So in your case, even though the GitSCM is cloning by SSH, the plugin is still attempting it's own build task by HTTP, which your server is rejecting. The upcoming feature I mentioned (https://issues.jenkins-ci.org/browse/JENKINS-60492) will change the clone URL based on the credentials provided. We do not intend to choose our clone URL depending on other SCMs on a given project however, so you will need to use the same project for cloning (which uses the Git SCM under the hood anyway, and should have the same effect).

          Having SSH credentials in your `.ssh` config is interesting- the feature shipping in 2.0.1 will not have that feature but I feel it would be a good improvement.

          I will leave this ticket open for now, but will mark it as an improvement (as the behaviour you describe is expected). Thanks again

          Martin Henschke added a comment - Hi Kent, Thanks for clarifying. To answer your question, yes we derive the clone URL from the provided HTTP address in the server config and the project and repo name. So in your case, even though the GitSCM is cloning by SSH, the plugin is still attempting it's own build task by HTTP, which your server is rejecting. The upcoming feature I mentioned ( https://issues.jenkins-ci.org/browse/JENKINS-60492 ) will change the clone URL based on the credentials provided. We do not intend to choose our clone URL depending on other SCMs on a given project however, so you will need to use the same project for cloning (which uses the Git SCM under the hood anyway, and should have the same effect). Having SSH credentials in your `.ssh` config is interesting- the feature shipping in 2.0.1 will not have that feature but I feel it would be a good improvement. I will leave this ticket open for now, but will mark it as an improvement (as the behaviour you describe is expected). Thanks again

          Corey Olson added a comment -

          mhenschke_atlassian it looks like the linked ticket got closed, but I'm not sure that the feature got released.  Did it?

          Corey Olson added a comment - mhenschke_atlassian  it looks like the linked ticket got closed, but I'm not sure that the feature got released.  Did it?

          The feature in JENKINS-60492 got merged as PR #189 and released in 2.1.0; the plugin lets you configure sshCredentialsId for BitbucketSCM or BitbucketSCMSource, and if you do so, then it will clone the repository over SSH. However, I believe the Jenkins controller still needs to access the REST APIs of Bitbucket Server over HTTP.

          Kalle Niemitalo added a comment - The feature in JENKINS-60492 got merged as PR #189 and released in 2.1.0; the plugin lets you configure sshCredentialsId for BitbucketSCM or BitbucketSCMSource, and if you do so, then it will clone the repository over SSH. However, I believe the Jenkins controller still needs to access the REST APIs of Bitbucket Server over HTTP.

          That's correct. We require a HTTP URL for your server configuration as this is used for resolving the repository for other REST endpoints like repo name resolution, webhook registration etc. We're explicit in that requirement, asking for a base URL using HTTP, not a clone URL. This issue is specifically for using .ssh credentials, and I've updated the name and priority to reflect that.

          If you're experiencing some issue with SSH credentials not working or not appearing in your job config, could you please raise a separate ticket?

          Martin Henschke added a comment - That's correct. We require a HTTP URL for your server configuration as this is used for resolving the repository for other REST endpoints like repo name resolution, webhook registration etc. We're explicit in that requirement, asking for a base URL using HTTP, not a clone URL. This issue is specifically for using .ssh credentials, and I've updated the name and priority to reflect that. If you're experiencing some issue with SSH credentials not working or not appearing in your job config, could you please raise a separate ticket?

          Corey Olson added a comment -

          I was able to find the option to checkout the repo using SSH, which worked great.  Thanks!

          Corey Olson added a comment - I was able to find the option to checkout the repo using SSH, which worked great.  Thanks!

            Unassigned Unassigned
            kengra1 Kent Granström
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: