-
Bug
-
Resolution: Not A Defect
-
Major
-
None
-
Jenkins 2.303.3
git version 2.17.1
java.runtime.version 1.8.0_312
bitbucket 214.v2fd4234d0554
os.arch amd64
os.name Linux
os.version 5.4.0-1065-aws
Hello Team,
I have an issue that we started facing after Bitbucket stopped supporting account passwords to connect to the repo (last March).
Before March we never have faced any authentication issue before, and all our pipelines were able to connect and retrieve the code from Bitbucket without any issues.
Knowing that Bitbucket will stop supporting the account passwords, we have updated the existing Credential in Jenkins Credential store (lets name existing credential as sC1) with app password to connect to our bitbucket, and we have switched to that on the day of the cut-off.
Since that time we are facing the intermittent issue where we receive the below error
> git --version # timeout=10 > git --version # 'git version 2.17.1' using GIT_ASKPASS to set credentials hykeadmin user for all PIPELINES > git ls-remote -h – https://bitbucket.org/organization-x/hyke-cicd.git # timeout=10 Found match: refs/heads/feature/sonarqube-testing revision 185086485f0039595e94f6f69cf21f06b2776233 Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential BitBucketAdmin > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/com_promotion-service_squad5-dev@libs/hyke-devops-libs/.git # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://bitbucket.org/organization-x/hyke-cicd.git # timeout=10 Fetching without tags Fetching upstream changes from https://bitbucket.org/organization-x/hyke-cicd.git > git --version # timeout=10 > git --version # 'git version 2.17.1' using GIT_ASKPASS to set credentials hykeadmin user for all PIPELINES > git fetch --no-tags --progress – https://bitbucket.org/organization-x/hyke-cicd.git +refs/heads/:refs/remotes/origin/ # timeout=10 Checking out Revision 185086485f0039595e94f6f69cf21f06b2776233 (feature/sonarqube-testing) > git config core.sparsecheckout # timeout=10 > git checkout -f 185086485f0039595e94f6f69cf21f06b2776233 # timeout=10 Commit message: "add jacocoTestReport parameter" [Bitbucket] Notifying commit build result [Pipeline] Start of Pipeline [Pipeline] node Running on ProductionFleetCloud i-068f8ef3********* in /var/lib/jenkins/workspace/com_promotion-service_squad5-dev [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: NONE using credential BitBucketAdmin Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://bitbucket.org/organization-x/promotion-service.git > git init /var/lib/jenkins/workspace/com_promotion-service_squad5-dev # timeout=10 Fetching upstream changes from https://bitbucket.org/organization-x/promotion-service.git > git --version # timeout=10 > git --version # 'git version 2.17.1' using GIT_ASKPASS to set credentials hykeadmin user for all PIPELINES > git fetch --no-tags --progress – https://bitbucket.org/organization-x/promotion-service.git +refs/heads/squad5-dev:refs/remotes/origin/squad5-dev # timeout=10 ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --no-tags --progress – https://bitbucket.org/organization-x/promotion-service.git +refs/heads/squad5-dev:refs/remotes/origin/squad5-dev" returned status code 128: stdout: stderr: remote: Bitbucket Cloud recently stopped supporting account passwords for Git authentication. remote: See our community post for more details: https://atlassian.community/t5/x/x/ba-p/1948231 remote: App passwords are recommended for most use cases and can be created in your Personal settings: remote: https://bitbucket.org/account/settings/app-passwords/ fatal: Authentication failed for 'https://bitbucket.org/organization-x/promotion-service.git/'at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:847) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:376) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to i-068f8ef3edf1ef8e3 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1001) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143) at sun.reflect.GeneratedMethodAccessor1303.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129) at com.sun.proxy.$Proxy105.execute(Unknown Source) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1224) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ... 4 more [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] stage [Pipeline] { (Declarative: Post Actions) Error when executing always post condition: groovy.lang.MissingPropertyException: No such property: WORKSPACE for class: gradleBuild at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:458) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.getProperty(DefaultInvoker.java:39) at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20) at gradleBuild.call(gradleBuild.groovy:306) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:761) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:760) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:750) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executePostBuild(ModelInterpreter.groovy:728) at __cps.transform__(Native Method) at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66) at sun.reflect.GeneratedMethodAccessor857.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)[Pipeline] } [Pipeline] // stage [Pipeline] End of Pipeline ERROR: Error cloning remote repo 'origin' [Bitbucket] Notifying commit build result [Bitbucket] Build result notified Finished: FAILURE
After this we have created a new Jenkins credential (C2) which uses the same username and app password which we are using in all our pipelines. however even when doing so, we are still facing the same intermittent problem and the same error is being thrown from time to time, may be once a week or once every 2 weeks, noting that we run our pipelines multiple times per day.
Regarding our Jenkins setup, we are working with Master/slave Jenkins setup, where the Master is running on an on demand AWS EC2 instance, and all the slaves are on-spot fleet instances, i.e. the slave running the workloads changes regularly (multiple times a day).
Currently we are resolving the issue by running the pipeline again (without doing any change) and it runs fine the second or third time we trigger it.
We contacted Bitbucket support to check the issue, they suggested resetting the credentials store, which we did, but we are still getting the intermittent issue.
They were not able to help further, and hence I thought to check here how this can be resolved.
