Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71788

After upgrade to 2.401.3 getting remote stack trace error when calling git branch:xxx url:xxx within withVault block

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • hashicorp-vault-plugin
    • None

      We are running the Jenkins image 2.401.3-lts on kubernetes 1.24.6

       

      This is what I see from the Jenkins pod logs

       

      2023-08-09 14:42:54.474+0000 [id=231]    INFO    hudson.slaves.NodeProvisioner#update: acr-sync-230809.0742-xkbj7-f8dqc provisioning successfully completed. We have now 2 computer(s)
2023-08-09 14:42:54.556+0000 [id=246]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes terraform/acr-sync-230809.0742-xkbj7-f8dqc
2023-08-09 14:42:56.969+0000 [id=246]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes terraform/acr-sync-230809.0742-xkbj7-f8dqc
2023-08-09 14:42:58.403+0000 [id=260]    INFO    com.bettercloud.vault.Vault#<init>: The NameSpace ssc/myproject/production has been bound to this Vault instance. Please keep this in mind when running operations.
2023-08-09 14:43:00.635+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent acr-sync-230809.0742-xkbj7-f8dqc
2023-08-09 14:43:00.672+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent terraform/acr-sync-230809.0742-xkbj7-f8dqc
2023-08-09 14:43:00.673+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer acr-sync-230809.0742-xkbj7-f8dqc
2023-08-09 14:43:00.675+0000 [id=257]    INFO    hudson.remoting.Request$2#run: Failed to send back a reply to the request RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2): hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@5b467799:acr-sync-230809.0742-xkbj7-f8dqc": channel is already closed
2023-08-09 14:43:00.682+0000 [id=19]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-19 for acr-sync-230809.0742-xkbj7-f8dqc terminated: java.nio.channels.ClosedChannelException
2023-08-09 14:43:00.683+0000 [id=19]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-19 for acr-sync-230809.0742-xkbj7-f8dqc terminated: java.nio.channels.ClosedChannelException
2023-08-09 14:43:01.163+0000 [id=246]    INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://github.enterprise.internal/api/v3'.

       

       

      Our Jenkins job is defined as follows 

      if (env.BRANCH_NAME == 'master' && env.JENKINS_URL == 'https://jenkins.prod/') {
    docker_registry= "prod.azurecr.io"
    prod_acr = "prid"
    dev_acr = "dev.azurecr.io"
    VAULT_BOT = 'vault_bot'
    VAULT_NAMESPACE = 'ssc/myproject/production'
    VAULT_GBAAS_PATH = 'kv/acrsync'
    WORK_DIR = "images-sync/scripts"
}else{
    return
}properties([
    parameters ([
        string(name: 'version', description: 'e.g 2104.rc / 2105.rc'),
        string(name: 'product', description: 'e.g myproject, web3, pa'),
    ])
])def now = new Date()
def label = 'acr-sync-' + now.format("yyMMdd.HHmm", TimeZone.getTimeZone('America/Los_Angeles'))def download_repo(branchName, targetDir, repository) {
    dir(targetDir){
        withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'git_bot_username'], [vaultKey: 'git_bot_token']]]]){
                git branch: branchName, url: "https://${git_bot_username}:${git_bot_token}@github.enterprise.internal/myproject/${repository}.git"
        }
    }
}def azLogin() {
    withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'AZURE_CLIENT_ID'], [vaultKey: 'AZURE_CLIENT_SECRET'], [vaultKey: 'AZURE_SUBSCRIPTION_ID'], [vaultKey: 'AZURE_TENANT_ID']]]]) {
        sh script: """
            az login --service-principal -u ${AZURE_CLIENT_ID} -p ${AZURE_CLIENT_SECRET} --tenant ${AZURE_TENANT_ID}
            az account set --subscription ${AZURE_SUBSCRIPTION_ID}
        """
    }
}podTemplate(label: label, yaml: """apiVersion: v1
kind: Pod
metadata:
  namespace: terraform
  labels:
    ${label}: yes
spec:
  imagePullSecrets:
    - name: acr-sp-secret
  containers:
    - name: terraform-base
      imagePullPolicy: Always
      image: "${docker_registry}/base-images/terraform:v4.8"
      command:
        - cat
      tty: true
""")
{
    node(label) {
        stage ('download repo') {
            download_repo("master", "images-sync", "images-sync")
        }
        stage('Push images'){
            container(name: 'terraform-base') {
                script {
                    azLogin()
                    dir (WORK_DIR) {
                        withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'AZURE_DEV_SP_CLIENT_ID'], [vaultKey: 'AZURE_DEV_SP_CLIENT_SECRET'], [vaultKey: 'slack_bot']]]]) {
                            sh """
                                ./sync-images.sh --prod-acr ${prod_acr} --dev-acr ${dev_acr} --username ${AZURE_DEV_SP_CLIENT_ID} --password ${AZURE_DEV_SP_CLIENT_SECRET} --version ${version} --slack-bot-key ${slack_bot} --product ${product}
                            """
                        }
                    }
                }
            }
        }
    }
}

      And when I run the job manually I see the following error

      The recommended git tool is: NONENo credentials specifiedCloning the remote Git repository[Pipeline] }[Pipeline] // withVault[Pipeline] }[Pipeline] // dir[Pipeline] }[Pipeline] // stage[Pipeline] }[Pipeline] // node[Pipeline] }[Pipeline] // podTemplate[Pipeline] End of PipelineAlso:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to acr-sync-230809.0742-xkbj7-f8dqc		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)		at hudson.remoting.Channel.call(Channel.java:1000)		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)		at java.base/java.lang.reflect.Method.invoke(Method.java:566)		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)		at com.sun.proxy.$Proxy181.execute(Unknown Source)		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)		at java.base/java.lang.Thread.run(Thread.java:829)Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 55964d83-f998-4ac1-bdfb-b2f008c16f13java.lang.IllegalStateException: Not running on the Jenkins controller JVM	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)	at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)	at java.base/java.io.PrintStream.write(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)	at java.base/java.io.OutputStreamWriter.flushBuffer(Unknown Source)	at java.base/java.io.PrintStream.newLine(Unknown Source)	at java.base/java.io.PrintStream.println(Unknown Source)	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:807)	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)	at hudson.remoting.UserRequest.perform(UserRequest.java:211)	at hudson.remoting.UserRequest.perform(UserRequest.java:54)	at hudson.remoting.Request$2.run(Request.java:377)	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)	at java.base/java.lang.Thread.run(Unknown Source)
Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected

GitHub has been notified of this commit’s build result

          [JENKINS-71788] After upgrade to 2.401.3 getting remote stack trace error when calling git branch:xxx url:xxx within withVault block

          Scott Watson created issue -
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

          Scott Watson added a comment -

          So the problem is caused when git branch is called within the withVault block.  If I move the git branch outside of the withVault block it works.  

          So what could be the issue with the git command running inside the withVault block?

          Scott Watson added a comment - So the problem is caused when git branch is called within the withVault block.  If I move the git branch outside of the withVault block it works.   So what could be the issue with the git command running inside the withVault block?
          Scott Watson made changes -
          Summary Original: After upgrade to 2.401.3 getting remote stack trace error New: After upgrade to 2.401.3 getting remote stack trace error when calling git branch:xxx url:xxx within withVault block
          Scott Watson made changes -
          Priority Original: Minor [ 4 ] New: Blocker [ 1 ]

          Anders Sigfridsson added a comment -

          We also have this problem, but in our case it happens when we call "tool" inside a withVault block, like this:

          withVault(
    configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
    vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
        [envVar: 'FOO', vaultKey: 'BAR'],
        [envVar: 'BAZ', vaultKey: 'QUX']]]]) {
    def toolPath = tool "govc-v0.24.0"
}

          Which results in:

           

          Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 8194a8d4-e114-420d-9b8b-32909be468c6
java.lang.IllegalStateException: Not running on the Jenkins controller JVM
	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
	at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
	at java.base/java.io.PrintStream.write(PrintStream.java:568)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313)
	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:111)
	at java.base/java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:178)
	at java.base/java.io.PrintStream.writeln(PrintStream.java:723)
	at java.base/java.io.PrintStream.println(PrintStream.java:1051)
	at hudson.Launcher.printCommandLine(Launcher.java:817)
	at hudson.Launcher.maskedPrintCommandLine(Launcher.java:832)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:977)
	at hudson.Launcher$ProcStarter.start(Launcher.java:509)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1398)
	at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1340)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

          I believe it might be related to a new version of credentials-binding: https://github.com/jenkinsci/credentials-binding-plugin/releases/tag/631.v861c06d062b_4

          It includes PR https://github.com/jenkinsci/credentials-binding-plugin/pull/260 and in the comments there we see the exact same exception:

          java.lang.IllegalStateException: Not running on the Jenkins controller JVM 
    at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
    at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)

          When we downgrade to credentials:1254.vb_96f366e7b_a_d the problem goes away.

           

           

          Anders Sigfridsson added a comment - We also have this problem, but in our case it happens when we call "tool" inside a withVault block, like this: withVault(     configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role' , vaultUrl: "https: //vault.example.com:8200" ],     vaultSecrets: [[path: "secret/apps/jenkins" , secretValues: [         [envVar: 'FOO' , vaultKey: 'BAR' ],         [envVar: 'BAZ' , vaultKey: 'QUX' ]]]]) {     def toolPath = tool "govc-v0.24.0" } Which results in:   Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 8194a8d4-e114-420d-9b8b-32909be468c6 java.lang.IllegalStateException: Not running on the Jenkins controller JVM at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46) at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57) at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43) at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75) at java.base/java.io.PrintStream.write(PrintStream.java:568) at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313) at java.base/sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:111) at java.base/java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:178) at java.base/java.io.PrintStream.writeln(PrintStream.java:723) at java.base/java.io.PrintStream.println(PrintStream.java:1051) at hudson.Launcher.printCommandLine(Launcher.java:817) at hudson.Launcher.maskedPrintCommandLine(Launcher.java:832) at hudson.Launcher$LocalLauncher.launch(Launcher.java:977) at hudson.Launcher$ProcStarter.start(Launcher.java:509) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1398) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1340) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:377) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang. Thread .run( Thread .java:833) I believe it might be related to a new version of credentials-binding: https://github.com/jenkinsci/credentials-binding-plugin/releases/tag/631.v861c06d062b_4 It includes PR https://github.com/jenkinsci/credentials-binding-plugin/pull/260 and in the comments there we see the exact same exception: java.lang.IllegalStateException: Not running on the Jenkins controller JVM at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46) at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57) When we downgrade to credentials:1254.vb_96f366e7b_a_d the problem goes away.    

          Scott Watson added a comment -

          ansig thanks for the info.  I actually tried to downgrade my credentials earlier but then Jenkins would not start and had to put it back.  Did you downgrade something else besides credentials?   I had used the advanced option and downloaded the hpi.

          Scott Watson added a comment - ansig thanks for the info.  I actually tried to downgrade my credentials earlier but then Jenkins would not start and had to put it back.  Did you downgrade something else besides credentials?   I had used the advanced option and downloaded the hpi.

          Anders Sigfridsson added a comment - - edited

          Sorry for the late reply rscottwatson !

          We downgraded:

          • credentials-binding:631.v861c06d062b_4 -> credentials-binding:604.vb_64480b_c56ca_
          • credentials:1271.v54b_1c2c6388a_ -> credentials:1254.vb_96f366e7b_a_d

          Then also had to downgrade the following since they depended on a newer credentials:

          • ansible:253.v4fe719ffdd8a_ -> ansible:240.vc26740a_625c0
          • email-ext:2.100 -> email-ext:2.99
          • ssh-slaves:2.916.vd17b_43357ce4 -> ssh-slaves:2.877.v365f5eb_a_b_eec
          • ssh-credentials:308.ve4497b_ccd8f4 -> ssh-credentials:305.v8f4381501156

          Anders Sigfridsson added a comment - - edited Sorry for the late reply rscottwatson ! We downgraded: credentials-binding:631.v861c06d062b_4 -> credentials-binding:604.vb_64480b_c56ca_ credentials:1271.v54b_1c2c6388a_ -> credentials:1254.vb_96f366e7b_a_d Then also had to downgrade the following since they depended on a newer credentials: ansible:253.v4fe719ffdd8a_ -> ansible:240.vc26740a_625c0 email-ext:2.100 -> email-ext:2.99 ssh-slaves:2.916.vd17b_43357ce4 -> ssh-slaves:2.877.v365f5eb_a_b_eec ssh-credentials:308.ve4497b_ccd8f4 -> ssh-credentials:305.v8f4381501156

          Scott Watson added a comment -

          ansig Thank you so much for taking the time to post your answer.  This has got me up and running again.  It has been a few very stressful days.  For anyone else reading this I also had to downgrade email-ext from version 2.100 to 2.99.

           

          THANK YOU!!

          Scott Watson added a comment - ansig Thank you so much for taking the time to post your answer.  This has got me up and running again.  It has been a few very stressful days.  For anyone else reading this I also had to downgrade email-ext from version 2.100 to 2.99.   THANK YOU!!

          Caleb added a comment - - edited

          I was also hitting this same issue on our instance and downgrading the email-ext plugin solved our issue too, thanks a million ansig and rscottwatson!

          Caleb added a comment - - edited I was also hitting this same issue on our instance and downgrading the email-ext plugin solved our issue too, thanks a million ansig and rscottwatson !

            jetersen Joseph Petersen
            rscottwatson Scott Watson
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: