Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71788

After upgrade to 2.401.3 getting remote stack trace error when calling git branch:xxx url:xxx within withVault block

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • hashicorp-vault-plugin
    • None

      We are running the Jenkins image 2.401.3-lts on kubernetes 1.24.6

       

      This is what I see from the Jenkins pod logs

       

      2023-08-09 14:42:54.474+0000 [id=231]    INFO    hudson.slaves.NodeProvisioner#update: acr-sync-230809.0742-xkbj7-f8dqc provisioning successfully completed. We have now 2 computer(s)
      2023-08-09 14:42:54.556+0000 [id=246]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes terraform/acr-sync-230809.0742-xkbj7-f8dqc
      2023-08-09 14:42:56.969+0000 [id=246]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes terraform/acr-sync-230809.0742-xkbj7-f8dqc
      2023-08-09 14:42:58.403+0000 [id=260]    INFO    com.bettercloud.vault.Vault#<init>: The NameSpace ssc/myproject/production has been bound to this Vault instance. Please keep this in mind when running operations.
      2023-08-09 14:43:00.635+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent acr-sync-230809.0742-xkbj7-f8dqc
      2023-08-09 14:43:00.672+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent terraform/acr-sync-230809.0742-xkbj7-f8dqc
      2023-08-09 14:43:00.673+0000 [id=231]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer acr-sync-230809.0742-xkbj7-f8dqc
      2023-08-09 14:43:00.675+0000 [id=257]    INFO    hudson.remoting.Request$2#run: Failed to send back a reply to the request RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2): hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@5b467799:acr-sync-230809.0742-xkbj7-f8dqc": channel is already closed
      2023-08-09 14:43:00.682+0000 [id=19]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-19 for acr-sync-230809.0742-xkbj7-f8dqc terminated: java.nio.channels.ClosedChannelException
      2023-08-09 14:43:00.683+0000 [id=19]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-19 for acr-sync-230809.0742-xkbj7-f8dqc terminated: java.nio.channels.ClosedChannelException
      2023-08-09 14:43:01.163+0000 [id=246]    INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://github.enterprise.internal/api/v3'.
       

       

       

      Our Jenkins job is defined as follows 

      if (env.BRANCH_NAME == 'master' && env.JENKINS_URL == 'https://jenkins.prod/') {
          docker_registry= "prod.azurecr.io"
          prod_acr = "prid"
          dev_acr = "dev.azurecr.io"
          VAULT_BOT = 'vault_bot'
          VAULT_NAMESPACE = 'ssc/myproject/production'
          VAULT_GBAAS_PATH = 'kv/acrsync'
          WORK_DIR = "images-sync/scripts"
      }else{
          return
      }properties([
          parameters ([
              string(name: 'version', description: 'e.g 2104.rc / 2105.rc'),
              string(name: 'product', description: 'e.g myproject, web3, pa'),
          ])
      ])def now = new Date()
      def label = 'acr-sync-' + now.format("yyMMdd.HHmm", TimeZone.getTimeZone('America/Los_Angeles'))def download_repo(branchName, targetDir, repository) {
          dir(targetDir){
              withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'git_bot_username'], [vaultKey: 'git_bot_token']]]]){
                      git branch: branchName, url: "https://${git_bot_username}:${git_bot_token}@github.enterprise.internal/myproject/${repository}.git"
              }
          }
      }def azLogin() {
          withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'AZURE_CLIENT_ID'], [vaultKey: 'AZURE_CLIENT_SECRET'], [vaultKey: 'AZURE_SUBSCRIPTION_ID'], [vaultKey: 'AZURE_TENANT_ID']]]]) {
              sh script: """
                  az login --service-principal -u ${AZURE_CLIENT_ID} -p ${AZURE_CLIENT_SECRET} --tenant ${AZURE_TENANT_ID}
                  az account set --subscription ${AZURE_SUBSCRIPTION_ID}
              """
          }
      }podTemplate(label: label, yaml: """apiVersion: v1
      kind: Pod
      metadata:
        namespace: terraform
        labels:
          ${label}: yes
      spec:
        imagePullSecrets:
          - name: acr-sp-secret
        containers:
          - name: terraform-base
            imagePullPolicy: Always
            image: "${docker_registry}/base-images/terraform:v4.8"
            command:
              - cat
            tty: true
      """)
      {
          node(label) {
              stage ('download repo') {
                  download_repo("master", "images-sync", "images-sync")
              }
              stage('Push images'){
                  container(name: 'terraform-base') {
                      script {
                          azLogin()
                          dir (WORK_DIR) {
                              withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vautl.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'AZURE_DEV_SP_CLIENT_ID'], [vaultKey: 'AZURE_DEV_SP_CLIENT_SECRET'], [vaultKey: 'slack_bot']]]]) {
                                  sh """
                                      ./sync-images.sh --prod-acr ${prod_acr} --dev-acr ${dev_acr} --username ${AZURE_DEV_SP_CLIENT_ID} --password ${AZURE_DEV_SP_CLIENT_SECRET} --version ${version} --slack-bot-key ${slack_bot} --product ${product}
                                  """
                              }
                          }
                      }
                  }
              }
          }
      } 

      And when I run the job manually I see the following error

      The recommended git tool is: NONENo credentials specifiedCloning the remote Git repository[Pipeline] }[Pipeline] // withVault[Pipeline] }[Pipeline] // dir[Pipeline] }[Pipeline] // stage[Pipeline] }[Pipeline] // node[Pipeline] }[Pipeline] // podTemplate[Pipeline] End of PipelineAlso:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to acr-sync-230809.0742-xkbj7-f8dqc		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)		at hudson.remoting.Channel.call(Channel.java:1000)		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)		at java.base/java.lang.reflect.Method.invoke(Method.java:566)		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)		at com.sun.proxy.$Proxy181.execute(Unknown Source)		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)		at java.base/java.lang.Thread.run(Thread.java:829)Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 55964d83-f998-4ac1-bdfb-b2f008c16f13java.lang.IllegalStateException: Not running on the Jenkins controller JVM	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)	at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)	at java.base/java.io.PrintStream.write(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)	at java.base/java.io.OutputStreamWriter.flushBuffer(Unknown Source)	at java.base/java.io.PrintStream.newLine(Unknown Source)	at java.base/java.io.PrintStream.println(Unknown Source)	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:807)	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)	at hudson.remoting.UserRequest.perform(UserRequest.java:211)	at hudson.remoting.UserRequest.perform(UserRequest.java:54)	at hudson.remoting.Request$2.run(Request.java:377)	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)	at java.base/java.lang.Thread.run(Unknown Source)
      Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected
      
      GitHub has been notified of this commit’s build result 

            jetersen Joseph Petersen
            rscottwatson Scott Watson
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: