Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45885

Recieving context deadline exceeded when Helm and Tiller are interact

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Running a pipeline job with 2 stages

      1. Helm dry run install of an image
      2. Helm install of an image

      I can see in the Jenkins master log that even when helm and tiller managed to connect I got the same error, that the latch was already released

      <

      SEVERE: onClose called but latch already finished. This indicates a bug in the kubernetes-plugin
      Jul 31, 2017 1:45:28 PM org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive
      WARNING: org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@303b5d9c; decorates hudson.Launcher$RemoteLauncher@561d1a44 on hudson.remoting.Channel@428191f0:Channel to /10.2.13.41 does not seem able to determine whether processes are alive or not
      ^[[45;38REXITCODE   0/home/jenkins # "ps" "-o" "pid=" "9999"
          1
         44
         48
         49
         50
         83
         88
      /home/jenkins # printf "EXITCODE %3d" $?; exit
      Jul 31, 2017 1:45:28 PM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose
      SEVERE: onClose called but latch already finished. This indicates a bug in the kubernetes-plugin
      Jul 31, 2017 1:45:28 PM org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive
      WARNING: org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@63f09283; decorates hudson.Launcher$RemoteLauncher@58a8b185 on hudson.remoting.Channel@428191f0:Channel to /10.2.13.41 does not seem able to determine whether processes are alive or not
      ^[[45;38REXITCODE   0/home/jenkins/workspace/marketing-site_chen2-master-JZLCNMYKV67ZVLSGU6NZ6TNEGT7XMATLTBJO5DINEQDOFPPSCFCA # printf "EXITCODE %3d" $?; exit
      Jul 31, 2017 1:45:29 PM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose
      SEVERE: onClose called but latch already finished. This indicates a bug in the kubernetes-plugin
      Jul 31, 2017 1:45:29 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      INFO: Terminating Kubernetes instance for slave jenkins-slave-tgrfl-9qbfr
      Jul 31, 2017 1:45:29 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      INFO: Terminated Kubernetes instance for slave jenkins-slave-tgrfl-9qbfr
      EXITCODE   0Terminated Kubernetes instance for slave jenkins-slave-tgrfl-9qbfr
      Jul 31, 2017 1:45:29 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      INFO: Disconnected computer jenkins-slave-tgrfl-9qbfr
      Jul 31, 2017 1:45:29 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      INFO: marketing-site/chen2-master #6 completed: FAILURE

       

       

      podTemplate(label: 'jenkins-pipeline', containers: [    
          containerTemplate(name: 'jnlp', image: 'jenkinsci/jnlp-slave:2.62', args: '${computer.jnlpmac} ${computer.name}', workingDir: '/home/jenkins', resourceRequestCpu: '200m', resourceLimitCpu: '800m', resourceRequestMemory: '256Mi', resourceLimitMemory: '2048Mi'),    
          containerTemplate(name: 'helm', image: 'coalmineadmin/k8s-helm:2.5.1', command: 'cat', ttyEnabled: true)
          
      ]{

      stage ('Test Cluster deployment') {

            container('helm') {

              // run helm chart linter
              pipeline.helmLint(chart_dir)

              // run dry-run helm chart installation
              pipeline.helmDeploy(
                dry_run       : true,
                name          : config.app.name,
                version_tag   : 'chen2-master-6bdd0e3',
                chart_dir     : chart_dir,
                replicas      : config.app.replicas,
                cpu           : config.app.cpu,
                memory        : config.app.memory
              )

            }
          }
            
         stage ('Deploy to Kubernetes') {
              container('helm') {
                // Deploy using Helm chart
                pipeline.helmDeploy(
                  dry_run       : false,
                  name          : config.app.name,
                  namespace     : config.app.namespace,
                  version_tag   : 'chen2-master-6bdd0e3',
                  chart_dir     : chart_dir,
                  replicas      : config.app.replicas,
                  cpu           : config.app.cpu,
                  memory        : config.app.memory
                )
                
                //  Run helm tests
                if (config.app.test) {
                  pipeline.helmTest(
                    name          : config.app.name
                  )
                }
              }
      }       
      }

       

       

        Attachments

          Activity

          Hide
          bjoernweiler Bjoern Weiler added a comment - - edited

          i seem to have the same issue

          Helm install/package runs smoothly on a jenkins slave containing a buildimage but after running some functionaltests the final delete of the helm deployment fails about 50% of the time.

          Jenkins Log:

          Aug 14, 2017 11:50:18 AM WARNING org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive
          
          org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2986bb3a; decorates hudson.Launcher$RemoteLauncher@513c8c76 on hudson.remoting.Channel@77313096:JNLP4-connect connection from 172.25.1.0/172.25.1.0:56754 does not seem able to determine whether processes are alive or not
          
          Aug 14, 2017 11:50:19 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose
          
          onClose called but latch already finished. This indicates a bug in the kubernetes-plugin
          
          Aug 14, 2017 11:50:19 AM WARNING org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive
          
          org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@52d2379d; decorates hudson.Launcher$RemoteLauncher@4e9688bb on hudson.remoting.Channel@77313096:JNLP4-connect connection from 172.25.1.0/172.25.1.0:56754 does not seem able to determine whether processes are alive or not
          
          Aug 14, 2017 11:50:19 AM SEVERE io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
          
          Exec Failure: HTTP:0. Message:No response
          java.io.EOFException
              at okio.RealBufferedSource.require(RealBufferedSource.java:59)
              at okio.RealBufferedSource.readByte(RealBufferedSource.java:72)
              at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
              at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
              at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:262)
              at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:201)
              at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
              at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          
          Aug 14, 2017 11:50:19 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose
          
          onClose called but latch already finished. This indicates a bug in the kubernetes-plugin
          
          Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
          
          Terminating Kubernetes instance for slave jenkins-slave-wlv4r-vnz8n
          
          Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
          
          Terminated Kubernetes instance for slave jenkins-slave-wlv4r-vnz8n
          
          Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
          
          Disconnected computer jenkins-slave-wlv4r-vnz8n
          
          Aug 14, 2017 11:50:19 AM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
          
          Computer.threadPoolForRemoting [#23] for jenkins-slave-wlv4r-vnz8n terminated
          java.nio.channels.ClosedChannelException
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
              at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
              at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
              at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
              at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
              at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
              at hudson.remoting.Channel.close(Channel.java:1304)
              at hudson.remoting.Channel.close(Channel.java:1272)
              at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
              at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
              at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
          
          Aug 14, 2017 11:50:19 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
          
          HelmJenkinsJob Service/master #1244 completed: FAILURE
          
          

          Pipeline error:

          11:50:17 + helm delete crusty-lynx
          
          11:50:19 java.io.EOFException
          11:50:19 	at okio.RealBufferedSource.require(RealBufferedSource.java:59)
          11:50:19 	at okio.RealBufferedSource.readByte(RealBufferedSource.java:72)
          11:50:19 	at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
          11:50:19 	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
          11:50:19 	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:262)
          11:50:19 	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:201)
          11:50:19 	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
          11:50:19 	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
          11:50:19 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          11:50:19 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          11:50:19 	at java.lang.Thread.run(Thread.java:745)
          11:50:19 Error: Get https://10.10.10.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: unexpected EOF
          
          Show
          bjoernweiler Bjoern Weiler added a comment - - edited i seem to have the same issue Helm install/package runs smoothly on a jenkins slave containing a buildimage but after running some functionaltests the final delete of the helm deployment fails about 50% of the time. Jenkins Log: Aug 14, 2017 11:50:18 AM WARNING org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2986bb3a; decorates hudson.Launcher$RemoteLauncher@513c8c76 on hudson.remoting.Channel@77313096:JNLP4-connect connection from 172.25.1.0/172.25.1.0:56754 does not seem able to determine whether processes are alive or not Aug 14, 2017 11:50:19 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose onClose called but latch already finished. This indicates a bug in the kubernetes-plugin Aug 14, 2017 11:50:19 AM WARNING org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@52d2379d; decorates hudson.Launcher$RemoteLauncher@4e9688bb on hudson.remoting.Channel@77313096:JNLP4-connect connection from 172.25.1.0/172.25.1.0:56754 does not seem able to determine whether processes are alive or not Aug 14, 2017 11:50:19 AM SEVERE io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure Exec Failure: HTTP:0. Message:No response java.io.EOFException     at okio.RealBufferedSource.require(RealBufferedSource.java:59)     at okio.RealBufferedSource.readByte(RealBufferedSource.java:72)     at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)     at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)     at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:262)     at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:201)     at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)     at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     at java.lang. Thread .run( Thread .java:745) Aug 14, 2017 11:50:19 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1$1 onClose onClose called but latch already finished. This indicates a bug in the kubernetes-plugin Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminating Kubernetes instance for slave jenkins-slave-wlv4r-vnz8n Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminated Kubernetes instance for slave jenkins-slave-wlv4r-vnz8n Aug 14, 2017 11:50:19 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Disconnected computer jenkins-slave-wlv4r-vnz8n Aug 14, 2017 11:50:19 AM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Computer.threadPoolForRemoting [#23] for jenkins-slave-wlv4r-vnz8n terminated java.nio.channels.ClosedChannelException     at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)     at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)     at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)     at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)     at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)     at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)     at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)     at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)     at hudson.remoting.Channel.close(Channel.java:1304)     at hudson.remoting.Channel.close(Channel.java:1272)     at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)     at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)     at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)     at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)     at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     at java.lang. Thread .run( Thread .java:745) Aug 14, 2017 11:50:19 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish HelmJenkinsJob Service/master #1244 completed: FAILURE Pipeline error: 11:50:17 + helm delete crusty-lynx 11:50:19 java.io.EOFException 11:50:19 at okio.RealBufferedSource.require(RealBufferedSource.java:59) 11:50:19 at okio.RealBufferedSource.readByte(RealBufferedSource.java:72) 11:50:19 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113) 11:50:19 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97) 11:50:19 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:262) 11:50:19 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:201) 11:50:19 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141) 11:50:19 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 11:50:19 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 11:50:19 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 11:50:19 at java.lang. Thread .run( Thread .java:745) 11:50:19 Error: Get https: //10.10.10.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: unexpected EOF
          Hide
          csanchez Carlos Sanchez added a comment -

          this is probably fixed in master already with JENKINS-40825 https://github.com/jenkinsci/kubernetes-plugin/pull/182

          Show
          csanchez Carlos Sanchez added a comment - this is probably fixed in master already with JENKINS-40825 https://github.com/jenkinsci/kubernetes-plugin/pull/182
          Hide
          lionelve Lionel Orellana added a comment -

          Carlos Sanchez Using the latest kubernetes-plugin build from master (0.13-SNAPSHOT) I don't see the latch error. However some of the accompanying WARNINGs are swamping the logs non-stop while the pod is running. 

          /home/jenkins $ printf "EXITCODE %3d" $?; exit
          Aug 24, 2017 11:47:46 AM org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive
          WARNING: org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5893078; decorates hudson.Launcher$RemoteLauncher@4404eecd on hudson.remoting.Channel@2bda8345:Channel to /172.23.152.1 does not seem able to determine whether processes are alive or not
          Aug 24, 2017 11:47:47 AM okhttp3.internal.platform.Platform log
          INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
          EXITCODE 0/home/jenkins $ "ps" "-o" "pid=" "9999"
          1
          7
          772
          778
          779
          780
          1431
          1437
          /home/jenkins $ printf "EXITCODE %3d" $?; exit
          Show
          lionelve Lionel Orellana added a comment - Carlos Sanchez Using the latest kubernetes-plugin build from master (0.13-SNAPSHOT) I don't see the latch error. However some of the accompanying WARNINGs are swamping the logs non-stop while the pod is running.  /home/jenkins $ printf "EXITCODE %3d" $?; exit Aug 24, 2017 11:47:46 AM org.jenkinsci.plugins.durabletask.ProcessLiveness isAlive WARNING: org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5893078; decorates hudson.Launcher$RemoteLauncher@4404eecd on hudson.remoting.Channel@2bda8345:Channel to /172.23.152.1 does not seem able to determine whether processes are alive or not Aug 24, 2017 11:47:47 AM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path? EXITCODE 0/home/jenkins $ "ps" "-o" "pid=" "9999" 1 7 772 778 779 780 1431 1437 /home/jenkins $ printf "EXITCODE %3d" $?; exit
          Hide
          csanchez Carlos Sanchez added a comment -

          that comes from another plugin, you would have to reduce the logging for org.jenkinsci.plugins.durabletask.ProcessLiveness
          This happens when using alpine images

          Show
          csanchez Carlos Sanchez added a comment - that comes from another plugin, you would have to reduce the logging for org.jenkinsci.plugins.durabletask.ProcessLiveness This happens when using alpine images

            People

            Assignee:
            magnayn magnayn
            Reporter:
            chenfli Chen Fliesher
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: