Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59000

KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Kubernetes v1.15.3
    • Similar Issues:
    • Released As:
      1.18.2

      Description

      I've upgraded a Kubernetes cluster from v1.15.2 to v1.15.3 and started noticing kubernetes-plugin cannot provision any pods. I was able to rollback to fix the issue, roll forward again to cause it, and generally isolate it to the specific cluster version bump. I can change flip just the apiserver version back and forth to cause and fix the issue. I've tried both kubernetes-plugin v1.16.4 and v1.18.1 (both affected).

      What may be relevant is that Kubernetes v1.13.10, v1.14.6, and v1.15.3 was published today with CVE fixes for Go net/http vulnerabilities. Maybe one of those fixes interferes with kubernetes-plugin operation (that plugin code seems to concern the websocket and HTTP watch). I haven't confirmed the issue is present on Kubernetes v1.13.10 or v1.14.6, but suspect its on all the patched releases.

      Error in provisioning; agent=KubernetesSlave name: python-pod-0ppnb-qnvc7, template=PodTemplate{, name='python-pod-0ppnb', namespace='jenkins', label='python-pod', nodeUsageMode=EXCLUSIVE, containers=[ContainerTemplate{name='python-agent', image='quay.io/dghubble/python-agent:4462978780179489a5608c480d9c64f8cb61bc7f', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
      io.fabric8.kubernetes.client.KubernetesClientException: 
      	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
      	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
      	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
      	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
      	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatchConnectionManager.java

      https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#changelog-since-v1152

       

        Attachments

          Issue Links

            Activity

            Show
            csanchez Carlos Sanchez added a comment - PR at https://github.com/jenkinsci/kubernetes-plugin/pull/582
            Hide
            asidhu11702 Ahuhu uhuhuh added a comment -

            Hi Carlos, what is this pull request going to do for us? The work around mentioned in the thread currently works for us which is https://kubernetes.default:443 in the plugin.

            Show
            asidhu11702 Ahuhu uhuhuh added a comment - Hi Carlos, what is this pull request going to do for us? The work around mentioned in the thread currently works for us which is https://kubernetes.default:443  in the plugin.
            Hide
            csanchez Carlos Sanchez added a comment -

            The pr avoids the need for the workaround

            Show
            csanchez Carlos Sanchez added a comment - The pr avoids the need for the workaround
            Hide
            asidhu11702 Ahuhu uhuhuh added a comment -

            carlos is the problem fixed? can we download the latest plugin that has the problems fixed?

            Show
            asidhu11702 Ahuhu uhuhuh added a comment - carlos is the problem fixed? can we download the latest plugin that has the problems fixed?
            Hide
            kmushegi Kote Mushegiani added a comment -

            Was having the same issue this morning, upgrading the plug-in to the most recent version (1.18.3) fixed the issue and doesn't require the port workaround!

            Show
            kmushegi Kote Mushegiani added a comment - Was having the same issue this morning, upgrading the plug-in to the most recent version (1.18.3) fixed the issue and doesn't require the port workaround!

              People

              Assignee:
              csanchez Carlos Sanchez
              Reporter:
              dghubble Dalton Hubble
              Votes:
              9 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: