Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59000

KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • kubernetes-plugin
    • None
    • Kubernetes v1.15.3
    • 1.18.2

      I've upgraded a Kubernetes cluster from v1.15.2 to v1.15.3 and started noticing kubernetes-plugin cannot provision any pods. I was able to rollback to fix the issue, roll forward again to cause it, and generally isolate it to the specific cluster version bump. I can change flip just the apiserver version back and forth to cause and fix the issue. I've tried both kubernetes-plugin v1.16.4 and v1.18.1 (both affected).

      What may be relevant is that Kubernetes v1.13.10, v1.14.6, and v1.15.3 was published today with CVE fixes for Go net/http vulnerabilities. Maybe one of those fixes interferes with kubernetes-plugin operation (that plugin code seems to concern the websocket and HTTP watch). I haven't confirmed the issue is present on Kubernetes v1.13.10 or v1.14.6, but suspect its on all the patched releases.

      Error in provisioning; agent=KubernetesSlave name: python-pod-0ppnb-qnvc7, template=PodTemplate{, name='python-pod-0ppnb', namespace='jenkins', label='python-pod', nodeUsageMode=EXCLUSIVE, containers=[ContainerTemplate{name='python-agent', image='quay.io/dghubble/python-agent:4462978780179489a5608c480d9c64f8cb61bc7f', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
      io.fabric8.kubernetes.client.KubernetesClientException: 
      	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
      	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
      	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
      	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
      	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatchConnectionManager.java

      https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#changelog-since-v1152

       

            csanchez Carlos Sanchez
            dghubble Dalton Hubble
            Votes:
            9 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: