Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55123

Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None

      Kubernetes plugin does not respect 'no_proxy' Java property, Environment Variable or Jenkins Proxy config from 'Jenkins->Manage Jenkins->Manage Plugins->Advanced Tab'.

       

      I have confirmed Kubernetes API access works from the console of the Jenkins server via 'curl' with appropriate proxy environment variables set.

       

      I have used the 'Validate Proxy' button from the 'Jenkins->Manage Jenkins->Manage Plugins->Advanced Tab' I get ' Failed to connect to https://xxxxxxxxxxxxxx.

      javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated' and confirmed the request never hits the proxy using proxy logs.

       

      If I configure the Kubernetes plugin slaves never start and if I use the 'Test Connection' button in the plugin config I get 'Error testing connection https://xxxxxxxxxxxxxx: java.io.IOException: Unexpected response code for CONNECT: 403' and have confirmed the request does hit the proxy using proxy logs.

       

          [JENKINS-55123] Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

          Viktor K added a comment -

          I have exactly the same problem..Could you please let us know what is the problem here?

          Viktor K added a comment - I have exactly the same problem..Could you please let us know what is the problem here?

          Steve Collins added a comment - - edited

          I also have this problem when the docker image has 'no_proxy' set in the image.

          I had to remove the no_proxy env var from the image to be able to set no_proxy using Env/Node/etc properties.

          This was not a good workaround as it pushed proxy info back to the user (100's of Jenkins) and not able to use as we hope in other contexts outside of Jenkins without passing in no_proxy env var.

          thanks. Plugin is really great!

           

          Steve Collins added a comment - - edited I also have this problem when the docker image has 'no_proxy' set in the image. I had to remove the no_proxy env var from the image to be able to set no_proxy using Env/Node/etc properties. This was not a good workaround as it pushed proxy info back to the user (100's of Jenkins) and not able to use as we hope in other contexts outside of Jenkins without passing in no_proxy env var. thanks. Plugin is really great!  

          Dax Games added a comment -

          I found this code that passes the master env vars related to proxy settings to the slave.

          The problem is the pod never starts because the plugin can't talk to the Kubernetes API because it ignores the no_proxy settings.

          Dax Games added a comment - I found this code that passes the master env vars related to proxy settings to the slave. The problem is the pod never starts because the plugin can't talk to the Kubernetes API because it ignores the no_proxy settings.

          If you are talking about the communication between Jenkins and Kubernetes API the following system properties are used to configure proxy

          • http.proxy
          • https.proxy
          • no.proxy
          • proxy.username
          • proxy.password

          https://github.com/fabric8io/kubernetes-client/blob/9e850ad769134566e32f8c41498e5e64ecf8b799/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L106

          You can see a test that is passing here https://github.com/jenkinsci/kubernetes-plugin/pull/477

          Carlos Sanchez added a comment - If you are talking about the communication between Jenkins and Kubernetes API the following system properties are used to configure proxy http.proxy https.proxy no.proxy proxy.username proxy.password https://github.com/fabric8io/kubernetes-client/blob/9e850ad769134566e32f8c41498e5e64ecf8b799/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L106 You can see a test that is passing here https://github.com/jenkinsci/kubernetes-plugin/pull/477

          Also see issue in kubernetes-client library https://github.com/fabric8io/kubernetes-client/issues/1335

          Carlos Sanchez added a comment - Also see issue in kubernetes-client library https://github.com/fabric8io/kubernetes-client/issues/1335

          Dax Games added a comment - - edited

          Yeah been there looked at that also.

          I am now running Jenkins using the following java properties:

          -Dhttp.proxy=http://x.x.x.x:xxxx -Dhttps.proxy=https://x.x.x.x:xxxx -Dno.Proxy=xxx -Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=xxxx -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=xxxx -Dhttp.nonProxyHosts=xxx
          

          Testing the connection from the Jenkins server running in Kubernetes through the Kubernetes plugin using the 'test connection' results in:

          Error testing connection https://kubernetes.default: java.io.IOException: Unexpected response code for CONNECT: 502
          

          and the following log entry on the proxy server:

          1557513166.881     67 192.168.1.73 TCP_TUNNEL/502 0 CONNECT kubernetes.default.svc:443 - FIRSTUP_PARENT/192.168.1.100 -
          

          Testing the connection from the Jenkins server running in Kubernetes through the 'Manage Jenkins->Manage Plugins->Advanced' tab using the 'validate proxy' button with the test URL of 'https://kubernetes.default.svc' results in the following expected failure since we are not authenticated:

          Failed to connect to https://kubernetes.default.svc.
          javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
          	at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
          	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(SSLProtocolSocketFactory.java:257)
          	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:115)
          	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:156)
          	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:714)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:394)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)
          

          From the command line of the same Jenkins server that has the exact same proxy settings defined in the java properties above set in environment variables. I can issue the following and retrieve a json object containing all pods in the namespace:

          curl $APISERVER/api/v1/namespaces/pd-dgames-test/pods/ --header "Authorization: Bearer $TOKEN" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          

          Kubernetes plugin spinning up a slave repeatedly gives me this:

          May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
          INFO: Excess workload after pending Kubernetes agents: 1
          May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
          INFO: Template for label pd-slave: Kubernetes Pod Template
          May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
          WARNING: Failed to count the # of live instances on Kubernetes
          java.io.IOException: Unexpected response code for CONNECT: 502
                  at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:389)
                  at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:211)
                  at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:152)
                  at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
                  at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
                  at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
                  at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                  at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                  at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
                  at okhttp3.RealCall.execute(RealCall.java:77)
                  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
                  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
                  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
                  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:193)
                  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:618)
                  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:68)
                  at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:505)
                  at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:458)
                  at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
                  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
                  at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
                  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
                  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
                  at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          

          Dax Games added a comment - - edited Yeah been there looked at that also. I am now running Jenkins using the following java properties: -Dhttp.proxy=http: //x.x.x.x:xxxx -Dhttps.proxy=https://x.x.x.x:xxxx -Dno.Proxy=xxx -Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=xxxx -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=xxxx -Dhttp.nonProxyHosts=xxx Testing the connection from the Jenkins server running in Kubernetes through the Kubernetes plugin using the 'test connection' results in: Error testing connection https: //kubernetes. default : java.io.IOException: Unexpected response code for CONNECT: 502 and the following log entry on the proxy server: 1557513166.881 67 192.168.1.73 TCP_TUNNEL/502 0 CONNECT kubernetes.default.svc:443 - FIRSTUP_PARENT/192.168.1.100 - Testing the connection from the Jenkins server running in Kubernetes through the 'Manage Jenkins->Manage Plugins->Advanced' tab using the 'validate proxy' button with the test URL of 'https://kubernetes.default.svc' results in the following expected failure since we are not authenticated: Failed to connect to https: //kubernetes. default .svc. javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450) at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(SSLProtocolSocketFactory.java:257) at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:115) at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:156) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:714) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:394) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330) From the command line of the same Jenkins server that has the exact same proxy settings defined in the java properties above set in environment variables. I can issue the following and retrieve a json object containing all pods in the namespace: curl $APISERVER /api/v1/namespaces/pd-dgames-test/pods/ --header "Authorization: Bearer $TOKEN " --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kubernetes plugin spinning up a slave repeatedly gives me this: May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision INFO: Excess workload after pending Kubernetes agents: 1 May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision INFO: Template for label pd-slave: Kubernetes Pod Template May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision WARNING: Failed to count the # of live instances on Kubernetes java.io.IOException: Unexpected response code for CONNECT: 502 at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:389) at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:211) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:152) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:193) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:618) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:68) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:505) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:458) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)

          so you don't want https://kubernetes.default.svc to go through the proxy, correct?
          do you have -Dno.Proxy=kubernetes.default.svc,somethingelse ?

          Carlos Sanchez added a comment - so you don't want https://kubernetes.default.svc to go through the proxy, correct? do you have -Dno.Proxy=kubernetes.default.svc,somethingelse ?

          Dax Games added a comment - - edited

          Yes that is in my -Dno.Proxy kubernetes.default.svc,... that starts Jenkins.

          APISERVER=https://kubernetes.default.svc in the curl command.

          Dax Games added a comment - - edited Yes that is in my -Dno.Proxy kubernetes.default.svc,... that starts Jenkins. APISERVER= https://kubernetes.default.svc in the curl command.

          davidkarlsen added a comment -

          what's strange also is that this only happens when declaring a yaml/yamlFile in a declarative pipeline, not if it is defined as a podTemplate on jenkins master.

          davidkarlsen added a comment - what's strange also is that this only happens when declaring a yaml/yamlFile in a declarative pipeline, not if it is defined as a podTemplate on jenkins master.

          Should be supported since 1.27.2 https://issues.jenkins.io/browse/JENKINS-63584 ?

          Allan BURDAJEWICZ added a comment - Should be supported since 1.27.2 https://issues.jenkins.io/browse/JENKINS-63584 ?

            Unassigned Unassigned
            daxgames Dax Games
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: