Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39695

Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • kubernetes-plugin
    • None
    • Jenkins: 2.19.2
      kubernetes-plugin: 0.9

      Hi,
      I'm trying the job example, but the job fails on the shell step:

      [Pipeline] container
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Build a Go project)
      [Pipeline] sh
      [k8s-test2] Running shell script
      Executing shell script inside container [golang] of pod [jenkins-slave-7be4b22b164184]
      java.net.UnknownHostException: kubernetes.default.svc: unknown error
      	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
      	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
      	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
      	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
      	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
      	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
      	at com.squareup.okhttp.Dns$1.lookup(Dns.java:39)
      	at com.squareup.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175)
      	at com.squareup.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141)
      	at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83)
      	at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174)
      	at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
      	at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
      	at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
      	at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
      	at com.squareup.okhttp.Call.getResponse(Call.java:286)
      	at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
      	at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
      	at com.squareup.okhttp.Call.access$100(Call.java:35)
      	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:171)
      	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // container
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      

      But kube-dns is up and running :

      @kubectl run -i --tty busybox --image=busybox --generator="run-pod/v1"
      Waiting for pod default/busybox to be running, status is Pending, pod ready: false
      
      Hit enter for command prompt
      
      / # nslookup kubernetes.default.svc
      Server:    172.18.200.10
      Address 1: 172.18.200.10 kube-dns.kube-system.svc.cluster.local
      
      Name:      kubernetes.default.svc
      Address 1: 172.18.200.1 kubernetes.default.svc.cluster.local
      

      Any ideas?

          [JENKINS-39695] Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error

          Prabhu Inbarajan added a comment - - edited

          iocanel - Can you please clarify the comment , my setup is somewhat similar. I am running my jenkins master in docker swarm, but have configured it to run my slave workloads on remote kubernetes.
          i have looked at the kubernetes-plugin code somewhat superficially , but dont see how the stack trace above correlates with any logic from the kubernetes-client plugin. Interestingly enough, this problem seems to impact all the shell commands in pipeline DSL , the other jvm calls seems okay.

          https://github.com/jenkinsci/kubernetes-plugin/blob/52c0af572ef091399fa76657be34471c033d1464/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L92
          
          

          " if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes."
          what configuration are you referring to (or) perhaps you are referring to the usage of the kubernetes-client within the kubernetes-plugin?

          i tried setting the kubernetes_master (env) or kubernetes.master (system property) for jenkins master - but there is no change to the behavior

          Prabhu Inbarajan added a comment - - edited iocanel - Can you please clarify the comment , my setup is somewhat similar. I am running my jenkins master in docker swarm, but have configured it to run my slave workloads on remote kubernetes. i have looked at the kubernetes-plugin code somewhat superficially , but dont see how the stack trace above correlates with any logic from the kubernetes-client plugin. Interestingly enough, this problem seems to impact all the shell commands in pipeline DSL , the other jvm calls seems okay. https: //github.com/jenkinsci/kubernetes-plugin/blob/52c0af572ef091399fa76657be34471c033d1464/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L92 " if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes." what configuration are you referring to (or) perhaps you are referring to the usage of the kubernetes-client within the kubernetes-plugin? i tried setting the kubernetes_master (env) or kubernetes.master (system property) for jenkins master - but there is no change to the behavior

          prabhuinbarajan Thanks for investigating this, your feedback is really helpful.

          It seems that the "containaier" exec part is using a default instance of the client, rather than obtaining one through the Kubernetes Cloud.
          And this is why it ignores your change. I'll fix that asap.

          Thanks again!

          Ioannis Canellos added a comment - prabhuinbarajan Thanks for investigating this, your feedback is really helpful. It seems that the "containaier" exec part is using a default instance of the client, rather than obtaining one through the Kubernetes Cloud. And this is why it ignores your change. I'll fix that asap. Thanks again!

          Ioannis Canellos added a comment - - edited

          It's seems that its already fixed (yesterday).

          Ioannis Canellos added a comment - - edited It's seems that its already fixed (yesterday).

          Prabhu Inbarajan added a comment - - edited

          Thanks iocanel . That was super fast. Appreciate the quick turn around.

          Can you point out where the fix is ? seems like it is in the kubernetes-plugin? if it is in a branch , I would be happy to test it out. I looked at the commit histories , seems like this is the one.

          https://github.com/jenkinsci/kubernetes-plugin/commit/2be70ba43db9c00ab872e8d2c1eac7db27f084e0
          

          Thanks
          prabhu

          Prabhu Inbarajan added a comment - - edited Thanks iocanel . That was super fast. Appreciate the quick turn around. Can you point out where the fix is ? seems like it is in the kubernetes-plugin? if it is in a branch , I would be happy to test it out. I looked at the commit histories , seems like this is the one. https: //github.com/jenkinsci/kubernetes-plugin/commit/2be70ba43db9c00ab872e8d2c1eac7db27f084e0 Thanks prabhu

          It was merged to master! So if you could try out the latest master that would be great!

          Ioannis Canellos added a comment - It was merged to master! So if you could try out the latest master that would be great!

          the fix works as advertised. i dont see the original issue anymore. however, a couple of observations - please validate:
          1. now each container step is required to declare which cloud it needs to be launched in. would be great to have this inherit this info from the pod template, but seems like the design challenge is each pod templates can be inherited and each pod template can bind to a cloud, we cant derive the intent clean.
          2. if the containerstep fails to declare the cloud, that results in a null pointer exception, we may need to handle it in a clean way.

          test snippet below for reference (modified slightly from the original for build time):
          Note:
          1. that my jenkins instance runs from docker swarm , where as the slave workloads are scheduled on minikube
          2. built with latest from master (0.10-SNAPSHOT)

          podTemplate(
               cloud: 'minikube',
               label: 'docker',
               containers: [
                  //containerTemplate(name: 'jnlp', image: 'jenkinsci/jnlp-slave:2.62-alpine', args: '${computer.jnlpmac} ${computer.name}'),
                  containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
                  containerTemplate(name: 'golang', image: 'golang:1.6.3-alpine', ttyEnabled: true, command: 'cat')
                ],
                volumes: [secretVolume(secretName: 'shared-secret', mountPath: '/etc/shared-secrets')],
                ) {
                  node ('docker') {
                      stage 'Get a Maven project'
                      git 'https://github.com/pgrimard/spring-boot-hello-world.git'
                      container(name:'maven', cloud:'minikube') {
                          stage 'Build a Maven project'
                          try {
                          sh 'mvn clean install'
                          }catch (Exception ex) {
                              ex.printStackTrace()
                          }
                      }
              
                      stage 'Get a Golang project'
                      git url: 'https://github.com/golang/example.git'
                      container(name:'golang', cloud:'minikube') {
                          stage 'Build a Go project'
                          sh """
                          mkdir -p /go/src/github.com/golang/
                          ln -s `pwd` /go/src/github.com/golang/example
                          export GOPATH=/go
                          cd /go/src/github.com/golang/example/hello && go build
                          """
                      }
                  }   
          }
          

          Prabhu Inbarajan added a comment - the fix works as advertised. i dont see the original issue anymore. however, a couple of observations - please validate: 1. now each container step is required to declare which cloud it needs to be launched in. would be great to have this inherit this info from the pod template, but seems like the design challenge is each pod templates can be inherited and each pod template can bind to a cloud, we cant derive the intent clean. 2. if the containerstep fails to declare the cloud, that results in a null pointer exception, we may need to handle it in a clean way. test snippet below for reference (modified slightly from the original for build time): Note: 1. that my jenkins instance runs from docker swarm , where as the slave workloads are scheduled on minikube 2. built with latest from master (0.10-SNAPSHOT) podTemplate( cloud: 'minikube' , label: 'docker' , containers: [ //containerTemplate(name: 'jnlp' , image: 'jenkinsci/jnlp-slave:2.62-alpine' , args: '${computer.jnlpmac} ${computer.name}' ), containerTemplate(name: 'maven' , image: 'maven:3.3.9-jdk-8-alpine' , ttyEnabled: true , command: 'cat' ), containerTemplate(name: 'golang' , image: 'golang:1.6.3-alpine' , ttyEnabled: true , command: 'cat' ) ], volumes: [secretVolume(secretName: 'shared-secret' , mountPath: '/etc/shared-secrets' )], ) { node ( 'docker' ) { stage 'Get a Maven project' git 'https: //github.com/pgrimard/spring-boot-hello-world.git' container(name: 'maven' , cloud: 'minikube' ) { stage 'Build a Maven project' try { sh 'mvn clean install' } catch (Exception ex) { ex.printStackTrace() } } stage 'Get a Golang project' git url: 'https: //github.com/golang/example.git' container(name: 'golang' , cloud: 'minikube' ) { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/golang/ ln -s `pwd` /go/src/github.com/golang/example export GOPATH=/go cd /go/src/github.com/golang/example/hello && go build """ } } }

          Prabhu Inbarajan , can you provide an example for " each container step is required to declare which cloud it needs to be launched in".

          Cause, I am getting NPE and have tried different ways do declare cloud for container, but nothing helps.
          If I am downgrading to version 0,9, I am getting an issue from the ticket.
          (My Jenkins is outside the cloud)

          Aleksejs Cibinogins added a comment - Prabhu Inbarajan , can you provide an example for " each container step is required to declare which cloud it needs to be launched in". Cause, I am getting NPE and have tried different ways do declare cloud for container, but nothing helps. If I am downgrading to version 0,9, I am getting an issue from the ticket. (My Jenkins is outside the cloud)

          ragdaj -
          please see the comment above for the code snippet in context:
          I refer to the container step as follows, where each container (step) needs to be bound to the cloud. the full snippet is posted above. the NPE is very likely because the containerstep (https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerStepExecution.java#L51) the KubernetesCloud object is initialized from the ContainerStep's cloud name. Please let me know if this is unclear and i will add more.

          container(name:'golang', cloud:'minikube')

          { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/golang/ ln -s `pwd` /go/src/github.com/golang/example export GOPATH=/go cd /go/src/github.com/golang/example/hello && go build """ }

          Prabhu Inbarajan added a comment - ragdaj - please see the comment above for the code snippet in context: I refer to the container step as follows, where each container (step) needs to be bound to the cloud. the full snippet is posted above. the NPE is very likely because the containerstep ( https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerStepExecution.java#L51 ) the KubernetesCloud object is initialized from the ContainerStep's cloud name. Please let me know if this is unclear and i will add more. container(name:'golang', cloud:'minikube') { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/golang/ ln -s `pwd` /go/src/github.com/golang/example export GOPATH=/go cd /go/src/github.com/golang/example/hello && go build """ }

          Missed that, thank you. Yeah, it works that way.

          Aleksejs Cibinogins added a comment - Missed that, thank you. Yeah, it works that way.

          this should be fixed already

          Carlos Sanchez added a comment - this should be fixed already

            csanchez Carlos Sanchez
            sdouche Sébastien Douche
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: