Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45910

Template namespace is ignored when deleting pod

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.60.2
      Kubernetes plugin 0.12
      Kubernetes 1.7.2
    • Similar Issues:

      Description

      We are dynamically creating and destroying build slaves and they have now started to exit in a Error state.

      Although the build job completes successfully it seems that the slave tries to reconnect.

      Log from slave:

      Aug 01, 2017 1:37:46 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Terminated
      Aug 01, 2017 1:37:56 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
      INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@863f8a3
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up slave: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins-service]
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: jenkins-service
        Agent port:    50000
        Identity:      51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: 51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      INFO: [JNLP4-connect connection to jenkins-service/10.251.248.121:50000] Local headers refused by remote: Unknown client name: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP4-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:375)
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: java.nio.channels.ClosedChannelException
      		... 7 more
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Server reports protocol JNLP4-plaintext not supported, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP3-connect is not enabled, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP2-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP2-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: The server rejected the connection: None of the protocols were accepted
      java.lang.Exception: The server rejected the connection: None of the protocols were accepted
      	at hudson.remoting.Engine.onConnectionRejected(Engine.java:484)
      	at hudson.remoting.Engine.innerRun(Engine.java:448)
      	at hudson.remoting.Engine.run(Engine.java:287)
      

      Log from master:

      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminating Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Disconnected computer jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
      
      Computer.threadPoolForRemoting [#324] for jenkins-build-slave-qvdpn terminated
      java.nio.channels.ClosedChannelException
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
      	at hudson.remoting.Channel.close(Channel.java:1295)
      	at hudson.remoting.Channel.close(Channel.java:1263)
      	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
      	at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
      	at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 3:37:46 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      
      api-gateway #269 completed: SUCCESS
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP4-connect connection #145 from 10.248.2.34/10.248.2.34:34644
      
      Aug 01, 2017 3:37:58 PM INFO org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      
      [JNLP4-connect connection from 10.248.2.34/10.248.2.34:34644] Refusing headers from remote: Unknown client name: jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP2-connect connection #146 from /10.248.2.34:34646
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP-connect connection #147 from /10.248.2.34:34648
      

      From kubectl describe

      State:		Terminated
            Reason:		Error
            Exit Code:	255
      

        Attachments

        1. first-slave-describe
          4 kB
        2. first-slave-log
          7 kB
        3. kubernetes-plugin.log
          17 kB
        4. kubernetes-plugin.log
          7 kB
        5. okhttp3-log-recorder.png
          okhttp3-log-recorder.png
          25 kB
        6. second-slave-describe
          4 kB
        7. second-slave-log
          8 kB

          Issue Links

            Activity

            Hide
            juhtie01 Juha Tiensyrjä added a comment -

            We seem to be exhibiting the same problem. I can reproduce it with this simple Pipeline script:

            node('jnlp') {
                sh("echo 'foo'")
            }

            Jenkins master and Kubernetes plugin versions are the same as in the report, but the cluster is running 1.5.6.

            Show
            juhtie01 Juha Tiensyrjä added a comment - We seem to be exhibiting the same problem. I can reproduce it with this simple Pipeline script: node( 'jnlp' ) { sh( "echo 'foo' " ) } Jenkins master and Kubernetes plugin versions are the same as in the report, but the cluster is running 1.5.6.
            Hide
            tsan Tyler Sanders added a comment -

            Same problem here in my slave logs.  The slave spins up and executes the job fine, but then goes into an "Error" state when finished.

            Jenkins 2.67
            Kubernetes plugin 0.12 
            Kubernetes 1.7.2

            Show
            tsan Tyler Sanders added a comment - Same problem here in my slave logs.  The slave spins up and executes the job fine, but then goes into an "Error" state when finished. Jenkins 2.67 Kubernetes plugin 0.12  Kubernetes 1.7.2
            Hide
            vnikitov Vladislav Trayanov added a comment -

            Same problem here. 

            Jenkins 2.60.2 
            Kubernetes plugin 0.12 
            Kubernetes 1.7.2

            Show
            vnikitov Vladislav Trayanov added a comment - Same problem here.  Jenkins 2.60.2  Kubernetes plugin 0.12  Kubernetes 1.7.2
            Hide
            csanchez Carlos Sanchez added a comment -

            The log says

            Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
            Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn
            

            which should mean the pod is deleted

            Would need the debug logs for plugin and okhttp3 as described in https://github.com/jenkinsci/kubernetes-plugin#debugging and the pod events from describe, to check whether the pod is not being terminated or if it takes a while to terminate and in that meantime it tries to reconnect

            Show
            csanchez Carlos Sanchez added a comment - The log says Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn which should mean the pod is deleted Would need the debug logs for plugin and okhttp3 as described in https://github.com/jenkinsci/kubernetes-plugin#debugging and the pod events from describe, to check whether the pod is not being terminated or if it takes a while to terminate and in that meantime it tries to reconnect
            Hide
            csanchez Carlos Sanchez added a comment -

            Are pods deleted by jenkins or are they leftover ?
            The only theory I have is that in your cluster the pod deletion takes longer and before the pod is actually deleted it still has time to reconnect to jenkins

            Show
            csanchez Carlos Sanchez added a comment - Are pods deleted by jenkins or are they leftover ? The only theory I have is that in your cluster the pod deletion takes longer and before the pod is actually deleted it still has time to reconnect to jenkins
            Hide
            h_rut hugo rut added a comment -

            Same problem here, slave executes the job, then goes into an "Error" state.

            Jenkins 2.67
            Kubernetes plugin 0.12 
            Kubernetes 1.6.7

            Show
            h_rut hugo rut added a comment - Same problem here, slave executes the job, then goes into an "Error" state. Jenkins 2.67 Kubernetes plugin 0.12  Kubernetes 1.6.7
            Hide
            vikgun Viktor Gunnarson added a comment -

            After debugging a little bit more it seems as the kubernetes-plugin actually creates two slave pods that fails.

            First it creates a pod that fails fast (see attachment first-slave-log and first-slave-describe) and then tries to create a new pod. This second pod successfully completes the job but then tries to reconnect to the master and then fails (see attachment second-slave-log and second-slave-describe).

            The job started at 2017-08-07 13:01:10 UTC and finished at 2017-08-07 13:04:28 UTC.

            The second slave logs:

            Aug 07, 2017 1:04:29 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Terminated
            

            but then logs:

            Aug 07, 2017 1:04:39 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
            INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@43361483
            

            I have attached the kubernetes-plugin.log but I was unable to get any output from the okhttp log recorder, please advise (screenshot of okhttp3-log-recorder.png).

            Show
            vikgun Viktor Gunnarson added a comment - After debugging a little bit more it seems as the kubernetes-plugin actually creates two slave pods that fails. First it creates a pod that fails fast (see attachment first-slave-log and first-slave-describe ) and then tries to create a new pod. This second pod successfully completes the job but then tries to reconnect to the master and then fails (see attachment second-slave-log and second-slave-describe ). The job started at 2017-08-07 13:01:10 UTC and finished at 2017-08-07 13:04:28 UTC. The second slave logs: Aug 07, 2017 1:04:29 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated but then logs: Aug 07, 2017 1:04:39 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@43361483 I have attached the kubernetes-plugin.log but I was unable to get any output from the okhttp log recorder, please advise (screenshot of okhttp3-log-recorder.png ).
            Hide
            vikgun Viktor Gunnarson added a comment - - edited

            The pods are never deleted, they are left in state "Terminated" and reason "Error".

            Our work around has been a small script that deletes any jenkins-slave pod in error state every 30 seconds.

            Show
            vikgun Viktor Gunnarson added a comment - - edited The pods are never deleted, they are left in state "Terminated" and reason "Error". Our work around has been a small script that deletes any jenkins-slave pod in error state every 30 seconds.
            Hide
            mattdoran Matt Doran added a comment -

            We have just started seeing this too.  I'm sorry that I'm not sure whether this aligns with an update to the plugin.

            We see the same sort of logs, and then the pod dying with 255 and hanging around.

            This hasn't been a problem recently.

            We have some slave images that don't see this error problem.  We see similar reconnection errors, but they shutdown properly.   I'm not sure what else I can do to diagnose.

            This is a high-impact bug.

            Show
            mattdoran Matt Doran added a comment - We have just started seeing this too.  I'm sorry that I'm not sure whether this aligns with an update to the plugin. We see the same sort of logs, and then the pod dying with 255 and hanging around. This hasn't been a problem recently. We have some slave images that don't see this error problem.  We see similar reconnection errors, but they shutdown properly.   I'm not sure what else I can do to diagnose. This is a high-impact bug.
            Hide
            urmas Urmas Talimaa added a comment -

            We also started seeing this problem after upgrade to 0.12, the issue is not present in 0.11 (we have downgraded for the time being).

            Show
            urmas Urmas Talimaa added a comment - We also started seeing this problem after upgrade to 0.12, the issue is not present in 0.11 (we have downgraded for the time being).
            Hide
            zmeggyesi Zalan Meggyesi added a comment -

            I ran into the same issue on my cluster. For the time being, I've remained on 0.12 and configured the recommended log recorders, see if I can pick up anything there.

            What is strange is that for me, the issue only seems to affect pods that have service containers attached to them. I have several pod templates, and the ones that are only the jnlp container finish properly and are destroyed, but the ones that have two support containers running services remain errored out, cluttering the cluster.

            Show
            zmeggyesi Zalan Meggyesi added a comment - I ran into the same issue on my cluster. For the time being, I've remained on 0.12 and configured the recommended log recorders, see if I can pick up anything there. What is strange is that for me, the issue only seems to affect pods that have service containers attached to them. I have several pod templates, and the ones that are only the jnlp container finish properly and are destroyed, but the ones that have two support containers running services remain errored out, cluttering the cluster.
            Hide
            jcamou jota camou added a comment -

            Same problem with the following environment:

            Jenkins: 2.73

            Kubernetes Plugin versions: 0.11 and 0.12

            Kubernetes: 1.7.3

            I'm running two different Jenkins instances in the same cluster, only one seems to have the problem.

            Show
            jcamou jota camou added a comment - Same problem with the following environment: Jenkins: 2.73 Kubernetes Plugin versions: 0.11 and 0.12 Kubernetes: 1.7.3 I'm running two different Jenkins instances in the same cluster, only one seems to have the problem.
            Hide
            csanchez Carlos Sanchez added a comment -

            What you'd need to check on the master logs is if you are getting a Terminated Kubernetes instance for slave... line or any error.

            Show
            csanchez Carlos Sanchez added a comment - What you'd need to check on the master logs is if you are getting a Terminated Kubernetes instance for slave... line or any error.
            Hide
            zmeggyesi Zalan Meggyesi added a comment -

            I am, yes. I'm attaching the relevant piece of the log (pod name redacted). Disconnecting the pod takes place immediately, then the pod terminates, then waits around to be re-scheduled before getting the exception from the master.

            kubernetes-plugin.log

            Show
            zmeggyesi Zalan Meggyesi added a comment - I am, yes. I'm attaching the relevant piece of the log (pod name redacted). Disconnecting the pod takes place immediately, then the pod terminates, then waits around to be re-scheduled before getting the exception from the master. kubernetes-plugin.log
            Hide
            csanchez Carlos Sanchez added a comment -

            "then the pod terminates, then waits around" is the pod terminating in kubernetes? because if it is how can it wait around?

            Show
            csanchez Carlos Sanchez added a comment - "then the pod terminates, then waits around" is the pod terminating in kubernetes? because if it is how can it wait around?
            Hide
            zmeggyesi Zalan Meggyesi added a comment -

            No, as far as I can tell, when it receives the termination signal in these circumstances, it remains as "Running" in kubectl's output, before switching to "Error" after the exception. The logs, however, still state that the pod was terminated (line 4).

            If you can put out an hpi with enhanced logging, I could install it in my cluster to tell you more.

            Show
            zmeggyesi Zalan Meggyesi added a comment - No, as far as I can tell, when it receives the termination signal in these circumstances, it remains as "Running" in kubectl's output, before switching to "Error" after the exception. The logs, however, still state that the pod was terminated (line 4). If you can put out an hpi with enhanced logging, I could install it in my cluster to tell you more.
            Hide
            csanchez Carlos Sanchez added a comment -

            ok, so it is the same issue as above, the plugin is sending the "delete" to the pod, it seems to succeed (no errors from api), but the pod continues running.

            okhttp3 logs would help as described in https://github.com/jenkinsci/kubernetes-plugin#debugging although there are some cases when they are not populated, may need a restart of jenkins.
            Would be good to see what events show up in kubectl describe pod/jenkins-slave... at the bottom

            Show
            csanchez Carlos Sanchez added a comment - ok, so it is the same issue as above, the plugin is sending the "delete" to the pod, it seems to succeed (no errors from api), but the pod continues running. okhttp3 logs would help as described in https://github.com/jenkinsci/kubernetes-plugin#debugging although there are some cases when they are not populated, may need a restart of jenkins. Would be good to see what events show up in kubectl describe pod/jenkins-slave... at the bottom
            Hide
            vikgun Viktor Gunnarson added a comment -

            Carlos Sanchez

            Here are the events from a `kubectl describe pod`:

            Events:
              FirstSeen	LastSeen	Count	From								SubObjectPath		Type		Reason			Message
              ---------	--------	-----	----								-------------		--------	------			-------
              1m		1m		1	default-scheduler									Normal		Scheduled		Successfully assigned jenkins-build-slave-pd2tx to gke-default-gci-pool-0e10fc26-j9nc
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "workspace-volume" 
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "volume-1" 
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "volume-0" 
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-j3cga" 
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc	spec.containers{jnlp}	Normal		Pulling			pulling image "gcr.io/cloud-solutions-images/jenkins-k8s-slave"
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc	spec.containers{jnlp}	Normal		Pulled			Successfully pulled image "gcr.io/cloud-solutions-images/jenkins-k8s-slave"
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc	spec.containers{jnlp}	Normal		Created			Created container
              1m		1m		1	kubelet, gke-default-gci-pool-0e10fc26-j9nc	spec.containers{jnlp}	Normal		Started			Started container
            

            Here are logs from okhttp3:

            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            --> GET https://kubernetes.default/api/v1/pods?labelSelector=jenkins%3Dslave http/1.1
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Host: kubernetes.default
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Connection: Keep-Alive
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Accept-Encoding: gzip
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            User-Agent: okhttp/3.7.0
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            --> END GET
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            <-- 200 OK https://kubernetes.default/api/v1/pods?labelSelector=jenkins%3Dslave (8ms)
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Date: Thu, 10 Aug 2017 11:52:04 GMT
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Length: 116
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/pods","resourceVersion":"41458438"},"items":[]}
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            <-- END HTTP (116-byte body)
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            --> GET https://kubernetes.default/api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue http/1.1
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Host: kubernetes.default
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Connection: Keep-Alive
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            Accept-Encoding: gzip
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            User-Agent: okhttp/3.7.0
            
            Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log
            
            --> END GET
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            <-- 200 OK https://kubernetes.default/api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue (8ms)
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Date: Thu, 10 Aug 2017 11:52:04 GMT
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Length: 116
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/pods","resourceVersion":"41458438"},"items":[]}
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            <-- END HTTP (116-byte body)
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            --> POST https://kubernetes.default/api/v1/namespaces/default/pods http/1.1
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json; charset=utf-8
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Length: 1475
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Host: kubernetes.default
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Connection: Keep-Alive
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Accept-Encoding: gzip
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            User-Agent: okhttp/3.7.0
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"finalizers":[],"labels":{"jenkins":"slave","jenkins/jenkins-build-slave":"true"},"name":"jenkins-build-slave-pd2tx","ownerReferences":[]},"spec":{"containers":[{"args":[],"command":[],"env":[{"name":"JENKINS_LOCATION_URL","value":"https://jenkins/"},{"name":"JENKINS_SECRET","value":"3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3"},{"name":"JENKINS_JNLP_URL","value":"http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp"},{"name":"JENKINS_TUNNEL","value":"jenkins-service:50000"},{"name":"JENKINS_NAME","value":"jenkins-build-slave-pd2tx"},{"name":"JENKINS_URL","value":"http://jenkins-service"},{"name":"HOME","value":"/home/jenkins"}],"image":"gcr.io/cloud-solutions-images/jenkins-k8s-slave","imagePullPolicy":"Always","name":"jnlp","ports":[],"resources":{"limits":{},"requests":{"memory":"2600Mi"}},"securityContext":{"privileged":false},"tty":false,"volumeMounts":[{"mountPath":"/home/jenkins/.m2","name":"volume-0","readOnly":false},{"mountPath":"/var/run/docker.sock","name":"volume-1","readOnly":false},{"mountPath":"/home/jenkins","name":"workspace-volume","readOnly":false}],"workingDir":"/home/jenkins"}],"imagePullSecrets":[],"nodeSelector":{},"restartPolicy":"Never","volumes":[{"hostPath":{"path":"/var/tmp"},"name":"volume-0"},{"hostPath":{"path":"/var/run/docker.sock"},"name":"volume-1"},{"emptyDir":{"medium":""},"name":"workspace-volume"}]}}
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            --> END POST (1475-byte body)
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            <-- 201 Created https://kubernetes.default/api/v1/namespaces/default/pods (7ms)
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Date: Thu, 10 Aug 2017 11:52:04 GMT
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Transfer-Encoding: chunked
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            {"kind":"Pod","apiVersion":"v1","metadata":{"name":"jenkins-build-slave-pd2tx","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/jenkins-build-slave-pd2tx","uid":"53c5e2e5-7dc2-11e7-b3bc-42010af0001e","resourceVersion":"41458439","creationTimestamp":"2017-08-10T11:52:04Z","labels":{"jenkins":"slave","jenkins/jenkins-build-slave":"true"},"annotations":{"kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu request for container jnlp"}},"spec":{"volumes":[{"name":"volume-0","hostPath":{"path":"/var/tmp"}},{"name":"volume-1","hostPath":{"path":"/var/run/docker.sock"}},{"name":"workspace-volume","emptyDir":{"sizeLimit":"0"}},{"name":"default-token-j3cga","secret":{"secretName":"default-token-j3cga","defaultMode":420}}],"containers":[{"name":"jnlp","image":"gcr.io/cloud-solutions-images/jenkins-k8s-slave","workingDir":"/home/jenkins","env":[{"name":"JENKINS_LOCATION_URL","value":"https://jenkins/"},{"name":"JENKINS_SECRET","value":"3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3"},{"name":"JENKINS_JNLP_URL","value":"http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp"},{"name":"JENKINS_TUNNEL","value":"jenkins-service:50000"},{"name":"JENKINS_NAME","value":"jenkins-build-slave-pd2tx"},{"name":"JENKINS_URL","value":"http://jenkins-service"},{"name":"HOME","value":"/home/jenkins"}],"resources":{"requests":{"cpu":"100m","memory":"2600Mi"}},"volumeMounts":[{"name":"volume-0","mountPath":"/home/jenkins/.m2"},{"name":"volume-1","mountPath":"/var/run/docker.sock"},{"name":"workspace-volume","mountPath":"/home/jenkins"},{"name":"default-token-j3cga","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"privileged":false}}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.alpha.kubernetes.io/notReady","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.alpha.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Pending","qosClass":"Burstable"}}
            
            Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log
            
            <-- END HTTP (2352-byte body)
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            --> GET https://kubernetes.default/api/v1/namespaces/default/pods/jenkins-build-slave-pd2tx http/1.1
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Host: kubernetes.default
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Connection: Keep-Alive
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Accept-Encoding: gzip
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            User-Agent: okhttp/3.7.0
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            --> END GET
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            <-- 200 OK https://kubernetes.default/api/v1/namespaces/default/pods/jenkins-build-slave-pd2tx (4ms)
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Date: Thu, 10 Aug 2017 11:52:10 GMT
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Transfer-Encoding: chunked
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            {"kind":"Pod","apiVersion":"v1","metadata":{"name":"jenkins-build-slave-pd2tx","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/jenkins-build-slave-pd2tx","uid":"53c5e2e5-7dc2-11e7-b3bc-42010af0001e","resourceVersion":"41458444","creationTimestamp":"2017-08-10T11:52:04Z","labels":{"jenkins":"slave","jenkins/jenkins-build-slave":"true"},"annotations":{"kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu request for container jnlp"}},"spec":{"volumes":[{"name":"volume-0","hostPath":{"path":"/var/tmp"}},{"name":"volume-1","hostPath":{"path":"/var/run/docker.sock"}},{"name":"workspace-volume","emptyDir":{"sizeLimit":"0"}},{"name":"default-token-j3cga","secret":{"secretName":"default-token-j3cga","defaultMode":420}}],"containers":[{"name":"jnlp","image":"gcr.io/cloud-solutions-images/jenkins-k8s-slave","workingDir":"/home/jenkins","env":[{"name":"JENKINS_LOCATION_URL","value":"https://jenkins/"},{"name":"JENKINS_SECRET","value":"3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3"},{"name":"JENKINS_JNLP_URL","value":"http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp"},{"name":"JENKINS_TUNNEL","value":"jenkins-service:50000"},{"name":"JENKINS_NAME","value":"jenkins-build-slave-pd2tx"},{"name":"JENKINS_URL","value":"http://jenkins-service"},{"name":"HOME","value":"/home/jenkins"}],"resources":{"requests":{"cpu":"100m","memory":"2600Mi"}},"volumeMounts":[{"name":"volume-0","mountPath":"/home/jenkins/.m2"},{"name":"volume-1","mountPath":"/var/run/docker.sock"},{"name":"workspace-volume","mountPath":"/home/jenkins"},{"name":"default-token-j3cga","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"privileged":false}}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"gke-default-gci-pool-0e10fc26-j9nc","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.alpha.kubernetes.io/notReady","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.alpha.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-08-10T11:52:04Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-08-10T11:52:05Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-08-10T11:52:04Z"}],"hostIP":"10.240.0.5","podIP":"10.248.2.117","startTime":"2017-08-10T11:52:04Z","containerStatuses":[{"name":"jnlp","state":{"running":{"startedAt":"2017-08-10T11:52:05Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/cloud-solutions-images/jenkins-k8s-slave:latest","imageID":"docker://sha256:a3f51422197c75f12c673f351c3da04a79d1c1508b8f5899eeb5005156f0c719","containerID":"docker://9f6cf64a243b105abf35da0f1e66a03f6b625d02de35d030318fdfeb3c9ddb50"}],"qosClass":"Burstable"}}
            
            Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log
            
            <-- END HTTP (3203-byte body)
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            --> DELETE https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx http/1.1
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json; charset=utf-8
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Length: 67
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Host: kubernetes.default
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Connection: Keep-Alive
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Accept-Encoding: gzip
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            User-Agent: okhttp/3.7.0
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            {"apiVersion":"v1","kind":"DeleteOptions","orphanDependents":false}
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            --> END DELETE (67-byte body)
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            <-- 404 Not Found https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx (3ms)
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Type: application/json
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Date: Thu, 10 Aug 2017 11:53:34 GMT
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Content-Length: 174
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource","reason":"NotFound","details":{},"code":404}
            
            Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log
            
            <-- END HTTP (174-byte body)
            
            Show
            vikgun Viktor Gunnarson added a comment - Carlos Sanchez Here are the events from a `kubectl describe pod`: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 default -scheduler Normal Scheduled Successfully assigned jenkins-build-slave-pd2tx to gke- default -gci-pool-0e10fc26-j9nc 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "workspace-volume" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-1" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "volume-0" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume " default -token-j3cga" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulling pulling image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Pulled Successfully pulled image "gcr.io/cloud-solutions-images/jenkins-k8s-slave" 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Created Created container 1m 1m 1 kubelet, gke- default -gci-pool-0e10fc26-j9nc spec.containers{jnlp} Normal Started Started container Here are logs from okhttp3: Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave (8ms) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue http/1.1 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:03 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/pods?labelSelector=jenkins%3Dslave,jenkins/jenkins-build-slave%3Dtrue (8ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 116 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" :{ "selfLink" : "/api/v1/pods" , "resourceVersion" : "41458438" }, "items" :[]} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (116- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> POST https: //kubernetes. default /api/v1/namespaces/ default /pods http/1.1 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Length: 1475 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "Pod" , "metadata" :{ "annotations" :{}, "finalizers" :[], "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "name" : "jenkins-build-slave-pd2tx" , "ownerReferences" :[]}, "spec" :{ "containers" :[{ "args" :[], "command" :[], "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "imagePullPolicy" : "Always" , "name" : "jnlp" , "ports" :[], "resources" :{ "limits" :{}, "requests" :{ "memory" : "2600Mi" }}, "securityContext" :{ "privileged" : false }, "tty" : false , "volumeMounts" :[{ "mountPath" : "/home/jenkins/.m2" , "name" : "volume-0" , "readOnly" : false },{ "mountPath" : "/ var /run/docker.sock" , "name" : "volume-1" , "readOnly" : false },{ "mountPath" : "/home/jenkins" , "name" : "workspace-volume" , "readOnly" : false }], "workingDir" : "/home/jenkins" }], "imagePullSecrets" :[], "nodeSelector" :{}, "restartPolicy" : "Never" , "volumes" :[{ "hostPath" :{ "path" : "/ var /tmp" }, "name" : "volume-0" },{ "hostPath" :{ "path" : "/ var /run/docker.sock" }, "name" : "volume-1" },{ "emptyDir" :{ "medium" : ""}," name ":" workspace-volume"}]}} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log --> END POST (1475- byte body) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- 201 Created https: //kubernetes. default /api/v1/namespaces/ default /pods (7ms) Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:04 GMT Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458439" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Pending" , "qosClass" : "Burstable" }} Aug 10, 2017 1:52:04 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (2352- byte body) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> GET https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- 200 OK https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx (4ms) Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:52:10 GMT Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Transfer-Encoding: chunked Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Pod" , "apiVersion" : "v1" , "metadata" :{ "name" : "jenkins-build-slave-pd2tx" , "namespace" : " default " , "selfLink" : "/api/v1/namespaces/ default /pods/jenkins-build-slave-pd2tx" , "uid" : "53c5e2e5-7dc2-11e7-b3bc-42010af0001e" , "resourceVersion" : "41458444" , "creationTimestamp" : "2017-08-10T11:52:04Z" , "labels" :{ "jenkins" : "slave" , "jenkins/jenkins-build-slave" : " true " }, "annotations" :{ "kubernetes.io/limit-ranger" : "LimitRanger plugin set: cpu request for container jnlp" }}, "spec" :{ "volumes" :[{ "name" : "volume-0" , "hostPath" :{ "path" : "/ var /tmp" }},{ "name" : "volume-1" , "hostPath" :{ "path" : "/ var /run/docker.sock" }},{ "name" : "workspace-volume" , "emptyDir" :{ "sizeLimit" : "0" }},{ "name" : " default -token-j3cga" , "secret" :{ "secretName" : " default -token-j3cga" , "defaultMode" :420}}], "containers" :[{ "name" : "jnlp" , "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave" , "workingDir" : "/home/jenkins" , "env" :[{ "name" : "JENKINS_LOCATION_URL" , "value" : "https: //jenkins/" },{ "name" : "JENKINS_SECRET" , "value" : "3d8b55a21d7d84f4ad54f95f24a730b858e23edfa09d2bbcbaefd85a42df15c3" },{ "name" : "JENKINS_JNLP_URL" , "value" : "http://jenkins-service/computer/jenkins-build-slave-pd2tx/slave-agent.jnlp" },{ "name" : "JENKINS_TUNNEL" , "value" : "jenkins-service:50000" },{ "name" : "JENKINS_NAME" , "value" : "jenkins-build-slave-pd2tx" },{ "name" : "JENKINS_URL" , "value" : "http://jenkins-service" },{ "name" : "HOME" , "value" : "/home/jenkins" }], "resources" :{ "requests" :{ "cpu" : "100m" , "memory" : "2600Mi" }}, "volumeMounts" :[{ "name" : "volume-0" , "mountPath" : "/home/jenkins/.m2" },{ "name" : "volume-1" , "mountPath" : "/ var /run/docker.sock" },{ "name" : "workspace-volume" , "mountPath" : "/home/jenkins" },{ "name" : " default -token-j3cga" , "readOnly" : true , "mountPath" : "/ var /run/secrets/kubernetes.io/serviceaccount" }], "terminationMessagePath" : "/dev/termination-log" , "terminationMessagePolicy" : "File" , "imagePullPolicy" : "Always" , "securityContext" :{ "privileged" : false }}], "restartPolicy" : "Never" , "terminationGracePeriodSeconds" :30, "dnsPolicy" : "ClusterFirst" , "serviceAccountName" : " default " , "serviceAccount" : " default " , "nodeName" : "gke- default -gci-pool-0e10fc26-j9nc" , "securityContext" :{}, "schedulerName" : " default -scheduler" , "tolerations" :[{ "key" : "node.alpha.kubernetes.io/notReady" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300},{ "key" : "node.alpha.kubernetes.io/unreachable" , " operator " : "Exists" , "effect" : "NoExecute" , "tolerationSeconds" :300}]}, "status" :{ "phase" : "Running" , "conditions" :[{ "type" : "Initialized" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" },{ "type" : "Ready" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:05Z" },{ "type" : "PodScheduled" , "status" : "True" , "lastProbeTime" : null , "lastTransitionTime" : "2017-08-10T11:52:04Z" }], "hostIP" : "10.240.0.5" , "podIP" : "10.248.2.117" , "startTime" : "2017-08-10T11:52:04Z" , "containerStatuses" :[{ "name" : "jnlp" , "state" :{ "running" :{ "startedAt" : "2017-08-10T11:52:05Z" }}, "lastState" :{}, "ready" : true , "restartCount" :0, "image" : "gcr.io/cloud-solutions-images/jenkins-k8s-slave:latest" , "imageID" : "docker://sha256:a3f51422197c75f12c673f351c3da04a79d1c1508b8f5899eeb5005156f0c719" , "containerID" : "docker://9f6cf64a243b105abf35da0f1e66a03f6b625d02de35d030318fdfeb3c9ddb50" }], "qosClass" : "Burstable" }} Aug 10, 2017 1:52:10 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (3203- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx http/1.1 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json; charset=utf-8 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 67 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VueC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW3tajNjZ2EiLCJrdWJlcm5ldGVzLmlvL3NldnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjdiZmY0Njc5LWIxODctMTFlNS1iZTM0LTQyMDEwYWYwMDAxNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.JdJtcAa1nalzsqQnqiTlK0FmE_6OtjSrrjqeEb56Bq0TmRMwnq_Av9UOhGy6yPXF0OsJjbBOaQdFXLOKJ3K7ZBIJVm-oH5yvIu36B1kEdl4QlG82tP6XwooadmEKWSn0qczfHS3yz3ozV0yGEMJZ6CeYA32y5QAcpMwqpfuTGY5FKXpArCdgEaDT4D-XYSYyTrpZViXBkHHHbk54i7xzEj9jEFikV5S7IuwvPV1RpgQz7nQhGnBLCFQ27MAri80OGxpZkrUvihZBQN6VyKvp2D943rl8iaDGqFXJuXjEC04H_o1-a0hhM2nJlZn3mRXSXlm_xKUDB1riljZqvMlubg Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.7.0 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "apiVersion" : "v1" , "kind" : "DeleteOptions" , "orphanDependents" : false } Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log --> END DELETE (67- byte body) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- 404 Not Found https: //kubernetes. default /api/v1/pods/jenkins-build-slave-pd2tx (3ms) Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Date: Thu, 10 Aug 2017 11:53:34 GMT Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 174 Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" : "the server could not find the requested resource" , "reason" : "NotFound" , "details" :{}, "code" :404} Aug 10, 2017 1:53:34 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (174- byte body)
            Hide
            juhtie01 Juha Tiensyrjä added a comment -

            Looks like this is missing the namespace?
            DELETE https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx http/1.1
             

            Show
            juhtie01 Juha Tiensyrjä added a comment - Looks like this is missing the namespace? DELETE https://kubernetes.default/api/v1/pods/jenkins-build-slave-pd2tx http/1.1  
            Hide
            csanchez Carlos Sanchez added a comment -

            do you have a namespace in the cloud definition or in the template ?

            have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released

            Show
            csanchez Carlos Sanchez added a comment - do you have a namespace in the cloud definition or in the template ? have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released
            Hide
            juhtie01 Juha Tiensyrjä added a comment -

            I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.

            Show
            juhtie01 Juha Tiensyrjä added a comment - I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.
            Hide
            zmeggyesi Zalan Meggyesi added a comment -

            I have the specific namespace (not default) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the Error state.

            Show
            zmeggyesi Zalan Meggyesi added a comment - I have the specific namespace (not default ) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the  Error state.
            Hide
            vikgun Viktor Gunnarson added a comment -

            I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.

            Show
            vikgun Viktor Gunnarson added a comment - I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.
            Hide
            vikgun Viktor Gunnarson added a comment -

            Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace:

            Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log
            --> DELETE https://kubernetes.default/api/v1/namespaces/default/pods/jenkins-build-slave-pw0h7 http/1.1
            
            Show
            vikgun Viktor Gunnarson added a comment - Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace: Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pw0h7 http/1.1

              People

              Assignee:
              csanchez Carlos Sanchez
              Reporter:
              vikgun Viktor Gunnarson
              Votes:
              10 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: