-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins 2.32.2
I tried to narrow this bug down, but there isn't much information. We just upgraded to all newest plugins, but unfortunately we upgraded a lot at once, so no idea which one.
This is spamming out logs every few seconds:
00:19:42.695 Cannot contact kubernetes-ef39fe82c8a541be84bd780e4d7c1ddb-ce4d47fc96bfc: java.io.IOException: corrupted content in /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-96fa79b7/pid: java.lang.NumberFormatException: For input string: "" 00:19:57.758 Cannot contact kubernetes-ef39fe82c8a541be84bd780e4d7c1ddb-ce4d47fc96bfc: java.io.IOException: corrupted content in /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-96fa79b7/pid: java.lang.NumberFormatException: For input string: "" 00:20:12.769 Cannot contact kubernetes-ef39fe82c8a541be84bd780e4d7c1ddb-ce4d47fc96bfc: java.io.IOException: corrupted content in /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-96fa79b7/pid: java.lang.NumberFormatException: For input string: ""
Add information from comments:
The same kind of problem has occurred in my Jenkins running on GKE. This problem occurred by upgrading Pipeline Nodes and Processes Plugin from 2.8 to 2.9. I am confirming that this problem temporarily resolves by downgrading that plugin from 2.9 to 2.8.
BTW workflow-durable-task-step 2.9 does add this log message but it is just exposing a problem that was already there, and simply being suppressed unless you were running a sufficiently fine logger. The problem is that this code is seeing a file which is supposed to contain a number once created, whereas it is being created as empty for some reason.
Again the bug probably exists in all versions, it is only printed to the build log as of 2.9. You can add a FINE logger to org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep to verify.
- blocks
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Closed
-
-
-
- Closed
-
- is duplicated by
-
-
- Resolved
-
-
-
- Closed
-
- is related to
-
-
- Resolved
-
- relates to
-
-
- Resolved
-
- links to
(1 is duplicated by, 1 is related to, 1 relates to, 1 links to)
[JENKINS-42048] Cannot Connect, PID NumberFormatException
Lars Lawoko
created issue -
J Knurek
added a comment - I'm experiencing the same recurring error message
the slave is using the golang docker image, and the pipeline is setup like this:
podTemplate(label: 'jenkpod', containers: [ containerTemplate(name: 'golang', image: 'golang:1.8', ttyEnabled: true, command: 'cat') ]) { node ('jenkpod') { container('golang') { stage('Pre-Build') { checkout scm sh 'make get' } } } }
the events for the slave pod:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
13m 13m 1 {default-scheduler } Normal Scheduled Successfully assigned kubernetes-f1e4a27973a941c2af08bebbc74cc080-10bf4e8527c4 to gke-jenkins
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Pulled Container image "golang:1.8" already present on machine
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Created Created container with docker id 97e4b71e323e; Security:[seccomp=unconfined]
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Started Started container with docker id 97e4b71e323e
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Created Created container with docker id 628623d03379; Security:[seccomp=unconfined]
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Started Started container with docker id 628623d03379
J Knurek
added a comment - I'm experiencing the same recurring error message
the slave is using the golang docker image, and the pipeline is setup like this:
podTemplate(label: 'jenkpod' , containers: [
containerTemplate(name: 'golang' , image: 'golang:1.8' , ttyEnabled: true , command: 'cat' )
]) {
node ( 'jenkpod' ) { container( 'golang' ) {
stage( 'Pre-Build' ) {
checkout scm
sh 'make get'
}
} }
}
the events for the slave pod:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
13m 13m 1 { default -scheduler } Normal Scheduled Successfully assigned kubernetes-f1e4a27973a941c2af08bebbc74cc080-10bf4e8527c4 to gke-jenkins
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Pulled Container image "golang:1.8" already present on machine
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Created Created container with docker id 97e4b71e323e; Security:[seccomp=unconfined]
13m 13m 1 {kubelet gke-jenkins} spec.containers{golang} Normal Started Started container with docker id 97e4b71e323e
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Created Created container with docker id 628623d03379; Security:[seccomp=unconfined]
13m 13m 1 {kubelet gke-jenkins} spec.containers{jnlp} Normal Started Started container with docker id 628623d03379
For more context we are running on the Google container engine (hosted k8), The weird thing is that it seems to be working right, i.e the pipeline builds, even with the constant exception.
The exception starts once the gradle shell is run:
[Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Build & run unit tests) [Pipeline] withEnv [Pipeline] { [Pipeline] sh 00:01:05.567 [Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A] Running shell script 00:01:05.572 Executing shell script inside container [gcloud-jdk7] of pod [kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9] 00:01:05.653 Executing command: sh -c echo $$ > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid'; jsc=durable-ca85172bfb8670e4c44f30557e14af18; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/script.sh' > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-result.txt' 00:01:05.694 # cd /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A 00:01:05.694 sh -c echo $$ > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid'; jsc=durable-ca85172bfb8670e4c44f30557e14af18; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/script.sh' > '/home/jenkins/workspace/Robusta_robusta_develop-6E# PNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-result.txt' 00:01:05.694 exit 00:01:05.909 + ./gradlew --stacktrace --parallel buildUnitTest 00:01:05.909 Downloading https://services.gradle.org/distributions/gradle-3.3-all.zip 00:01:05.995 Cannot contact kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9: java.io.IOException: corrupted content in /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid: java.lang.NumberFormatException: For input string: ""
Our jenkinsfile is pretty massive, but here is the core (has been edited):
podTemplate(label: 'JavaPod', containers: [ containerTemplate( name: 'gcloud-jdk7', image: 'gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7', ttyEnabled: true, args: 'cat', command: '/bin/sh -c', alwaysPullImage: true, workingDir: '/home/jenkins', resourceRequestCpu: '2', resourceRequestMemory: '8Gi', resourceLimitCpu: '5', resourceLimitMemory: '9Gi', ), containerTemplate( name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${computer.jnlpmac} ${computer.name}', resourceRequestCpu: '100m', resourceRequestMemory: '500Mi', resourceLimitCpu: '500m', resourceLimitMemory: '1Gi', ) ]) { node('JavaPod') { container('gcloud-jdk7') { timeout(30) { //assume something is wrong if it takes an half an hour stage('checkout source') { checkout scm } switch (env.BRANCH_NAME) { case 'develop': buildUnitTest() runIntegrationTests('local') } } } void buildUnitTest() { stage('Build & run unit tests') { withEnv(runEnv) { try { def command = './gradlew --stacktrace --parallel buildUnitTest' if (env.BRANCH_NAME == 'master'){ command = 'export ROBUSTA_PROD_ANALYTICS=true && ' + command } sh command } catch (Exception e) { junit allowEmptyResults: true, testResults: '**/build/test-results/**/*.xml' step([$class: 'CheckStylePublisher', canComputeNew: false, defaultEncoding: '', healthy: '', pattern: '**/main.xml,**/test.xml', unHealthy: '']) throw e } junit allowEmptyResults: true, testResults: '**/build/test-results/**/*.xml' step([$class: 'CheckStylePublisher', canComputeNew: false, defaultEncoding: '', healthy: '', pattern: '**/main.xml,**/test.xml', unHealthy: '']) } } } void runIntegrationTests(String targetEnv) { stage('Run integration tests') { withEnv(runEnv) { try { sh "./gradlew --stacktrace :robusta-integration-tests:integrationTest -PtestEnv=${targetEnv}" } catch (Exception e) { junit allowEmptyResults: true, testResults: '**/build/test-results/**/*.xml' throw e } junit allowEmptyResults: true, testResults: '**/build/test-results/**/*.xml' } } }
kubectl describe pod kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Name: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Namespace: default Node: gke-pci-default-pool-44b03267-0ztl/10.140.0.2 Start Time: Thu, 16 Feb 2017 09:38:06 +1100 Labels: jenkins=slave jenkins/JavaPod=true Status: Running IP: 10.40.20.94 Controllers: <none> Containers: gcloud-jdk7: Container ID: docker://28c3b60ae04d952cce366d5a03c2d950a171594828fd19446fe2aa9ed379dd33 Image: gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7 Image ID: docker://sha256:16e905ffe4f3393f6ee4b5125971a3029d6162ca0d3db5b2973f1f13b6201c3f Port: Command: /bin/sh -c Args: cat Limits: cpu: 5 memory: 9Gi Requests: cpu: 2 memory: 8Gi State: Running Started: Thu, 16 Feb 2017 09:38:07 +1100 Ready: True Restart Count: 0 Volume Mounts: /home/jenkins from workspace-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-y9hsd (ro) Environment Variables: JENKINS_SECRET: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346 JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 JENKINS_LOCATION_URL: https://build-robusta.papercut.software/ JENKINS_URL: http://kubectl describe pod kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Name: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Namespace: default Node: gke-pci-default-pool-44b03267-0ztl/10.140.0.2 Start Time: Thu, 16 Feb 2017 09:38:06 +1100 Labels: jenkins=slave jenkins/JavaPod=true Status: Running IP: 10.40.20.94 Controllers: <none> Containers: gcloud-jdk7: Container ID: docker://28c3b60ae04d952cce366d5a03c2d950a171594828fd19446fe2aa9ed379dd33 Image: gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7 Image ID: docker://sha256:16e905ffe4f3393f6ee4b5125971a3029d6162ca0d3db5b2973f1f13b6201c3f Port: Command: /bin/sh -c Args: cat Limits: cpu: 5 memory: 9Gi Requests: cpu: 2 memory: 8Gi State: Running Started: Thu, 16 Feb 2017 09:38:07 +1100 Ready: True Restart Count: 0 Volume Mounts: /home/jenkins from workspace-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-y9hsd (ro) Environment Variables: JENKINS_SECRET: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346 JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 JENKINS_LOCATION_URL: https://build-robusta.papercut.software/ JENKINS_URL: http://build-robusta JENKINS_JNLP_URL: http://build-robusta/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp HOME: /home/jenkins jnlp: Container ID: docker://b27bb762a03525763aee7d2a60a85b4c3331aa91c6ac7d40b40693f570c1b564 Image: jenkinsci/jnlp-slave:alpine Image ID: docker://sha256:254fd665eaf0229f38295a9eac6c7f9bf32a2f450ecbcc8212f3e53b96dd339d Port: Args: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346 kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Limits: cpu: 500m memory: 1Gi Requests: cpu: 100m memory: 500Mi State: Running Started: Thu, 16 Feb 2017 09:38:06 +1100 Ready: True Restart Count: 0 Volume Mounts: /home/jenkins from workspace-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-y9hsd (ro) Environment Variables: JENKINS_SECRET: <secret> JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 JENKINS_LOCATION_URL: <jenkins url> (I replaced) JENKINS_URL: <jenkins url> (I replaced) JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp HOME: /home/jenkins Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: workspace-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-y9hsd: Type: Secret (a volume populated by a Secret) SecretName: default-token-y9hsd QoS Class: Burstable Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 to gke-pci-default-pool-44b03267-0ztl 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Created Created container with docker id b27bb762a035; Security:[seccomp=unconfined] 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Started Started container with docker id b27bb762a035 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulling pulling image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7" 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulled Successfully pulled image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7" 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Created Created container with docker id 28c3b60ae04d; Security:[seccomp=unconfined] 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Started Started container with docker id 28c3b60ae04d JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp HOME: /home/jenkins jnlp: Container ID: docker://b27bb762a03525763aee7d2a60a85b4c3331aa91c6ac7d40b40693f570c1b564 Image: jenkinsci/jnlp-slave:alpine Image ID: docker://sha256:254fd665eaf0229f38295a9eac6c7f9bf32a2f450ecbcc8212f3e53b96dd339d Port: Args: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346 kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 Limits: cpu: 500m memory: 1Gi Requests: cpu: 100m memory: 500Mi State: Running Started: Thu, 16 Feb 2017 09:38:06 +1100 Ready: True Restart Count: 0 Volume Mounts: /home/jenkins from workspace-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-y9hsd (ro) Environment Variables: JENKINS_SECRET: <secret> JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 JENKINS_LOCATION_URL: <jenkins url> (I replaced) JENKINS_URL: <jenkins url> (I replaced) JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp HOME: /home/jenkins Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: workspace-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-y9hsd: Type: Secret (a volume populated by a Secret) SecretName: default-token-y9hsd QoS Class: Burstable Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 to gke-pci-default-pool-44b03267-0ztl 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Created Created container with docker id b27bb762a035; Security:[seccomp=unconfined] 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{jnlp} Normal Started Started container with docker id b27bb762a035 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulling pulling image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7" 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulled Successfully pulled image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7" 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Created Created container with docker id 28c3b60ae04d; Security:[seccomp=unconfined] 1m 1m 1 {kubelet gke-pci-default-pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Started Started container with docker id 28c3b60ae04d
Lars Lawoko
added a comment - - edited For more context we are running on the Google container engine (hosted k8), The weird thing is that it seems to be working right, i.e the pipeline builds, even with the constant exception.
The exception starts once the gradle shell is run:
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build & run unit tests)
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
00:01:05.567 [Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A] Running shell script
00:01:05.572 Executing shell script inside container [gcloud-jdk7] of pod [kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9]
00:01:05.653 Executing command: sh -c echo $$ > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid' ; jsc=durable-ca85172bfb8670e4c44f30557e14af18; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/script.sh' > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-result.txt'
00:01:05.694 # cd /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A
00:01:05.694 sh -c echo $$ > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid' ; jsc=durable-ca85172bfb8670e4c44f30557e14af18; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/script.sh' > '/home/jenkins/workspace/Robusta_robusta_develop-6E# PNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/jenkins-result.txt'
00:01:05.694 exit
00:01:05.909 + ./gradlew --stacktrace --parallel buildUnitTest
00:01:05.909 Downloading https: //services.gradle.org/distributions/gradle-3.3-all.zip
00:01:05.995 Cannot contact kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9: java.io.IOException: corrupted content in /home/jenkins/workspace/Robusta_robusta_develop-6EPNQBJK5BYEXOJV6L45MMZZGUIP7WO4Y6EGRUYNFFMRC7B2GL3A@tmp/durable-aa4bf913/pid: java.lang.NumberFormatException: For input string: ""
Our jenkinsfile is pretty massive, but here is the core (has been edited):
podTemplate(label: 'JavaPod' , containers: [
containerTemplate(
name: 'gcloud-jdk7' ,
image: 'gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7' ,
ttyEnabled: true ,
args: 'cat' ,
command: '/bin/sh -c' ,
alwaysPullImage: true ,
workingDir: '/home/jenkins' ,
resourceRequestCpu: '2' ,
resourceRequestMemory: '8Gi' ,
resourceLimitCpu: '5' ,
resourceLimitMemory: '9Gi' ,
),
containerTemplate(
name: 'jnlp' ,
image: 'jenkinsci/jnlp-slave:alpine' ,
args: '${computer.jnlpmac} ${computer.name}' ,
resourceRequestCpu: '100m' ,
resourceRequestMemory: '500Mi' ,
resourceLimitCpu: '500m' ,
resourceLimitMemory: '1Gi' ,
)
]) {
node( 'JavaPod' ) {
container( 'gcloud-jdk7' ) {
timeout(30) { //assume something is wrong if it takes an half an hour
stage( 'checkout source' ) {
checkout scm
}
switch (env.BRANCH_NAME) {
case 'develop' :
buildUnitTest()
runIntegrationTests( 'local' )
}
}
}
void buildUnitTest() {
stage( 'Build & run unit tests' ) {
withEnv(runEnv) {
try {
def command = './gradlew --stacktrace --parallel buildUnitTest'
if (env.BRANCH_NAME == 'master' ){
command = 'export ROBUSTA_PROD_ANALYTICS= true && ' + command
}
sh command
} catch (Exception e) {
junit allowEmptyResults: true , testResults: '**/build/test-results /**/ *.xml'
step([$class: 'CheckStylePublisher' , canComputeNew: false , defaultEncoding: '', healthy: ' ', pattern: ' **/main.xml,**/test.xml ', unHealthy: ' '])
throw e
}
junit allowEmptyResults: true , testResults: '**/build/test-results /**/ *.xml'
step([$class: 'CheckStylePublisher' , canComputeNew: false , defaultEncoding: '', healthy: ' ', pattern: ' **/main.xml,**/test.xml ', unHealthy: ' '])
}
}
}
void runIntegrationTests( String targetEnv) {
stage( 'Run integration tests' ) {
withEnv(runEnv) {
try {
sh "./gradlew --stacktrace :robusta-integration-tests:integrationTest -PtestEnv=${targetEnv}"
} catch (Exception e) {
junit allowEmptyResults: true , testResults: '**/build/test-results /**/ *.xml'
throw e
}
junit allowEmptyResults: true , testResults: '**/build/test-results /**/ *.xml'
}
}
}
kubectl describe pod kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Name: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Namespace: default
Node: gke-pci- default -pool-44b03267-0ztl/10.140.0.2
Start Time: Thu, 16 Feb 2017 09:38:06 +1100
Labels: jenkins=slave
jenkins/JavaPod= true
Status: Running
IP: 10.40.20.94
Controllers: <none>
Containers:
gcloud-jdk7:
Container ID: docker: //28c3b60ae04d952cce366d5a03c2d950a171594828fd19446fe2aa9ed379dd33
Image: gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7
Image ID: docker: //sha256:16e905ffe4f3393f6ee4b5125971a3029d6162ca0d3db5b2973f1f13b6201c3f
Port:
Command:
/bin/sh
-c
Args:
cat
Limits:
cpu: 5
memory: 9Gi
Requests:
cpu: 2
memory: 8Gi
State: Running
Started: Thu, 16 Feb 2017 09:38:07 +1100
Ready: True
Restart Count: 0
Volume Mounts:
/home/jenkins from workspace-volume (rw)
/ var /run/secrets/kubernetes.io/serviceaccount from default -token-y9hsd (ro)
Environment Variables:
JENKINS_SECRET: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346
JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
JENKINS_LOCATION_URL: https: //build-robusta.papercut.software/
JENKINS_URL: http: //kubectl describe pod kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Name: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Namespace: default
Node: gke-pci- default -pool-44b03267-0ztl/10.140.0.2
Start Time: Thu, 16 Feb 2017 09:38:06 +1100
Labels: jenkins=slave
jenkins/JavaPod= true
Status: Running
IP: 10.40.20.94
Controllers: <none>
Containers:
gcloud-jdk7:
Container ID: docker: //28c3b60ae04d952cce366d5a03c2d950a171594828fd19446fe2aa9ed379dd33
Image: gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7
Image ID: docker: //sha256:16e905ffe4f3393f6ee4b5125971a3029d6162ca0d3db5b2973f1f13b6201c3f
Port:
Command:
/bin/sh
-c
Args:
cat
Limits:
cpu: 5
memory: 9Gi
Requests:
cpu: 2
memory: 8Gi
State: Running
Started: Thu, 16 Feb 2017 09:38:07 +1100
Ready: True
Restart Count: 0
Volume Mounts:
/home/jenkins from workspace-volume (rw)
/ var /run/secrets/kubernetes.io/serviceaccount from default -token-y9hsd (ro)
Environment Variables:
JENKINS_SECRET: 16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346
JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
JENKINS_LOCATION_URL: https: //build-robusta.papercut.software/
JENKINS_URL: http: //build-robusta
JENKINS_JNLP_URL: http: //build-robusta/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp
HOME: /home/jenkins
jnlp:
Container ID: docker: //b27bb762a03525763aee7d2a60a85b4c3331aa91c6ac7d40b40693f570c1b564
Image: jenkinsci/jnlp-slave:alpine
Image ID: docker: //sha256:254fd665eaf0229f38295a9eac6c7f9bf32a2f450ecbcc8212f3e53b96dd339d
Port:
Args:
16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346
kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Limits:
cpu: 500m
memory: 1Gi
Requests:
cpu: 100m
memory: 500Mi
State: Running
Started: Thu, 16 Feb 2017 09:38:06 +1100
Ready: True
Restart Count: 0
Volume Mounts:
/home/jenkins from workspace-volume (rw)
/ var /run/secrets/kubernetes.io/serviceaccount from default -token-y9hsd (ro)
Environment Variables:
JENKINS_SECRET: <secret>
JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
JENKINS_LOCATION_URL: <jenkins url> (I replaced)
JENKINS_URL: <jenkins url> (I replaced)
JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp
HOME: /home/jenkins
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default -token-y9hsd:
Type: Secret (a volume populated by a Secret)
SecretName: default -token-y9hsd
QoS Class : Burstable
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 { default -scheduler } Normal Scheduled Successfully assigned kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 to gke-pci- default -pool-44b03267-0ztl
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Created Created container with docker id b27bb762a035; Security:[seccomp=unconfined]
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Started Started container with docker id b27bb762a035
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulling pulling image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7"
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulled Successfully pulled image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7"
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Created Created container with docker id 28c3b60ae04d; Security:[seccomp=unconfined]
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Started Started container with docker id 28c3b60ae04d
JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp
HOME: /home/jenkins
jnlp:
Container ID: docker: //b27bb762a03525763aee7d2a60a85b4c3331aa91c6ac7d40b40693f570c1b564
Image: jenkinsci/jnlp-slave:alpine
Image ID: docker: //sha256:254fd665eaf0229f38295a9eac6c7f9bf32a2f450ecbcc8212f3e53b96dd339d
Port:
Args:
16b02724728739b72e0b559940adc7b5da29e9e190e8a35e858cece4bbc92346
kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
Limits:
cpu: 500m
memory: 1Gi
Requests:
cpu: 100m
memory: 500Mi
State: Running
Started: Thu, 16 Feb 2017 09:38:06 +1100
Ready: True
Restart Count: 0
Volume Mounts:
/home/jenkins from workspace-volume (rw)
/ var /run/secrets/kubernetes.io/serviceaccount from default -token-y9hsd (ro)
Environment Variables:
JENKINS_SECRET: <secret>
JENKINS_NAME: kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9
JENKINS_LOCATION_URL: <jenkins url> (I replaced)
JENKINS_URL: <jenkins url> (I replaced)
JENKINS_JNLP_URL: <jenkins url> (I replaced)/computer/kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9/slave-agent.jnlp
HOME: /home/jenkins
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default -token-y9hsd:
Type: Secret (a volume populated by a Secret)
SecretName: default -token-y9hsd
QoS Class : Burstable
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 { default -scheduler } Normal Scheduled Successfully assigned kubernetes-9f544f8d984342c8bfa152fd3134608b-d1fdf7ba230b9 to gke-pci- default -pool-44b03267-0ztl
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Pulled Container image "jenkinsci/jnlp-slave:alpine" already present on machine
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Created Created container with docker id b27bb762a035; Security:[seccomp=unconfined]
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{jnlp} Normal Started Started container with docker id b27bb762a035
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulling pulling image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7"
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Pulled Successfully pulled image "gcr.io/pc-infrastructure/robusta-jenkins-gcloud-jdk7"
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Created Created container with docker id 28c3b60ae04d; Security:[seccomp=unconfined]
1m 1m 1 {kubelet gke-pci- default -pool-44b03267-0ztl} spec.containers{gcloud-jdk7} Normal Started Started container with docker id 28c3b60ae04d
Daichi Hirata
added a comment - The same kind of problem has occurred in my Jenkins running on GKE. This problem occurred by upgrading Pipeline Nodes and Processes Plugin from 2.8 to 2.9. I am confirming that this problem temporarily resolves by downgrading that plugin from 2.9 to 2.8.
Daichi Hirata
added a comment - The same kind of problem has occurred in my Jenkins running on GKE. This problem occurred by upgrading Pipeline Nodes and Processes Plugin from 2.8 to 2.9. I am confirming that this problem temporarily resolves by downgrading that plugin from 2.9 to 2.8. Lars Lawoko
made changes -
| Component/s | New: workflow-durable-task-step-plugin [ 21715 ] |
daichirata Thanks, just tried this and can confirm that Pipeline Nodes and Processes Plugin 2.9 is the issue
iocanel He narrowed it down to this plugin / version
Lars Lawoko
added a comment - - edited daichirata Thanks, just tried this and can confirm that Pipeline Nodes and Processes Plugin 2.9 is the issue
iocanel He narrowed it down to this plugin / version Awesome, that's really helpfull!
Unfortunately, I can only help regarding the kubernetes-plugin, can you reassign the issue to someone involved with `Pipeline Nodes and Processes Plugin`?
iocanel Will do, i believe jglick is part of that plugin team?
Also if you are in the kubernetes team and have time, could you have a quick look at JENKINS-40647 , I can confirm that it is still happening with the plugins updated to latest.
Lars Lawoko
added a comment - - edited iocanel Will do, i believe jglick is part of that plugin team?
Also if you are in the kubernetes team and have time, could you have a quick look at JENKINS-40647 , I can confirm that it is still happening with the plugins updated to latest. Lars Lawoko
made changes -
| Assignee | Original: Ioannis Canellos [ iocanel ] | New: Jesse Glick [ jglick ] |
It seems that you pod is created but for some reason can't read the pid, which is something I've never seen before.
Can you please tell us how your pipeline looks like and maybe get us the output of `kubectl describe` for the kubernetes-ef39fe82c8a541be84bd780e4d7c1ddb-ce4d47fc96bfc pod?