• Type: Bug
• Resolution: Not A Defect
• Priority: Critical
• Component/s: kubernetes-plugin
• Labels:

  • bug
• Environment:

  Running jenkins in a Kubernetes cluster on GCP

[JENKINS-55527] Builds fail randomly when running sh in container

Ahmed Kamel created issue - 2019-01-11 01:05

Ahmed Kamel made changes - 2019-01-11 01:12

Description

Original: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in the cluster. Here is the error message I see.  [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed... ... ...ERROR: script returned exit code 1 Finished: FAILURE

New: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.   [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE

Ahmed Kamel made changes - 2019-01-11 01:12

Description

Original: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.   [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE

New: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.  ```  [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE ```

Ahmed Kamel made changes - 2019-01-11 01:13

Description

Original: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.  ```  [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE ```

New: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.   [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE

Ahmed Kamel made changes - 2019-01-11 01:13

Description

Original: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.   [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ...ERROR: script returned exit code 1  Finished: FAILURE

New: My devs are complaining of builds failing randomly when a stage starts. The builds fail when attempting to run "sh" in a container in the pods running the job.  Here is the error message I see.  [Pipeline] shrpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3786794 to cgroups caused \"failed to write 3786794 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod70971cd7-153a-11e9-9fe5-42010a567404/6b66fd31d9718f168c34810477e328045af5caead06e9e7f48ed3b9431eb3d37/cgroup.procs: invalid argument\""[Pipeline] echoError: java.io.IOException: Pipe closed...  ...  ... ERROR: script returned exit code 1  Finished: FAILURE

Ahmed Kamel made changes - 2019-01-13 22:04

Priority

Original: Major [ 3 ]

New: Critical [ 2 ]

Collapse comment: Andy Powell added a comment - 2019-01-17 01:08

Andy Powell added a comment - 2019-01-17 01:08

We are seeing similar errors.
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3559144 to cgroups caused \"failed to write 3559144 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod03460050-19f2-11e9-beb6-42010a8e01b2/f4978b3f515bcf1f942bd0ed21ce084ca2039e6ae56f357870cbbe55517ed151/cgroup.procs: invalid argument\""

command terminated with non-zero exit code: Error executing in Docker Container: 126process apparently never started in /home/jenkins/workspace/le_platform-nodejs-hello_sandbox@tmp/durable-11cdeba1
Jenkins version = 2.150.1, running on GKE

 

Expand comment: Andy Powell added a comment - 2019-01-17 01:08

Andy Powell added a comment - 2019-01-17 01:08 We are seeing similar errors. rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:87: adding pid 3559144 to cgroups caused \"failed to write 3559144 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/kubepods/besteffort/pod03460050-19f2-11e9-beb6-42010a8e01b2/f4978b3f515bcf1f942bd0ed21ce084ca2039e6ae56f357870cbbe55517ed151/cgroup.procs: invalid argument\"" command terminated with non-zero exit code: Error executing in Docker Container: 126process apparently never started in /home/jenkins/workspace/le_platform-nodejs-hello_sandbox@tmp/durable-11cdeba1 Jenkins version = 2.150.1, running on GKE  

Collapse comment: Carlos Sanchez added a comment - 2019-01-17 08:06

Carlos Sanchez added a comment - 2019-01-17 08:06

This looks like https://github.com/moby/moby/issues/31230 and the fix could be in runc v1.0.0-rc6 https://github.com/opencontainers/runc/pull/1916

Expand comment: Carlos Sanchez added a comment - 2019-01-17 08:06

Carlos Sanchez added a comment - 2019-01-17 08:06 This looks like https://github.com/moby/moby/issues/31230 and the fix could be in runc v1.0.0-rc6 https://github.com/opencontainers/runc/pull/1916

Collapse comment: Ahmed Kamel added a comment - 2019-01-17 23:45

Ahmed Kamel added a comment - 2019-01-17 23:45

Got it. Thanks for posting these. I'll watch the issue over on github.

Expand comment: Ahmed Kamel added a comment - 2019-01-17 23:45

Ahmed Kamel added a comment - 2019-01-17 23:45 Got it. Thanks for posting these. I'll watch the issue over on github.

Collapse comment: Andy Powell added a comment - 2019-01-18 21:02

Andy Powell added a comment - 2019-01-18 21:02

Update: We were able to isolate this to a security scanner within our GKE cluster.  Turning it off made the problems go away.  

Expand comment: Andy Powell added a comment - 2019-01-18 21:02

Andy Powell added a comment - 2019-01-18 21:02 Update: We were able to isolate this to a security scanner within our GKE cluster.  Turning it off made the problems go away.  

Load more newer events