-
Bug
-
Resolution: Won't Fix
-
Blocker
-
jenkins 1.625
docker-commons:1.3.1
docker-workflow:1.2
durable-task:1.8
pipeline-rest-api:1.0
pipeline-stage-view:1.0
pipeline-utility-steps:1.0
workflow-aggregator:1.14
workflow-api:1.14
workflow-basic-steps:1.14
workflow-cps-checkpoint:1.9
workflow-cps-global-lib:1.14
workflow-cps:1.14
workflow-durable-task-step:1.14
workflow-job:1.14
workflow-multibranch:1.14
workflow-scm-step:1.14
workflow-step-api:1.14
workflow-support:1.14jenkins 1.625 docker-commons:1.3.1 docker-workflow:1.2 durable-task:1.8 pipeline-rest-api:1.0 pipeline-stage-view:1.0 pipeline-utility-steps:1.0 workflow-aggregator:1.14 workflow-api:1.14 workflow-basic-steps:1.14 workflow-cps-checkpoint:1.9 workflow-cps-global-lib:1.14 workflow-cps:1.14 workflow-durable-task-step:1.14 workflow-job:1.14 workflow-multibranch:1.14 workflow-scm-step:1.14 workflow-step-api:1.14 workflow-support:1.14
create a job called "my job" with the following pipeline
node ("docker") { sh 'pwd' img = docker.image("ubuntu") img.inside { sh 'pwd' } }
[Pipeline] Allocate node : Start
Running on docker in /root/workspace/workspace/test-jobs/jnord/docker inside
[Pipeline] node {
[Pipeline] sh
[docker inside] Running shell script
+ pwd
/root/workspace/workspace/test-jobs/jnord/docker inside
[Pipeline] sh
[docker inside] Running shell script
+ docker inspect -f . ubuntu
.
[Pipeline] Run build steps inside a Docker container : Start
$ docker run -t -d -u 0:0 -w "/root/workspace/workspace/test-jobs/jnord/docker inside" -v "/root/workspace/workspace/test-jobs/jnord/docker inside:/root/workspace/workspace/test-jobs/jnord/docker inside:rw" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** ubuntu cat
[Pipeline] withDockerContainer {
[Pipeline] sh
[docker inside] Running shell script
sh: 1: cannot create /root/workspace/workspace/test-jobs/jnord/docker inside@tmp/durable-5ea3c644/pid: Directory nonexistent
sh: 1: cannot create /root/workspace/workspace/test-jobs/jnord/docker inside@tmp/durable-5ea3c644/jenkins-log.txt: Directory nonexistent
sh: 1: cannot create /root/workspace/workspace/test-jobs/jnord/docker inside@tmp/durable-5ea3c644/jenkins-result.txt: Directory nonexistent
[Pipeline] } //withDockerContainer
$ docker stop 260e1e96255fd938e64d6bcd296ce616e01a40ece5ef093dfd97325418247904
$ docker rm -f 260e1e96255fd938e64d6bcd296ce616e01a40ece5ef093dfd97325418247904
[Pipeline] Run build steps inside a Docker container : End
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Closed
-
[JENKINS-33632] docker.inside broken with old client versions
Hello jglick,
I think I found a self-contained example showing that the issue is still around. Using the following versions of Docker
- Docker 1.12.2, build bb80604 (on linux).
- Docker 1.12.3, build 6b644ec, experimental (on windows).
And using this gist, I ended up with the following error on my two machines:
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/jobs/demo/workspace -v /var/jenkins_home/jobs/demo/workspace:/var/jenkins_home/jobs/demo/workspace:rw -v /var/jenkins_home/jobs/demo/workspace@tmp:/var/jenkins_home/jobs/demo/workspace@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat ubuntu
[Pipeline] {
[Pipeline] sh
[workspace] Running shell script
sh: 1: cannot create /var/jenkins_home/jobs/demo/workspace@tmp/durable-252b7e97/pid: Directory nonexistent
sh: 1: cannot create /var/jenkins_home/jobs/demo/workspace@tmp/durable-252b7e97/jenkins-log.txt: Directory nonexistent
sh: 1: cannot create /var/jenkins_home/jobs/demo/workspace@tmp/durable-252b7e97/jenkins-result.txt: Directory nonexistent
Version of the Docker pipeline is 1.9.1 at the time of this writing, but as you can see in the console dump, the -v flag is used instead of the --volume-from, as if the plugin did not detect it was running inside a container.
Steps to reproduce
- Copy this gist
- Set the executable permissions on the scripts.
- Run the build script and then the run script.
- The Jenkins instance will be available on port 8080. There is one job in it running the following:
node ('master') {
sh 'pwd'
img = docker.image('ubuntu')
img.inside {
sh 'pwd'
}
}
Do you think there is enough to reopen this issue?
Thanks
The errors from sh are irrelevant; the point is that your docker run command is missing --volumes-from. Offhand I do not know why. Someday when I have time I will try to follow the steps to reproduce and track down the problem.
Thank you for your answer. I had a quick look to the code, and wrote a failing unit test (that you can also find in the gist):
@Test
public void test_cgroup_string_matching_bug() {
final String[] possibleCgroupStrings = new String[] {
"14:name=systemd:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"13:pids:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"12:hugetlb:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"11:net_prio:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"10:perf_event:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"9:net_cls:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"8:freezer:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"7:devices:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"6:memory:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"5:blkio:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"4:cpuacct:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"3:cpu:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"2:cpuset:/docker/45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11",
"1:name=openrc:/docker"
};
for (final String possibleCgroupString : possibleCgroupStrings) {
final Pattern pattern = Pattern.compile(DockerClient.CGROUP_MATCHER_PATTERN);
Matcher matcher = pattern.matcher(possibleCgroupString);
Assert.assertTrue(matcher.find());
Assert.assertEquals("45686cf8ff804c6250e87c02f768f44c63f4d25987e904189ea9156af9f63a11", matcher.group(1));
}
}
As you can see, the pattern is not matching one of the known cgroup formats. I have no idea how to proceed from there. Note that I extracted this pattern from a docker running on top of windows 10. I also have the problem on Linux Mint, I can extract the content of the /proc/self/cgroup on it too if it can help.
I think the test is not correct. When I amended it as follows
diff --git a/src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java b/src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java index 2edee8e..5e665c7 100644 --- a/src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java +++ b/src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java @@ -94,15 +94,17 @@ public class DockerClientTest { public void test_cgroup_string_matching() { final String[] possibleCgroupStrings = new String[] { - "2:cpu:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", - "4:cpuset:/system.slice/docker-3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b.scope" + "2:cpu:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", + "4:cpuset:/system.slice/docker-3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b.scope", + "3:cpu:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", + "2:cpuset:/docker/3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", }; for (final String possibleCgroupString : possibleCgroupStrings) { final Pattern pattern = Pattern.compile(DockerClient.CGROUP_MATCHER_PATTERN); Matcher matcher = pattern.matcher(possibleCgroupString); - Assert.assertTrue(matcher.find()); - Assert.assertEquals("3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", matcher.group(1)); + Assert.assertTrue(possibleCgroupString, matcher.find()); + Assert.assertEquals(possibleCgroupString, "3dd988081e7149463c043b5d9c57d7309e079c5e9290f91feba1cc45a04d6a5b", matcher.group(1)); } }
it passes.
Hello again,
You are absolutely right and I went debugging again to find out...that the issue was on my side 
To give more context for people coming here, it is in fact the docker that was inside my jenkins docker image that was not correct. I installed it using apt-get install from the official jenkins image which was a mistake as I ended up with a docker client version 1.6.2 (communicating with a docker deamon on my host running a version 12...).
The problem finally came from the fact the docker inspect output format was changed for volumes in docker 1.8 (I'm pretty sure this comes from this PR). As a consequence, the plugin did not find any volumes, falling back to a --volume mount.
Installing the latest version of docker using curl -sSL https://get.docker.com/ | sh solved the issue. I guess this means the issue can be closed. If you are interested, I made a pull request to add an error in the logs saying : "The docker version is less than v1.8. Running a 'docker.inside' from inside a container will not work."
There is already a minimum client version specified in the plugin, so this could be updated (or is it the server version which matters?), or we could expand the list of parsable formats. Either way, definitely it would be nice for the plugin to print more helpful diagnostics.
I had a look at the non working and working environment and came up with the following:
working
Client: Version: 1.12.3 API version: 1.24 Server: Version: 1.12.3 API version: 1.24
non working
Client version: 1.6.2 Client API version: 1.18 Server version: 1.12.3 Server API version: 1.24
I guess the client version is the one the plugin should look at then.
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
http://jenkins-ci.org/commit/docker-workflow-plugin/98d2af626cca98f05d358c11431c9f0b5600809b
Log:
Merge pull request #80 from PierreBtz/dev
JENKINS-33632 Add a warning if the detected docker version is less than v1.8
Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/eb142f64eba3...98d2af626cca
Warning added with https://github.com/jenkinsci/docker-workflow-plugin/pull/80, thanks!
Version 1.9 had a related fix. Please do not reopen without complete, self-contained steps to reproduce from scratch.