-
Bug
-
Resolution: Unresolved
-
Blocker
-
Jenkins 2.319.1
Ansible Plugin 1.1
Kubernetes v1.21.3
Jenkins-operator
I have pipeline define as below
$ cat cicd/pipelines/k8s.jenkins #!/usr/bin/env groovy def label = "k8s-${UUID.randomUUID().toString()}" def home = "/home/jenkins" def workspace = "${home}/workspace/build-jenkins-operator" def workdir = "${workspace}/src/github.com/jenkinsci/kubernetes-operator/" podTemplate(label: label, containers: [ containerTemplate(name: 'ansible', image: 'my registry/ansible', ttyEnabled: true, command: 'cat'), // containerTemplate(name: 'ansible', image: 'python', ttyEnabled: true, command: 'cat'), ], ) { node(label) { stage('Run shell') { container('ansible') { // withCredentials([sshUserPrivateKey(credentialsId: 'ssh-private-key', keyFileVariable: 'keyfile')]) { // sh("ANSIBLE_HOST_KEY_CHECKING=False ansible all -i myhost, -m shell -a ls --private-key $keyFile -u devops") // } // sh 'pip install ansible' step([ $class: 'AnsibleAdHocCommandBuilder', ansibleName: 'Run command', inventory: [$class: 'InventoryPath', path: "myhost,"], // inventory: [$class: 'InventoryPath', path: "localhost,"], hostPattern: "all", // module: 'shell', module: 'pause', disableHostKeyChecking: true, command: 'minutes=5', // command: 'ls', forks: 1, additionalParameters: '-vvv', credentialsId: 'ssh-private-key' ]); sleep time: 60 } } } }
It convert the ansible command as {{Executing command: "ansible" "all" "i" "myhost," "-m" "pause" "-a" "minutes=5" "-f" "1" "-private-key" "/tmp/ssh11730288813016418676.key" "-u" "devops" }}
Then it gives error /tmp/ssh11730288813016418676.key file not exits.
Executing sh script inside container ansible of pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn Executing command: "ansible" "all" "-i" "myhost," "-m" "pause" "-a" "minutes=5" "-f" "1" "--private-key" "/tmp/ssh11730288813016418676.key" "-u" "devops" "-vvv" exit ansible 2.10.11 config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.8/site-packages/ansible executable location = /usr/local/bin/ansible python version = 3.8.8 (default, Mar 31 2021, 12:38:11) [GCC 8.3.0] Using /etc/ansible/ansible.cfg as config file Parsed myhost, inventory source with host_list plugin Skipping callback 'default', as we already have a stdout callback. Skipping callback 'minimal', as we already have a stdout callback. Skipping callback 'oneline', as we already have a stdout callback. META: ran handlers <myhost> ESTABLISH SSH CONNECTION FOR USER: devops <myhost> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/ssh11730288813016418676.key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="devops"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/bdd9473e2c myhost '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp `"&& mkdir "` echo /tmp/ansible-tmp-1639596575.6184068-17-61173843506614 `" && echo ansible-tmp-1639596575.6184068-17-61173843506614="` echo /tmp/ansible-tmp-1639596575.6184068-17-61173843506614 `" ) && sleep 0'"'"'' <myhost> (255, b'', b"Warning: Permanently added 'myhost,172.17.203.19' (ECDSA) to the list of known hosts.\r\n\n\n__________ __ __ _________ .____ .___\n\\______ \\_____ ____ | | ____ ________ ____ ____ _____/ |_ ___________ \\_ ___ \\| | | |\n | | _/\\__ \\ _/ ___\\| |/ / | \\____ \\_/ ___\\/ __ \\ / \\ __\\/ __ \\_ __ \\ / \\ \\/| | | |\n | | \\ / __ \\\\ \\___| <| | / |_> > \\__\\ ___/| | \\ | \\ ___/| | \\/ \\ \\___| |___| |\n |______ /(____ /\\___ >__|_ \\____/| __/ \\___ >___ >___| /__| \\___ >__| \\______ /_______ \\___|\n \\/ \\/ \\/ \\/ |__| \\/ \\/ \\/ \\/ \\/ \\/\n\n\n\nno such identity: /tmp/ssh11730288813016418676.key: No such file or directory\r\ndevops@myhost: Permission denied (publickey,password).\r\n") myhost | UNREACHABLE! => { "changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added 'myhost,172.17.203.19' (ECDSA) to the list of known hosts.\r\n\n\n__________ __ __ _________ .____ .___\n\\______ \\_____ ____ | | ____ ________ ____ ____ _____/ |_ ___________ \\_ ___ \\| | | |\n | | _/\\__ \\ _/ ___\\| |/ / | \\____ \\_/ ___\\/ __ \\ / \\ __\\/ __ \\_ __ \\ / \\ \\/| | | |\n | | \\ / __ \\\\ \\___| <| | / |_> > \\__\\ ___/| | \\ | \\ ___/| | \\/ \\ \\___| |___| |\n |______ /(____ /\\___ >__|_ \\____/| __/ \\___ >___ >___| /__| \\___ >__| \\______ /_______ \\___|\n \\/ \\/ \\/ \\/ |__| \\/ \\/ \\/ \\/ \\/ \\/\n\n\n\nno such identity: /tmp/ssh11730288813016418676.key: No such file or directory\r\ndevops@myhost: Permission denied (publickey,password).", "unreachable": true } FATAL: command execution failed hudson.AbortException: Ansible Ad-Hoc command execution failed at org.jenkinsci.plugins.ansible.AnsibleAdHocCommandBuilder.perform(AnsibleAdHocCommandBuilder.java:224) at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:100) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:70) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Ansible report that, it create this /tmp/ssh11730288813016418676.key, then I check where that file is.
From above message, it created k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn pod in our kubernetes cluster. When I did `describe` on that pod, I see there are 2 containers running in this pod.
# kubectl describe pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -n jenkins
Name: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
Namespace: jenkins
Priority: 0
...
...
Status: Running
IP: 192.168.221.84
IPs:
IP: 192.168.221.84
Containers:
ansible:
Container ID: containerd://400d9360a9b3a1415335d6e57148a58d2e64576649323e203a9b66a0de2b048a
Image: myrepo/ansible
Image ID: myrepo/ansible@sha256:6b797d6f80c7ec8c209bec8bde363b84c910a68b4b608c9db6a0411acff3b515
Port: <none>
Host Port: <none>
Command:
cat
State: Running
Started: Wed, 15 Dec 2021 19:45:30 +0000
Ready: True
Restart Count: 0
...
...
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrww8 (ro)
jnlp:
Container ID: containerd://f592bf372bf8fde5baca8a28918b275125fea9b66c9d6ea422172adb29c688ac
Image: jenkins/inbound-agent:4.3-4-jdk11
Image ID: docker.io/jenkins/inbound-agent@sha256:2ee9fb7405f135e15ea3c5efaa605b7c19bc0a93a306deec7a0826a489c1a364
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 15 Dec 2021 19:45:31 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 256Mi
Environment:
JENKINS_SECRET: 0e0f75a0d4e42f0b6c0964f12a1bd1d331d75b0578d8b0e1fd2a7fd9d1175b9d
JENKINS_TUNNEL: jenkins-operator-slave-jenkins-instance.jenkins.svc.cluster.local:50000
JENKINS_AGENT_NAME: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
JENKINS_NAME: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
JENKINS_AGENT_WORKDIR: /home/jenkins/agent
...
...
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrww8 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-mrww8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m13s default-scheduler Successfully assigned jenkins/k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn to myworker
Normal Pulled 3m13s kubelet Container image "stepladder-docker-local.artifacts.rackspace.net/ansiblelint:latest" already present on machine
Normal Created 3m13s kubelet Created container ansible
Normal Started 3m13s kubelet Started container ansible
Normal Pulled 3m13s kubelet Container image "jenkins/inbound-agent:4.3-4-jdk11" already present on machine
Normal Created 3m13s kubelet Created container jnlp
Normal Started 3m12s kubelet Started container jnlp
In the console log it has log line as Executing sh script inside container ansible of pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn means ansible command will run in ansible container of pod.
When I check the /tmp directory in ansible pod, I didn't see any files in that.
# kubectl exec -i -t k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -c ansible -n jenkins -- ls -al /tmp/
total 8
drwxrwxrwt 1 root root 4096 Dec 15 19:45 .
drwxr-xr-x 1 root root 4096 Dec 15 19:45 ..
While as per the ansible command, there should be file /tmp/ssh11730288813016418676.key exists in the /tmp folder. Then I check the jnlp container.
# kubectl exec -i -t k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -c jnlp -n jenkins -- ls -al /tmp/
total 20
drwxrwxrwt 1 root root 4096 Dec 15 19:45 .
drwxr-xr-x 1 root root 4096 Dec 15 19:45 ..
drwxr-xr-x 2 jenkins jenkins 4096 Dec 15 19:45 hsperfdata_jenkins
drwxr-xr-x 2 root root 4096 Mar 31 2020 hsperfdata_root
-r-------- 1 jenkins jenkins 3243 Dec 15 19:45 ssh11730288813016418676.key
I can see the sshXXXX.key file is created in jnlp container.
When ansible command runs on ansible container, it is not able to find the sshXXXXX.key file on that container, and it gives error that no such file or directory.
