-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.462.1
durable-task-plugin (568.v8fb_5c57e8417)
kubernetes-plugin (4304.v1b_39d4f98210)
remoting (3248.3250.v3277a_8e88c9b_)
Kubernetes (AWS EKS 1.30)
Windows Server 2019 Core (EKS Optimized 1.30)
JNLP Image (jenkins/inbound-agent:3248.3250.v3277a_8e88c9b_-1-jdk21-windowsservercore-ltsc2019)
Shell Image (jenkins/inbound-agent:3248.3250.v3277a_8e88c9b_-1-jdk21-windowsservercore-ltsc2019)
Jenkins 2.462.1 durable-task-plugin (568.v8fb_5c57e8417) kubernetes-plugin (4304.v1b_39d4f98210) remoting (3248.3250.v3277a_8e88c9b_) Kubernetes (AWS EKS 1.30) Windows Server 2019 Core (EKS Optimized 1.30) JNLP Image (jenkins/inbound-agent:3248.3250.v3277a_8e88c9b_-1-jdk21-windowsservercore-ltsc2019) Shell Image (jenkins/inbound-agent:3248.3250.v3277a_8e88c9b_-1-jdk21-windowsservercore-ltsc2019)
We're using the Kubernetes Plugin to spin up ephemeral agents (Windows containers) on a Kubernetes cluster hosted in AWS EKS and execute powershell commands.
Our Jenkins instance used to test has the following Java options enabled for the Durable Task plugin:
-Dorg.jenkinsci.plugins.durabletask.WindowsBatchScript.USE_BINARY_WRAPPER=true -Dorg.jenkinsci.plugins.durabletask.PowershellScript.USE_BINARY_WRAPPER=true
It appears the powershell and batch commands generated by the Durable Task plugin that are then executed by the Kubernetes plugin in the container are invalid when using the Durable Task binary wrapper. Jenkins jobs will attempt to run these invalid commands and hang indefinitely.
When USE_BINARY_WRAPPER is set to false, the commands generated by the Durable Task plugin are valid and our jobs succeed for both powershell and batch scripts.
Example Powershell:
From the Jenkins logs we can see the command generated by the Durable Task plugin that the Kubernetes plugin will then attempt to execute in the container:
Executing command: /home/jenkins/agent/home/jenkins/agent/caches/durable-task/durable_task_monitor_568.v8fb_5c57e8417_win_64.exe -daemon -executable=powershell -args=-NoProfile, -NonInteractive, -ExecutionPolicy, Bypass, -Command, [Console]::OutputEncoding = [Text.Encoding]::UTF8; [Console]::InputEncoding = [System.Text.Encoding]::UTF8; & {try {& \"C:\home\jenkins\agent\workspace\test@tmp\durable-d5d2a3eb\powershellScript.ps1\"} catch {throw}; exit $LASTEXITCODE} -controldir=C:\home\jenkins\agent\workspace\test@tmp\durable-d5d2a3eb -result=C:\home\jenkins\agent\workspace\test@tmp\durable-d5d2a3eb\jenkins-result.txt -log=C:\home\jenkins\agent\workspace\test@tmp\durable-d5d2a3eb\jenkins-log.txt
This command is invalid and isn't interpreted correctly by the shell in the container (missing quotes around args, using wrong escapes for the script path):
At line:1 char:158 + ... 7_win_64.exe -daemon -executable=powershell -args=-NoProfile, -NonInt ... + ~ Missing argument in parameter list. At line:1 char:175 + ... mon -executable=powershell -args=-NoProfile, -NonInteractive, -Execut ... + ~ Missing argument in parameter list. At line:1 char:193 + ... wershell -args=-NoProfile, -NonInteractive, -ExecutionPolicy, Bypass, ... + ~ Missing argument in parameter list. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : MissingArgument
At line:1 char:119 + ... ; & {try {& \C:\home\jenkins\agent\workspace\test@tmp\d... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (\C:\home\jenkin...hellScript.ps1\:String) [ ], ParentContainsErrorRecordException + FullyQualifiedErrorId : CommandNotFoundException
Example Batch:
Similar to the Powershell example, commands generated by the Durable Task plugin when using the binary wrapper are invalid for running Batch scripts:
Executing command: /home/jenkins/agent/home/jenkins/agent/caches/durable-task/durable_task_monitor_568.v8fb_5c57e8417_win_64.exe -daemon -executable=cmd -args=/C call \"C:\home\jenkins\agent\workspace\test@tmp\durable-d0395bff\jenkins-main.bat\" -controldir=C:\home\jenkins\agent\workspace\test@tmp\durable-d0395bff -result=C:\home\jenkins\agent\workspace\test@tmp\durable-d0395bff\jenkins-result.txt -log=C:\home\jenkins\agent\workspace\test@tmp\durable-d0395bff\jenkins-log.txt
The following required flags are missing: -controldir -result -log
Our pipeline configuration is basically identical to the samples that can be found in the Kubernetes plugin GitHub repository:
When USE_BINARY_WRAPPER is set to false, the commands generated by the Durable Task plugin are valid and our jobs succeed for both powershell and batch scripts.
However, we would like to be able to continue and use the binary wrapper on our Jenkins instances due to other issues, such as JENKINS-27617.
Please advise. Any other recommendations or workarounds would also be greatly appreciated!