Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72592

Unable to spawn new Docker Cloud agents after updating their template

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • docker-plugin
    • None
    • 1.6

      I have the latest version of Jenkins Master 2.441 running inside a container using the image. I have also configured a Docker cloud using docker-plugin, version 1.5.

      Starting from Jenkins 2.437, every time that I make a change to the Docker Agent templates and try to run a job, no new containers are created and the following comes up in the Jenkins Master logs:

      jenkins-1  | 2024-01-23 02:38:34.843+0000 [id=151]      SEVERE  c.n.j.p.docker.DockerCloud$1#run: Error in provisioning; template='DockerTemplate{configVersion=2, labelString='<redacted>', connector=DockerComputerJNLPConnector{user='jenkins', entryPointArguments=[-url, ${JENKINS_URL}, -secret, ${JNLP_SECRET}, -name, ${NODE_NAME}]}, instanceCap=10, mode=NORMAL, retentionStrategy=com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy@29, dockerTemplateBase=DockerTemplateBase{image='<redacted>/jenkins-inbound-agent:jdk21_latest', pullCredentialsId='ecr:<redacted>', environment=[TZ=Pacific/Auckland], bindAllPorts=false, cpuPeriod=0, cpuQuota=0, privileged=false, tty=false}, removeVolumes=false, stopTimeout=10, pullStrategy=PULL_LATEST, pullTimeout=600, disabled=BySystem,0 ms,4 min 59 sec,Template provisioning failed., name='inbound-agent-jdk21'}' for cloud='docker'
      jenkins-1  | java.lang.NullPointerException: Cannot read field "tunnel" because "this.jnlpLauncher" is null
      jenkins-1  |    at io.jenkins.docker.connector.DockerComputerJNLPConnector.beforeContainerCreated(DockerComputerJNLPConnector.java:201)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerTemplate.doProvisionNode(DockerTemplate.java:729)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerTemplate.provisionNode(DockerTemplate.java:682)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:414)
      jenkins-1  |    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      jenkins-1  |    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      jenkins-1  |    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
      jenkins-1  |    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
      jenkins-1  |    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
      jenkins-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
      jenkins-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
      jenkins-1  |    at java.base/java.lang.Thread.run(Thread.java:1583)
      jenkins-1  | 2024-01-23 02:38:34.844+0000 [id=151]      WARNING j.u.ErrorLoggingExecutorService#lambda$wrap$0
      jenkins-1  | java.lang.NullPointerException: Cannot read field "tunnel" because "this.jnlpLauncher" is null
      jenkins-1  |    at io.jenkins.docker.connector.DockerComputerJNLPConnector.beforeContainerCreated(DockerComputerJNLPConnector.java:201)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerTemplate.doProvisionNode(DockerTemplate.java:729)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerTemplate.provisionNode(DockerTemplate.java:682)
      jenkins-1  |    at com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:414)
      jenkins-1  |    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      jenkins-1  |    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      jenkins-1  |    at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
      jenkins-1  |    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
      jenkins-1  |    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
      jenkins-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
      jenkins-1  |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
      jenkins-1  |    at java.base/java.lang.Thread.run(Thread.java:1583) 

      Restarting the Jenkins Master preserves the new configuration and fixes the issue. New containers are spawned normally.

      Here's a excerpt of my JCasC configuration for the docker-cloud plugin (logs above show what happened after editing the container template's entrypoint to remove the TUNNEL* variables):

         clouds:
        - docker:
            dockerApi:
              connectTimeout: 60
              dockerHost:
                uri: "tcp://socket-proxy:2375"
              readTimeout: 60
            name: "docker"
            templates:
            - connector:
                jnlp:
                  entryPointArgumentsString: |-
                    ^${TUNNEL_ARG}
                    ^${TUNNEL}
                    -url
                    ^${JENKINS_URL}
                    -secret
                    ^${JNLP_SECRET}
                    -name
                    ^${NODE_NAME}
                  jnlpLauncher:
                    webSocket: true
                    workDirSettings:
                      disabled: false
                      failIfWorkDirIsMissing: false
                      internalDir: "remoting"
                  user: "jenkins"
              dockerTemplateBase:
                cpuPeriod: 0
                cpuQuota: 0
                environment:
                - "TZ=Pacific/Auckland"
                environmentsString: |-
                  TZ=Pacific/Auckland
                image: "<redacted>/jenkins-inbound-agent:jdk21_latest"
                pullCredentialsId: "ecr:<redacted>"
              instanceCapStr: "10"
              labelString: "<redacted>"
              name: "inbound-agent-jdk21"
              pullTimeout: 600

      this.jnlpLauncher only appears to be referenced here. Looking at the Jenkins changelog, it could be related to PR-8762 but I am not sure if that's the case.

          [JENKINS-72592] Unable to spawn new Docker Cloud agents after updating their template

          I have very similar configuration and for me even restart doesn't help.

          Kacper Piechota added a comment - I have very similar configuration and for me even restart doesn't help.

          Mark Waite added a comment -

          eburgueno and koziol could you try the pre-release build that is available at https://ci.jenkins.io/job/Plugins/job/docker-plugin/job/PR-1049/4/artifact/io/jenkins/docker/docker-plugin/1.6-rc1184.419d1ddd0e68/docker-plugin-1.6-rc1184.419d1ddd0e68.hpi . You can install it on your Jenkins controller from the "Advanced" page of the plugin manager. There is a URL field there where you paste that URL.

          I was able to see the null pointer exception with the released version of the plugin and am able to allocate agents with the pre-release of the bug fix.

          Mark Waite added a comment - eburgueno and koziol could you try the pre-release build that is available at https://ci.jenkins.io/job/Plugins/job/docker-plugin/job/PR-1049/4/artifact/io/jenkins/docker/docker-plugin/1.6-rc1184.419d1ddd0e68/docker-plugin-1.6-rc1184.419d1ddd0e68.hpi . You can install it on your Jenkins controller from the "Advanced" page of the plugin manager. There is a URL field there where you paste that URL. I was able to see the null pointer exception with the released version of the plugin and am able to allocate agents with the pre-release of the bug fix.

          Eric added a comment -

          I just updated to docker-plugin 1.6 and the issue seems to have been resolved. I did have to modify my JCasC as follows:

                 templates:
                - connector:
                    jnlp:
                      entryPointArgumentsString: |-
                        -url
                        ^${JENKINS_URL}
                        -secret
                        ^${JNLP_SECRET}
                        -name
                        ^${NODE_NAME}
                      user: "jenkins"

          Thank you!

          PS: This is not relevant to this issue and I don't think it merits one, but I thought I'd mention that the official inbound-agent image has deprecated passing secret and name as positional parameters, so I have to override the defaults with a custom entrypoint:

          WARNING: Providing the secret and agent name as positional arguments is deprecated; use "-secret" and "-name" instead. 

          Eric added a comment - I just updated to docker-plugin 1.6 and the issue seems to have been resolved. I did have to modify my JCasC as follows:       templates:       - connector:           jnlp:             entryPointArgumentsString: |-               -url               ^${JENKINS_URL}               -secret               ^${JNLP_SECRET}               -name               ^${NODE_NAME}             user: "jenkins" Thank you! PS: This is not relevant to this issue and I don't think it merits one, but I thought I'd mention that the official inbound-agent image has deprecated passing secret and name as positional parameters, so I have to override the defaults with a custom entrypoint: WARNING: Providing the secret and agent name as positional arguments is deprecated; use "-secret" and "-name" instead.

            markewaite Mark Waite
            eburgueno Eric
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: