Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47759

Updating podTemplate does not immediately come into effect.

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      When updating a PodTemplate (specifically a containerTemplate) through a jenkinsfile change, the change is not immediately put into effect.

      This seems to be some kind of caching issue, in older versions you could see this through the "Configure System" which would have the old template still populated. Is it especially apparent when someone is on a branch improving the build , but "old" template builds are still happening on master with the same label. In this situation it seems a bit of a race condition as to which version is picked up in the job.

      Workaround;
      We have resorted to changing the template label to force a cache bust. e.i
      label: 'GitPod1' ... label: 'GitPod2' etc. which works reliably.

      P.S Finally getting around to recording some bugs that I have known about for a while, sorry for the spam

        Attachments

          Issue Links

            Activity

            Hide
            vrobert78 Vincent Robert added a comment -

            Hello,
            I had the same problem with this kind of cache.
            I don't why, but my podTemplate retarted in loop (command was "/bin/sh -c" instead of "jenkins-slave").

             

            kubectl logs jenkins-slave-tbfxp-5n9g1
            jenkins-slave-tbfxp-5n9g1: 1: jenkins-slave-tbfxp-5n9g1: 1ad85d1550d28440a0b482989b8c607bcfc3b187c2e5d7f8640ee8d2c4332a96: not found

            So i changed my podTemplate, try a lof of things, but nothing happened, it was always in loop.

            I changed the label and this time my new parameters were in effect.

            Everything is fine by now.

            Show
            vrobert78 Vincent Robert added a comment - Hello, I had the same problem with this kind of cache. I don't why, but my podTemplate retarted in loop (command was "/bin/sh -c" instead of "jenkins-slave").   kubectl logs jenkins-slave-tbfxp-5n9g1 jenkins-slave-tbfxp-5n9g1: 1: jenkins-slave-tbfxp-5n9g1: 1ad85d1550d28440a0b482989b8c607bcfc3b187c2e5d7f8640ee8d2c4332a96: not found So i changed my podTemplate, try a lof of things, but nothing happened, it was always in loop. I changed the label and this time my new parameters were in effect. Everything is fine by now.
            Hide
            csanchez Carlos Sanchez added a comment -

            Without Jenkins logs there's not enough info to find out
            https://github.com/jenkinsci/kubernetes-plugin/blob/master/README.md#debugging

            Show
            csanchez Carlos Sanchez added a comment - Without Jenkins logs there's not enough info to find out https://github.com/jenkinsci/kubernetes-plugin/blob/master/README.md#debugging
            Hide
            jozue_noon Witold Konior added a comment - - edited

            Our scenario is as following:

            Jenkins docker container 2.101

            Our pod have 3 containers, where one is `postgresql`. Pipeline was working for long time (2 months) without changes. Suddenly we got `connection refused` to postgres (hint: job right before this one was failed due to compilation error in our code in other container). Then we were trying to add some `envVars` into postgres container to make sure that postgres have everything set properly, but those changes were not reflected into running pod.

            Finally figure out that stored `podTemplate` was overtaking declarative one. Not only by discarding `envVars` but also magically changing command and arguments for container, so we were running `/bin/bash -c cat` instead original postgres one.

            Carlos Sanchez I was digging through logs but could not find anything relevant, however we had no log recorder set by now. If this occurs again I promise to drop logs here.

            Show
            jozue_noon Witold Konior added a comment - - edited Our scenario is as following: Jenkins docker container 2.101 Our pod have 3 containers, where one is `postgresql`. Pipeline was working for long time (2 months) without changes. Suddenly we got `connection refused` to postgres (hint: job right before this one was failed due to compilation error in our code in other container). Then we were trying to add some `envVars` into postgres container to make sure that postgres have everything set properly, but those changes were not reflected into running pod. Finally figure out that stored `podTemplate` was overtaking declarative one. Not only by discarding `envVars` but also magically changing command and arguments for container, so we were running `/bin/bash -c cat` instead original postgres one. Carlos Sanchez I was digging through logs but could not find anything relevant, however we had no log recorder set by now. If this occurs again I promise to drop logs here.
            Hide
            vincentheet Vincent Heet added a comment -

            We are having a similar issue with Jenkins 2.105 and Kubernetes plugin 1.2. Changing the podTemplate in our Jenkinsfile did not change anything on pull request builds. We saw this issue appear when we changed the docker tag of the docker image to a new version (myimage:v02 to myimage:v03). I stumbled onto this issue and enabled logging for okhttp3 and this plugin. Examining the logs I see that okhttp3 is sending the wrong (myimage:v02) pod configuration over http to the Kubernetes API. This configuration is not coming from the Jenkinsfile or from the PodTemplate defined in the Jenkins GUI. Looking a bit further I found a file called "kubernetes-podtemplates.xml" on our Jenkins master in "/var/jenkins_home" The contents of this file contained an old/wrong slave configuration. Interesting fact was that the name of this template as specified in:

             

            <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
            <name>jenkins-slave-3zf30</name>

             

            was mentioning the same name of the wrong pod config that was send using okhttp3 to the Kubernetes API.

            When triggering a new PR build using "Build Now" in the Jenkins GUI the file "kubernetes-podtemplates.xml" was edited and a new pod template was added. So there were now 2 podTemplates in the configuration file:

             

            <java.util.concurrent.CopyOnWriteArrayList>
            <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
            <name>jenkins-slave-3zf30</name>
            <label>my-label</label>
            <containers>
            <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
            <name>jdk8-maven3</name>
            <image>url.io/project1/myimage:v02</image>
            </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
            <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
            <name>jenkins-slave-qlwck</name>
            <label>my-label</label>
            <containers>
            <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
            <name>jdk8-maven3</name>
            <image>url.io/project1/myimage:v03</image>
            </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
            </java.util.concurrent.CopyOnWriteArrayList>
            

             

             

            It seems like the pod that is created is the first (incorrect) one instead of the just added second (correct) one. After the build finishes it removes the correct podTemplate from the xml file but the incorrect one is kept around and therefore this bug occurs for consecutive builds. Probably the incorrect template is matched based on the label "my-label" which is the same for both podTemplate's and that explains why changing the label works as Lars Lawoko mentioned.

            Removing kubernetes-podtemplates.xml from /var/jenkins_home dit not fix my issue. After deletion and triggering a build the incorrect and correct podTemplate's are added to the xml file. And the build is run with the incorrect podTemplate.

            Hope this helps to reproduce the issue.

             

            Show
            vincentheet Vincent Heet added a comment - We are having a similar issue with Jenkins 2.105 and Kubernetes plugin 1.2. Changing the podTemplate in our Jenkinsfile did not change anything on pull request builds. We saw this issue appear when we changed the docker tag of the docker image to a new version (myimage:v02 to myimage:v03). I stumbled onto this issue and enabled logging for okhttp3 and this plugin. Examining the logs I see that okhttp3 is sending the wrong (myimage:v02) pod configuration over http to the Kubernetes API. This configuration is not coming from the Jenkinsfile or from the PodTemplate defined in the Jenkins GUI. Looking a bit further I found a file called "kubernetes-podtemplates.xml" on our Jenkins master in "/var/jenkins_home" The contents of this file contained an old/wrong slave configuration. Interesting fact was that the name of this template as specified in:   <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <name>jenkins-slave-3zf30</name>   was mentioning the same name of the wrong pod config that was send using okhttp3 to the Kubernetes API. When triggering a new PR build using "Build Now" in the Jenkins GUI the file "kubernetes-podtemplates.xml" was edited and a new pod template was added. So there were now 2 podTemplates in the configuration file:   <java.util.concurrent.CopyOnWriteArrayList> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <name>jenkins-slave-3zf30</name> <label>my-label</label> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>jdk8-maven3</name> <image>url.io/project1/myimage:v02</image> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <name>jenkins-slave-qlwck</name> <label>my-label</label> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>jdk8-maven3</name> <image>url.io/project1/myimage:v03</image> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> </java.util.concurrent.CopyOnWriteArrayList>     It seems like the pod that is created is the first (incorrect) one instead of the just added second (correct) one. After the build finishes it removes the correct podTemplate from the xml file but the incorrect one is kept around and therefore this bug occurs for consecutive builds. Probably the incorrect template is matched based on the label "my-label" which is the same for both podTemplate's and that explains why changing the label works as Lars Lawoko mentioned. Removing kubernetes-podtemplates.xml from /var/jenkins_home dit not fix my issue. After deletion and triggering a build the incorrect and correct podTemplate's are added to the xml file. And the build is run with the incorrect podTemplate. Hope this helps to reproduce the issue.  
            Hide
            csanchez Carlos Sanchez added a comment -

            jenkins will pick a pod template based on the label, if you have more than one pod template serving the same label you may get the wrong one.
            You should use uuids as labels to avoid that

            # this guarantees the node will use this template
            def label = "mypod-${UUID.randomUUID().toString()}"
            podTemplate(label: label) {
                node(label) {
            

            I have never seen the kubernetes-podtemplates.xml file

            Show
            csanchez Carlos Sanchez added a comment - jenkins will pick a pod template based on the label, if you have more than one pod template serving the same label you may get the wrong one. You should use uuids as labels to avoid that # this guarantees the node will use this template def label = "mypod-${UUID.randomUUID().toString()}" podTemplate(label: label) { node(label) { I have never seen the kubernetes-podtemplates.xml file
            Show
            csanchez Carlos Sanchez added a comment - See https://github.com/jenkinsci/kubernetes-plugin/pull/281
            Hide
            vincentheet Vincent Heet added a comment -

            Thanks for the hint of using a UUID Carlos Sanchez that will definitely fix the issue. I'm a bit confused by you saying you have never seen the kubernetes-podtemplate.xml. It is mentioned in the code here: https://github.com/jenkinsci/kubernetes-plugin/blob/023f8e252fab8896e0ade1fb1b233433f20ce7dc/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/PodTemplateMap.java#L138

            Show
            vincentheet Vincent Heet added a comment - Thanks for the hint of using a UUID Carlos Sanchez  that will definitely fix the issue. I'm a bit confused by you saying you have never seen the kubernetes-podtemplate.xml. It is mentioned in the code here:  https://github.com/jenkinsci/kubernetes-plugin/blob/023f8e252fab8896e0ade1fb1b233433f20ce7dc/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/PodTemplateMap.java#L138
            Hide
            csanchez Carlos Sanchez added a comment -

            ah, kubernetes-podtemplate.xml was just added to persist the pipeline defined templates. It should always match what's in the pipeline

            Show
            csanchez Carlos Sanchez added a comment - ah, kubernetes-podtemplate.xml was just added to persist the pipeline defined templates. It should always match what's in the pipeline
            Hide
            csanchez Carlos Sanchez added a comment -

            This is probably fixed at least partially with https://github.com/jenkinsci/kubernetes-plugin/pull/285

            Show
            csanchez Carlos Sanchez added a comment - This is probably fixed at least partially with https://github.com/jenkinsci/kubernetes-plugin/pull/285
            Hide
            inquisitive anshu pitlia added a comment -

            Carlos Sanchez Do you still think I should use UUID or the bug is fixed? Because I faced the same issue last night. It picked up a stale template with the same name. Where does it store/cache the template? What if I completely override the template file, but keep the label same? It doesn't work for me as of now, every time I change a template I have to change the label. 

            Show
            inquisitive anshu pitlia added a comment - Carlos Sanchez Do you still think I should use UUID or the bug is fixed? Because I faced the same issue last night. It picked up a stale template with the same name. Where does it store/cache the template? What if I completely override the template file, but keep the label same? It doesn't work for me as of now, every time I change a template I have to change the label. 

              People

              Assignee:
              vlatombe Vincent Latombe
              Reporter:
              larslawoko Lars Lawoko
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: