Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24938

On restart Jenkins rewrites some config.xml values

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • ec2-plugin
    • Ubuntu 1.582

      I have been seeing this behavior for a while now. I have a standard ubuntu installation using the official package. Every time I reboot the jenkins process to update a plugin of a version my Views and Slave Port configuration are overridden.

      For instance my original config.xml includes the following.

      views>
      <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
      </hudson.model.AllView>
      <listView>
      <owner class="hudson" reference="../../.."/>
      <name>iOS</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
      <jobNames>
      <comparator class="hudson.util.CaseInsensitiveComparator"/>
      </jobNames>
      <jobFilters/>
      <columns>
      <hudson.views.StatusColumn/>
      <hudson.views.WeatherColumn/>
      <hudson.views.JobColumn/>
      <hudson.views.LastSuccessColumn/>
      <hudson.views.LastFailureColumn/>
      <hudson.views.LastDurationColumn/>
      <hudson.views.BuildButtonColumn/>
      </columns>
      <includeRegex>(?i).IOS-.</includeRegex>
      <recurse>false</recurse>
      </listView>
      </views>
      <primaryView>All</primaryView>
      <slaveAgentPort>8098</slaveAgentPort>

      After a restart that section of the config.xml is reduced to.
      <views>
      <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
      </hudson.model.AllView>
      </views>
      <primaryView>All</primaryView>
      <slaveAgentPort>0</slaveAgentPort>

      I have watched the jenkins.log file closely and seen no errors or warnings in regard to why these two things specifically are being reset. What is also strange is that the rest of the core config is persisted and remains unchanged.

          [JENKINS-24938] On restart Jenkins rewrites some config.xml values

          Daniel Beck added a comment -

          Are you manually modifying the config.xml while Jenkins is running, or is this change only your observation because your view config is reset upon restart?

          Daniel Beck added a comment - Are you manually modifying the config.xml while Jenkins is running, or is this change only your observation because your view config is reset upon restart?

          Joseph Hughes added a comment -

          I am not manually modifying the config.xml while jenkins is running. I backed up the config before I did a restart so I could compare the two files. The result is what you see above.

          Joseph Hughes added a comment - I am not manually modifying the config.xml while jenkins is running. I backed up the config before I did a restart so I could compare the two files. The result is what you see above.

          Joseph Hughes added a comment -

          Also, when I make the changes through the UI then view the xml I see what I would expect to be there, but when I restart it is gone again. It does seem to be limited to just these two config values, Views and the Slave port. Everything else persists correctly.

          Joseph Hughes added a comment - Also, when I make the changes through the UI then view the xml I see what I would expect to be there, but when I restart it is gone again. It does seem to be limited to just these two config values, Views and the Slave port. Everything else persists correctly.

          Daniel Beck added a comment -

          Is there a file named init.groovy, or a folder init.groovy.d in the Jenkins home directory? If so, what are the contents?

          Is security in Jenkins enabled?

          Daniel Beck added a comment - Is there a file named init.groovy, or a folder init.groovy.d in the Jenkins home directory? If so, what are the contents? Is security in Jenkins enabled?

          Joseph Hughes added a comment -

          Security is enabled. I am using Jenkins user database along with the Role Strategy Plugin. https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Plugin

          I do not see any init.groovy or a init.groovy.d folder

          Joseph Hughes added a comment - Security is enabled. I am using Jenkins user database along with the Role Strategy Plugin. https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Plugin I do not see any init.groovy or a init.groovy.d folder

          Daniel Beck added a comment -

          Is JobConfigHistory plugin installed, and if so, does it record changes to the global config.xml? Which user is specified?

          Daniel Beck added a comment - Is JobConfigHistory plugin installed, and if so, does it record changes to the global config.xml? Which user is specified?

          Joseph Hughes added a comment -

          It is not installed... but if it would be helpful I could install it.

          Joseph Hughes added a comment - It is not installed... but if it would be helpful I could install it.

          Oleg Nenashev added a comment -

          @Joseph
          Any updates?

          Oleg Nenashev added a comment - @Joseph Any updates?

          AmberKorey Pairing added a comment - - edited

          We are experiencing a similar issue. We have recently upgraded from Jenkins 1.480.3 to 1.565.3 and it seems once any change is made to the global config.xml file by jenkins, it looses all configurations from the views section forward including our Global Environment Variables section.

          We have the JobConfigHistory plugin installed and it only shows that SYSTEM is making the changes to the config file.

          We have been working on this issue for several days now and have spun up a couple of test environments. We cannot seem to reproduce this issue consistently in our test environments.

          We have tailed stderr and stdout and have observed no exceptions on startup or restart of Jenkins

          Here are the steps to reproduce our issue (in one environment anyways)

          • Take a config file from a jenkins that is not showing the views and copy it into test environment master-workspace
          • Start up jenkins and verify that Views and Environment variables are not displayed in the UI
          • Verify that Views and Environment variables are still in the config.xml file
          • Change the System Message of the Jenkins Server in Jenkins->Configuration
          • Click Save
          • Observe that the Views section and Environment Variable section of config.xml are now clobbered. (No Views, No Environment Variables)

          Any thoughts?

          AmberKorey Pairing added a comment - - edited We are experiencing a similar issue. We have recently upgraded from Jenkins 1.480.3 to 1.565.3 and it seems once any change is made to the global config.xml file by jenkins, it looses all configurations from the views section forward including our Global Environment Variables section. We have the JobConfigHistory plugin installed and it only shows that SYSTEM is making the changes to the config file. We have been working on this issue for several days now and have spun up a couple of test environments. We cannot seem to reproduce this issue consistently in our test environments. We have tailed stderr and stdout and have observed no exceptions on startup or restart of Jenkins Here are the steps to reproduce our issue (in one environment anyways) Take a config file from a jenkins that is not showing the views and copy it into test environment master-workspace Start up jenkins and verify that Views and Environment variables are not displayed in the UI Verify that Views and Environment variables are still in the config.xml file Change the System Message of the Jenkins Server in Jenkins->Configuration Click Save Observe that the Views section and Environment Variable section of config.xml are now clobbered. (No Views, No Environment Variables) Any thoughts?

          Daniel Beck added a comment -

          amberkorey: I suggest checking 'Manage Old Data' to make sure no plugins are misbehaving.

          As your observed behavior is quite different, it appears to be unrelated to the issue reported and investigated here. Please ask e.g. in IRC or on the jenkinsci-users list for troubleshooting advice.

          Daniel Beck added a comment - amberkorey : I suggest checking 'Manage Old Data' to make sure no plugins are misbehaving. As your observed behavior is quite different, it appears to be unrelated to the issue reported and investigated here. Please ask e.g. in IRC or on the jenkinsci-users list for troubleshooting advice.

          Joseph Hughes added a comment -

          I am still seeing the same issue. The comment above is almost exactly what I see. System is making the cahnge

          Joseph Hughes added a comment - I am still seeing the same issue. The comment above is almost exactly what I see. System is making the cahnge

          @Joseph, which Jenkins version are you using?

          AmberKorey Pairing added a comment - @Joseph, which Jenkins version are you using?

          Update:

          We ran the 'Manage Old Data' tool and it was complaining about a tag in our EC2 Slave node (hudson.plugins.ec2.EC2Slave). The exact error message was:

          ConversionException: No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' ---- Debugging information ---- message : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' cause-exception : com.thoughtworks.xstream.converters.reflection.MissingFieldException cause-message : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' class : com.amazonaws.internal.ListWithAutoConstructFlag required-type : com.amazonaws.internal.ListWithAutoConstructFlag converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter path : /hudson/slaves/hudson.plugins.ec2.EC2Slave/lastFetchInstance/tags/com.amazonaws.internal.ListWithAutoConstructFlag/default/autoConstruct line number : 2431 -------------------------------, MissingFieldException: No field 'securityGroups' found in class 'hudson.model.Hudson', MissingFieldException: No field 'hypervisor' found in class 'hudson.model.Hudson', MissingFieldException: No field 'ebsOptimized' found in class 'hudson.model.Hudson'

          We took out the tag and now empty surrounding tags from all ec2 slave nodes listed in config.xml and restart Jenkins. Our views and our environment variables were back in the UI.

          So now we are tracking down how the autoConstruct tag made it into our config.xml in the first place.

          It looks like there is an xml section called lastFetchInstance that is present in our bad configs but not our good configs. We are not sure what creates that section at this point but will keep updating the thread as things progress.

          AmberKorey Pairing added a comment - Update: We ran the 'Manage Old Data' tool and it was complaining about a tag in our EC2 Slave node (hudson.plugins.ec2.EC2Slave). The exact error message was: ConversionException: No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' ---- Debugging information ---- message : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' cause-exception : com.thoughtworks.xstream.converters.reflection.MissingFieldException cause-message : No field 'autoConstruct' found in class 'com.amazonaws.internal.ListWithAutoConstructFlag' class : com.amazonaws.internal.ListWithAutoConstructFlag required-type : com.amazonaws.internal.ListWithAutoConstructFlag converter-type : com.thoughtworks.xstream.converters.reflection.SerializableConverter path : /hudson/slaves/hudson.plugins.ec2.EC2Slave/lastFetchInstance/tags/com.amazonaws.internal.ListWithAutoConstructFlag/default/autoConstruct line number : 2431 -------------------------------, MissingFieldException: No field 'securityGroups' found in class 'hudson.model.Hudson', MissingFieldException: No field 'hypervisor' found in class 'hudson.model.Hudson', MissingFieldException: No field 'ebsOptimized' found in class 'hudson.model.Hudson' We took out the tag and now empty surrounding tags from all ec2 slave nodes listed in config.xml and restart Jenkins. Our views and our environment variables were back in the UI. So now we are tracking down how the autoConstruct tag made it into our config.xml in the first place. It looks like there is an xml section called lastFetchInstance that is present in our bad configs but not our good configs. We are not sure what creates that section at this point but will keep updating the thread as things progress.

          Daniel Beck added a comment -

          Assuming the culprit is EC2 for now.

          Daniel Beck added a comment - Assuming the culprit is EC2 for now.

          Joseph Hughes added a comment -

          I am using Jenkins 1.582. I also have the ec2 plugin installed but do not have anything come up in the managed old data. For me I notice that the Views and the Jenkins Slave port are the two pieces of data that consistently disappear on reboots. I have started uninstalling any non essential plugins but have not seen any changes yet. I need the ec2 plugin so unfortunately uninstalling it is not an option.

          Joseph Hughes added a comment - I am using Jenkins 1.582. I also have the ec2 plugin installed but do not have anything come up in the managed old data. For me I notice that the Views and the Jenkins Slave port are the two pieces of data that consistently disappear on reboots. I have started uninstalling any non essential plugins but have not seen any changes yet. I need the ec2 plugin so unfortunately uninstalling it is not an option.

          Joseph Hughes added a comment -

          I am using version 1.24 of the ec2 plugin.

          Joseph Hughes added a comment - I am using version 1.24 of the ec2 plugin.

          Joseph Hughes added a comment -

          <com.amazonaws.internal.ListWithAutoConstructFlag>
          <default>
          <autoConstruct>true</autoConstruct>
          </default>
          </com.amazonaws.internal.ListWithAutoConstructFlag>
          </blockDeviceMappings>

          This is where I see autoConstruct in my config. I see the autoConstruct tag all over the ec2 config. Of course this is not causing any errors in my config to come up in "Manage Old Data" and the xml seems valid.

          Joseph Hughes added a comment - <com.amazonaws.internal.ListWithAutoConstructFlag> <default> <autoConstruct>true</autoConstruct> </default> </com.amazonaws.internal.ListWithAutoConstructFlag> </blockDeviceMappings> This is where I see autoConstruct in my config. I see the autoConstruct tag all over the ec2 config. Of course this is not causing any errors in my config to come up in "Manage Old Data" and the xml seems valid.

          Code changed in jenkins
          User: Francis Upton
          Path:
          src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java
          http://jenkins-ci.org/commit/ec2-plugin/d823b5e5da73f2ea9f57f11d890246df5f6c5110
          Log:
          Merge pull request #112 from jglick/transient-lastFetchInstance

          JENKINS-24938 Various fields should be transient

          Compare: https://github.com/jenkinsci/ec2-plugin/compare/7ebb4886cc05...d823b5e5da73

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2AbstractSlave.java http://jenkins-ci.org/commit/ec2-plugin/d823b5e5da73f2ea9f57f11d890246df5f6c5110 Log: Merge pull request #112 from jglick/transient-lastFetchInstance JENKINS-24938 Various fields should be transient Compare: https://github.com/jenkinsci/ec2-plugin/compare/7ebb4886cc05...d823b5e5da73

          Korey Earl added a comment -

          @Francis Upton
          Can you kindly explain why changing the fields to be transient will solve the issue?

          Cheers,
          Amber & Korey Pairing (Our old account got locked out).

          Korey Earl added a comment - @Francis Upton Can you kindly explain why changing the fields to be transient will solve the issue? Cheers, Amber & Korey Pairing (Our old account got locked out).

          Daniel Beck added a comment -

          Transient fields don't get serialized to XML on disk by Xstream.

          Daniel Beck added a comment - Transient fields don't get serialized to XML on disk by Xstream.

          Joseph Hughes added a comment -

          I have tested the fix and it works. Basically Xstream was not deserializing some values correctly and those values were not meant to be persisted in the first place.

          I have updated my masters with the patch and the fix works.

          Joseph Hughes added a comment - I have tested the fix and it works. Basically Xstream was not deserializing some values correctly and those values were not meant to be persisted in the first place. I have updated my masters with the patch and the fix works.

          Korey Earl added a comment -

          We have installed the EC2 plugin with these changes and we are not seeing this problem anymore!

          Thank you!

          Korey Earl added a comment - We have installed the EC2 plugin with these changes and we are not seeing this problem anymore! Thank you!

          Korey Earl added a comment -

          When will the next release of the Amazon EC2 plugin be available?

          Korey Earl added a comment - When will the next release of the Amazon EC2 plugin be available?

            francisu Francis Upton
            jjhughes57 Joseph Hughes
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: