Thanks. I may be missing something then. It looks like Jenkins loses the formatting from CLI/REST data because the XML is parsed and written do disk based on the in-memory model rather than what was received through either API. It seems like Jenkins chooses different ways to write the xml files (or different in-memory representations persist differently) based on what's requesting the save (UI button click vs API update received) rather than the content of the save, rather than it being a case of the data received being different each time.

We are having the same problem with our Jenkins instance. Locally we work around it by normalizing the XML we receive from the master and showing the diff of the normalized data (https://github.com/ros-infrastructure/ros_buildfarm/blob/16bc85c4e312fb1975d061c6e749fb4af9e3c4ff/ros_buildfarm/jenkins.py#L229-L249). But the Job Configuration History plugin shows many unnecessary changes which makes it more difficult to see the actual differences. Please see the attached picture which shows a huge diff with actually no changes at all.

If this is considered "not important enough" to be addressed by the devs it would be great if someone could shed some light into why the output is different and provide some pointers to the involved code blocks. Maybe we can come up with a patch for it.

It is intentional that the provided XML is parsed and the model resaved, so do not expect your formatting to survive intact. I do not know offhand why there would be any difference in formatting relative to doConfigSubmit-saved configurations, though. Possibly the Document object improperly retains some memory of the origin’s formatting when using CLI/REST uploads.

I don't mind if the server decides to "modify" the style of the xml before it is written to the config file. But I do mind that it deterministically uses two different approaches when editing the config using the web ui vs. using the remote api. That makes the job config history much less useful and clutters the diff.

This is why XML diff tools exist. I'd consider this a bug in Job Config History.

Imo the task of the Job Config History is to store a history of job configurations. The fact that Jenkins doesn't store the same configuration in a deterministic way would require to implement a more complex diff logic. Since Jenkins is already changing the formatting of the XML when it is received through e.g. the remote API I don't see a reason why it shouldn't store the content in a deterministic way. A user might even want to compare the current config file in the filesystem with a backup from some time ago. Why should that be more effort than absolutely necessary?

Even if you disagree I am just asking for some pointers where these two ways of updating the config files are happening in the code. I am more than happy to work on a patch myself if nobody else considers that this is necessary / useful. So far I have found these locations which seem to be relevant. I just don't see yet where the two paths diverge:

• https://github.com/jenkinsci/jenkins/blob/25ef87d2e243fbe7e6f582b08f00574b032a5fb9/core/src/main/java/hudson/model/AbstractItem.java#L609 is where an update coming from the web ui is being handled

• which gets passed down here: https://github.com/jenkinsci/jenkins/blob/25ef87d2e243fbe7e6f582b08f00574b032a5fb9/core/src/main/java/hudson/model/AbstractItem.java#L666
• and finally the content is being written to the config file in https://github.com/jenkinsci/jenkins/blob/25ef87d2e243fbe7e6f582b08f00574b032a5fb9/core/src/main/java/jenkins/util/xml/XMLUtils.java#L209

What I am not sure about is where the remote API config is coming in and which code it is taking until being written to the config file. Any pointers would help. Thanks.

I'm not sure what the remote API is used for, but we see these inexplicable differences in XML format when modifying the config from the web UI as compared to generating the config from a seed job (using the groovy DSL).  There are things like "<foo/>" vs. "<foo></foo>", and spacing differences, that render the diffs unusable for no apparent reason.

jglick danielbeck Could you please help us out with some pointers here? Even if you don't consider this to be a problem it would help others to look into this and come up with a patch.

I would conisder it to be a bug, albeit a very minor one. I do not have any tips offhand. You would need to fire up a debugger.

That's a core bug because the API should return a normalised and predictable XML format, regardless what tool was used to change the config.

I find unreasonable and close to impossible to use an external XML comparison tool in order to see what really changed in the config.

I see this as solvable in two ways: either make Jenkins reformat the config on save, or change the read API to perform the reformat on read. I am not sure which one is better, but any of them should address the issue of random fake changes in configuration, which can be a real issue if jobs are configured using configuration management tools (fake change history, so long that is impossible to stop real changes).