Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18187

scm-sync-configuration: Posibility to exclude files/directories from syncing

    XMLWordPrintable

Details

    Description

      To reduce the scm "noise" it may be good to be able to exclude some files from syncing.

      e. g.
      In our environment, the file hudson.plugins.disk_usage.DiskUsageProjectActionFactory.xml was synced every day, the changes reflect the current disc usage only this might be not necessary.

      Attachments

        Issue Links

          Activity

            wo_hauser Wolfgang Hauser created issue -

            Not really in favor to add excludes because, to my POV, it will complexify things

            I would rather be in favor to be more restrictive in the BasicPluginsConfigScmSyncStrategy default implementation (see here)

            WDYT ?

            fcamblor Frédéric Camblor added a comment - Not really in favor to add excludes because, to my POV, it will complexify things I would rather be in favor to be more restrictive in the BasicPluginsConfigScmSyncStrategy default implementation (see here ) WDYT ?
            wo_hauser Wolfgang Hauser made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-18189 [ JENKINS-18189 ]

            In our case, it was more an issue of the disk usage plugin which doesn't separate data from configuration.

            But as there was (as I know) no naming convention/location for files storing data, every plugin may provide some .xml files not necessarily have to be synced in scm.

            My personal opinion was to give the users the freedom to configure whatever was possible. This make the things flexible and reduces maintenance.

            wo_hauser Wolfgang Hauser added a comment - In our case, it was more an issue of the disk usage plugin which doesn't separate data from configuration. But as there was (as I know) no naming convention/location for files storing data, every plugin may provide some .xml files not necessarily have to be synced in scm. My personal opinion was to give the users the freedom to configure whatever was possible. This make the things flexible and reduces maintenance.
            fcamblor Frédéric Camblor added a comment - - edited

            It would be the case if the BasicPluginsConfigScmSyncStrategy would only define a white list (more restricted as today) of sync'ed xml files.

            Then, users could play with manual includes feature to add their own files they want to sync.

            fcamblor Frédéric Camblor added a comment - - edited It would be the case if the BasicPluginsConfigScmSyncStrategy would only define a white list (more restricted as today) of sync'ed xml files. Then, users could play with manual includes feature to add their own files they want to sync.

            You have to build up a list of items for syncing anyway, so an exclude list was possible. In case of manual includes especially for directories a exclude list may be handy too.

            The scm-sync-configration plugin was on the way to be a save and recovery tool for Jenkins configuration, so it should be possible to fine tune the content synced to scm.

            A fixed white list probably requires maintenance for every change/addition of plugins and core.

            An exclude list gives the responsibility to the administrator and no white list have to be maintained anymore.
            This will make the plugin more flexible.

            Exclude lists give an administrator the freedom to save only what he need for his purposes, e. g. save only several job configurations and plugin settings.
            This may be needed if there are temporary or not common jobs and plugins are used on a Jenkins server.

            wo_hauser Wolfgang Hauser added a comment - You have to build up a list of items for syncing anyway, so an exclude list was possible. In case of manual includes especially for directories a exclude list may be handy too. The scm-sync-configration plugin was on the way to be a save and recovery tool for Jenkins configuration, so it should be possible to fine tune the content synced to scm. A fixed white list probably requires maintenance for every change/addition of plugins and core. An exclude list gives the responsibility to the administrator and no white list have to be maintained anymore. This will make the plugin more flexible. Exclude lists give an administrator the freedom to save only what he need for his purposes, e. g. save only several job configurations and plugin settings. This may be needed if there are temporary or not common jobs and plugins are used on a Jenkins server.

            An other way to realize this may be a maintainable white list (as xml or plain text) predefined by the plugin. So an administrator could edit this list if necessary.

            But this will break the full control over the WEB interface.

            wo_hauser Wolfgang Hauser added a comment - An other way to realize this may be a maintainable white list (as xml or plain text) predefined by the plugin. So an administrator could edit this list if necessary. But this will break the full control over the WEB interface.

            This would, for sure, be a gain in flexibility. I completely agree with you.

            ... but what I don't want is to give administrators the power to shoot themselves in the foot.

            To my POV, syncing an entire directory is not recommended.
            Since Jenkins is storing everyting in XML files, you'll often have lots of noise brought by lots of unneeded xml files in file hierarchies.
            It was the idea behind manual includes : target only important specific files that would be backed up.

            I know this is my personal POV which is not obviously shared by everyone
            I'll consider adding the exclude list someday, but it's not on the top of my todolist right now.

            fcamblor Frédéric Camblor added a comment - This would, for sure, be a gain in flexibility. I completely agree with you. ... but what I don't want is to give administrators the power to shoot themselves in the foot. To my POV, syncing an entire directory is not recommended. Since Jenkins is storing everyting in XML files, you'll often have lots of noise brought by lots of unneeded xml files in file hierarchies. It was the idea behind manual includes : target only important specific files that would be backed up. I know this is my personal POV which is not obviously shared by everyone I'll consider adding the exclude list someday, but it's not on the top of my todolist right now.

            Its just a proposal for an improvement.

            BTW syncing a directory is possible, e. g. I have the need of including one to save a necessary tool needed by some jobs.
            There also may be directories containing supporting scripts needed for processing.

            We have to be able to restore the complete environment for building a released SW. So we should be able to restore the used Jenkins environment/version too.

            So we tag the synced Jenkins environment to be able to restore it.

            wo_hauser Wolfgang Hauser added a comment - Its just a proposal for an improvement. BTW syncing a directory is possible, e. g. I have the need of including one to save a necessary tool needed by some jobs. There also may be directories containing supporting scripts needed for processing. We have to be able to restore the complete environment for building a released SW. So we should be able to restore the used Jenkins environment/version too. So we tag the synced Jenkins environment to be able to restore it.
            sqbanietz Lukasz Hanik added a comment -

            @Frederic:
            I agree with Wolfgang that exclusion lists is really important enhancement. I had to disable scm-sync-configuration because I use ci-game which modifies user's xml after each commit (changing score for user who has committed change). In this particular situation, users' config files are modified too often making too many scm-sync-configuration commits (each user's commit means additional scm-sync-configuration commit). I cannot afford such amount of commits cause it makes big mess in our project. Exclusions will be great advantage.
            BTW. Do you think it is somehow possible to exclude whole users directory by svn ignore set on this directory?

            sqbanietz Lukasz Hanik added a comment - @Frederic: I agree with Wolfgang that exclusion lists is really important enhancement. I had to disable scm-sync-configuration because I use ci-game which modifies user's xml after each commit (changing score for user who has committed change). In this particular situation, users' config files are modified too often making too many scm-sync-configuration commits (each user's commit means additional scm-sync-configuration commit). I cannot afford such amount of commits cause it makes big mess in our project. Exclusions will be great advantage. BTW. Do you think it is somehow possible to exclude whole users directory by svn ignore set on this directory?

            @Lukasz I see what you're describing as another improvement possibility : allowing to define SYSTEM-commited files only periodically.

            I'd imagine some additionnal config file allowing to say "only sync SYSTEM-modified files every XXX hours"

            The more difficult would be to identify how to translate this on multiple files commited.

            WDYT ?

            fcamblor Frédéric Camblor added a comment - @Lukasz I see what you're describing as another improvement possibility : allowing to define SYSTEM-commited files only periodically. I'd imagine some additionnal config file allowing to say "only sync SYSTEM-modified files every XXX hours" The more difficult would be to identify how to translate this on multiple files commited. WDYT ?
            sqbanietz Lukasz Hanik added a comment -

            @Frederic:
            I like your proactive approach. It is indeed interesting improvement. However there is already existing solution to achieve periodical commits of any files (http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion) while there is no way to exclude default paths from scm-sync-configuration. This is the reason why exclusions (or possibility to define all includes from scratch without any default path) is more important from my point of view.

            sqbanietz Lukasz Hanik added a comment - @Frederic: I like your proactive approach. It is indeed interesting improvement. However there is already existing solution to achieve periodical commits of any files ( http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion ) while there is no way to exclude default paths from scm-sync-configuration. This is the reason why exclusions (or possibility to define all includes from scratch without any default path) is more important from my point of view.
            fcamblor Frédéric Camblor added a comment - - edited

            In fact, in correlation with this feature, I was thinking about making default bundled synchronization strategies (namely JENKINS_HOME/hudson*.xml, JENKINS_HOME/config.xml, JENKINS_HOME/jobs/*/config.xml, JENKINS_HOME/users/*/config.xml) activables or not (with simple checkbox)

            By default, every strategies would be activated, but everybody could deactivate specific strategies due to specific cases.

            I would greatly prefer this approach than the exclude one.

            WDYT ?

            fcamblor Frédéric Camblor added a comment - - edited In fact, in correlation with this feature, I was thinking about making default bundled synchronization strategies (namely JENKINS_HOME/hudson*.xml, JENKINS_HOME/config.xml, JENKINS_HOME/jobs/*/config.xml, JENKINS_HOME/users/*/config.xml) activables or not (with simple checkbox) By default, every strategies would be activated, but everybody could deactivate specific strategies due to specific cases. I would greatly prefer this approach than the exclude one. WDYT ?
            sqbanietz Lukasz Hanik added a comment -

            @Frederic:
            It sounds fine for me.

            sqbanietz Lukasz Hanik added a comment - @Frederic: It sounds fine for me.
            fcamblor Frédéric Camblor made changes -
            Link This issue is related to JENKINS-19659 [ JENKINS-19659 ]
            fcamblor Frédéric Camblor made changes -
            Link This issue is related to JENKINS-19660 [ JENKINS-19660 ]

            Just created JENKINS-19659 & JENKINS-19660

            @Wolfgang with this in mind, can I close current issue ?

            fcamblor Frédéric Camblor added a comment - Just created JENKINS-19659 & JENKINS-19660 @Wolfgang with this in mind, can I close current issue ?
            cforce cforce added a comment -

            It would be useful to be able to exclude xml elements in included config files from sync, like its done for the job-config-history plugin also.

            cforce cforce added a comment - It would be useful to be able to exclude xml elements in included config files from sync, like its done for the job-config-history plugin also.

            cforce, please take time to read the reasons why I don't want to exclude files, instead of cross commenting without providing any additionnal arguments.

            fcamblor Frédéric Camblor added a comment - cforce, please take time to read the reasons why I don't want to exclude files, instead of cross commenting without providing any additionnal arguments.
            cforce cforce added a comment -

            How else shall i make it possible to sync user's configs importnat settings changes, but keep noise down like issues for every build being failed or fixed. The jenkins game updates the user's pürofile with + or -+1 points for in user profile.xml to often. I only thing to rid of this by being able to explude changes in this setions triggering updates. I don't wanna exlude the content if syncec, but the cause's triggering this syncs.

            cforce cforce added a comment - How else shall i make it possible to sync user's configs importnat settings changes, but keep noise down like issues for every build being failed or fixed. The jenkins game updates the user's pürofile with + or -+1 points for in user profile.xml to often. I only thing to rid of this by being able to explude changes in this setions triggering updates. I don't wanna exlude the content if syncec, but the cause's triggering this syncs.

            Ok my bad, didn't understood you were talking about specific file "chunks" avoiding a new commit.

            To be honest, what you ask seems really complicated to implement (not triggering a commit based on the chunk modification, in a generic way).

            But JENKINS-19660 could bring a part of the answer : this +/-1 are SYSTEM modifications, so we shouldn't take them into account right now.

            fcamblor Frédéric Camblor added a comment - Ok my bad, didn't understood you were talking about specific file "chunks" avoiding a new commit. To be honest, what you ask seems really complicated to implement (not triggering a commit based on the chunk modification, in a generic way). But JENKINS-19660 could bring a part of the answer : this +/-1 are SYSTEM modifications, so we shouldn't take them into account right now .
            fcamblor Frédéric Camblor made changes -
            Link This issue is related to JENKINS-22540 [ JENKINS-22540 ]
            fcamblor Frédéric Camblor made changes -
            Link This issue is duplicated by JENKINS-28391 [ JENKINS-28391 ]
            rschulz Roland Schulz added a comment -

            If I understand you correctly you think that having an exclusion list is not a good option because including a directory is a bad choice, because it leads to too much noise. I disagree. If you list individual files you miss configuration files of new plugins you install if they don't follow a standard pattern. And if you exclude all files which would lead to too much noise then you also address the noise issue. Of course if you install a new plugin which creates a new file which leads to too much noise you have to add that to the exclusion filter (the same was as you now have to include others to the inclusion filter). But I would prefer to have to manually update the filter in that case because if I forget, I have the less big problem (too much noise instead of missing potentially important configurations).

            rschulz Roland Schulz added a comment - If I understand you correctly you think that having an exclusion list is not a good option because including a directory is a bad choice, because it leads to too much noise. I disagree. If you list individual files you miss configuration files of new plugins you install if they don't follow a standard pattern. And if you exclude all files which would lead to too much noise then you also address the noise issue. Of course if you install a new plugin which creates a new file which leads to too much noise you have to add that to the exclusion filter (the same was as you now have to include others to the inclusion filter). But I would prefer to have to manually update the filter in that case because if I forget, I have the less big problem (too much noise instead of missing potentially important configurations).
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 149485 ] JNJira + In-Review [ 177383 ]
            fcamblor Frédéric Camblor made changes -
            Assignee Frédéric Camblor [ fcamblor ] Craig Rodrigues [ rodrigc ]
            rodrigc Craig Rodrigues made changes -
            Assignee Craig Rodrigues [ rodrigc ]

            People

              Unassigned Unassigned
              wo_hauser Wolfgang Hauser
              Votes:
              8 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: