Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18124

Proposal : Modularizing scm-sync-config scm implementations

      Scm-sync-configuration is based on Maven SCM API & Jenkins SCM API to provide a generic way to synchronize config files with multiple possible scm.

      Today, every possible SCM implementation (namely svn & git) are bundled inside the scm-sync-configuration-plugin artefact, thus pulling every scm implementations even if you only rely on 1 of them.
      FYI, following dependencies will be pulled per scm implementation :

      • maven-scm-provider's scm implementation (at the moment, both maven-scm-provider-svnjava and maven-scm-provider-gitexe)
      • Jenkins' plugin providing authentication support for your scm (at the moment, only org.jenkins-ci.plugins:subversion but we should consider org.jenkins-ci.plugins:git to have homogeneous git authentication support accross the plugin and jenkins stored credentials, see JENKINS-14506)
        This point is an important point because as soon as a plugin is depending on another plugin, jenkins will install both plugins when the root plugin is installed. This means once mercurial/perforce/whatever will be supported, when you will install the plugin on your jenkins instance, it will pull & install lots of possible unwanted plugins on your instance.

      Another burden is the way scm implementations are maintained.
      I (Frédéric Camblor) will not be able to ensure every scm implementation will just "work" : it is already a burden for me to ensure, for only 1 given scm implementation, that every possible scm protocols (namely svn / svn+ssh / http / https) are supported (See JENKINS-8871).
      This is something hard to test (and I don't even speak about automatic testing) because it needs to setup a subversion server handling every supported protocols.

      These are the reasons why I would like to modularize the plugin.
      Technically, this should consist in :

      • Modularize needed generic/helpful classes for scm manipulation in a scm-sync-configuration-api artefact
      • Provide a maven parent pom for scm-sync-configuration's scm implementors, in order to help scm maintainer to only focus on their implementation, and not on the build / dependencies for the plugin
      • Externalize scm implementations into dedicated Jenkins plugins, relying on scm-sync-configuration-api and extending scm-sync-configuration-scm-parent pom
      • Provide an @ExtensionPoint in the core, used by scm plugin implementors to plug their implementation

      In terms of usability, this will imply for the user :

      • To download not only the scm-sync-configuration-plugin but both scm-sync-configuration-plugin & scm-sync-configuration-plugin-<your scm impl(s)> (for instance : scm-sync-configuration-plugin-subversion)
      • The basic "none" scm implementation will be provided, and a message will be displayed to the user asking him to download at least one implementation if he didn't
      • The plugin will not be backward compatible because we will go from (none/subversion/git) available implementations to (none) implementation only. I don't know, at the moment, how I could handle this backward incompatibility (if you have ideas, don't hesitate to submit them in comments).
        Maybe could I rename the plugin and deprecate the existing one ? I dunno.
      • Lifecycle between core and scm implementations won't be the same : a bugfix/improvement in an scm implementation won't need a release of the core
        Drawback for this is we will have to track api incompatibilities in the core, and align scm implementations when needed. Goal is to avoid errors when core is upgraded and implementations are not (eventually, we could enforce this detection in scm-sync-config-api)

      In terms of maintenability, this will imply :

      • Migration from an organization where I (Frederic Camblor) am a SPOF, to an organization where this is less the case : every scm implementor will be considered as a plugin maintainer with dedicated component in jira, project in github, and responsibilities.
      • The core will remain open to every contributor, particularly scm implementors (if one of them need an enhancement in the core, we'll have to talk about it but it will be welcomed)

      Don't hesitate to provide your POV in the comments, I'm opened to them

          [JENKINS-18124] Proposal : Modularizing scm-sync-config scm implementations

          scm-sync-config to expose an extension point for all repo operation would indeed be a more extensible design.
          I'm also not convinced about using maven-scm implementation. For sample, the git one don't let you create a commit passing author/committer identity. An alternate impl to use JGit could then be created. Afaik scm-sync only uses a limited set of scm methods, so such an implementation wouldn't be such a pain.

          Nicolas De Loof added a comment - scm-sync-config to expose an extension point for all repo operation would indeed be a more extensible design. I'm also not convinced about using maven-scm implementation. For sample, the git one don't let you create a commit passing author/committer identity. An alternate impl to use JGit could then be created. Afaik scm-sync only uses a limited set of scm methods, so such an implementation wouldn't be such a pain.

          Henri Gomez added a comment -

          I don't have problem with many plugins, aka scm-sync-configuration-plugin & scm-sync-configuration-plugin-xxxx-impl.

          About SPOF, it will be nice to have SCM SYNC in Jenkins organisation, next to the core, since you experience issues when core moved.
          Currently it's a nightmare to have a working matrix in Jenkins, aka Core and Plugins aligned, and some sensible parts, like SCM SYNC should be part of core.

          My 0.01€

          Henri Gomez added a comment - I don't have problem with many plugins, aka scm-sync-configuration-plugin & scm-sync-configuration-plugin-xxxx-impl. About SPOF, it will be nice to have SCM SYNC in Jenkins organisation, next to the core, since you experience issues when core moved. Currently it's a nightmare to have a working matrix in Jenkins, aka Core and Plugins aligned, and some sensible parts, like SCM SYNC should be part of core. My 0.01€

          I don't get your argumentation on maven parent pom for scm-sync implementors. If scm-sync expose an extension point, just need to create a very classic jenkins plugin and implement @Extension, what else ?

          for backward compatibility, scm-sync could have the scm-sync-svn-provider and scm-sync-git provider bundled as dependency.
          Initial implementation for those two plugins just need to extract code from scm-sync "as is" still using maven-scm, then let maintainer explore alternative implementation, either as improvements or in other dedicated plugins.

          Nicolas De Loof added a comment - I don't get your argumentation on maven parent pom for scm-sync implementors. If scm-sync expose an extension point, just need to create a very classic jenkins plugin and implement @Extension, what else ? for backward compatibility, scm-sync could have the scm-sync-svn-provider and scm-sync-git provider bundled as dependency. Initial implementation for those two plugins just need to extract code from scm-sync "as is" still using maven-scm, then let maintainer explore alternative implementation, either as improvements or in other dedicated plugins.

          @Nicolas I agree with you on the fact that only a little set of maven-scm is used.
          And relying on it shows its limits as soon as a feature is not "generically" available for every scms (see JENKINS-14841).
          So your proposal makes sense.
          What I'm not confident about this proposal is the way authentication will be handled on SCM. Is Jenkins SCM API providing generic way to authenticate against stored credentials in jenkins ?
          If it would be the case, it would be great !

          @Henri Dependency against core is the burden of every plugin maintainers.
          But I don't know there is a better way to handle this.
          This was the intention of the plugin-compat-tester .. but I didn't have enough time recently to improve it

          Frédéric Camblor added a comment - @Nicolas I agree with you on the fact that only a little set of maven-scm is used. And relying on it shows its limits as soon as a feature is not "generically" available for every scms (see JENKINS-14841 ). So your proposal makes sense. What I'm not confident about this proposal is the way authentication will be handled on SCM. Is Jenkins SCM API providing generic way to authenticate against stored credentials in jenkins ? If it would be the case, it would be great ! @Henri Dependency against core is the burden of every plugin maintainers. But I don't know there is a better way to handle this. This was the intention of the plugin-compat-tester .. but I didn't have enough time recently to improve it

          @Nicolas The main goal of maven parent is to provide some tricky things like the plexus components merging.
          ... Only needed if we stay with maven-scm-api.
          If there isn't any tricky build part, no parent will be needed.

          Frédéric Camblor added a comment - @Nicolas The main goal of maven parent is to provide some tricky things like the plexus components merging . ... Only needed if we stay with maven-scm-api. If there isn't any tricky build part, no parent will be needed.

          @Frédéric well we should be moving authentication credential storage to the credentials-plugin.

          I was discussing with @Nicolas what needs to be done to get, e.g. subversion-plugin using the credentials-plugin. There is a number of things that I would like to see in terms of additions to the credentials-plugin API that should take place before wiring it up to more plugins.

          The point of getting credentials 1.0 out was to let the API soak some more. After over a year of soaking I am mostly happy with the core of the API.

          Where I feel there are gaps are with respect to grouping credentials and providing an easy way for consumers to retrieve credentials.

          I am leaning towards adding Hamcrest matchers as a dependency, with methods for using those matchers to select a sub-set of credentials.

          Additionally I am thinking of adding some grouping extension point to allow grouping related credentials together.

          Finally, there needs to be a credential mapper extension point, thus you could have

          A grouping that reflects, e.g. an Active Directory / LDAP domain.
          Within the group a User/Password pair
          A credential mapper from User/Password to SSH User/Password
          A credential mapper from User/Password to HTTP(s) User/Password
          A credential mapper from User/Password to SVNSERVE User/Password

          Then the Credential consumer can ask for credentials within a Jenkins scope (i.e. specific job) for a specific user, against a specific host (which the grouping then matches) and for a specific protocol

          The user, on the other hand, sees just the one credential in the one group, and changing that password updates all of that credential's use by Jenkins in one go.

          The bits I need to refine are:

          • Credential mapper - and when a credential mapper should not be employed, e.g. if the SSH password is different in that scope for the same user name and there already is a SSH credential defined in the group
          • Credential group extension point - how to handle matching which group is appropriate... maybe not needed, if the group is just a "name" but it would be nice to have a better solution.
          • Whether to expose hamcrest matchers directly (pushing the API into Jenkins Core effectively as credentials is now in core) or whether to copy the "nice" bits of Hamcrest and do an NIH... leaning towards exposing as Hamcrest seems to be good at maintaining backwards compatibility (unlike some others... I'm looking at you BouncyCastle)

          Then the final bits should actually be mostly trivial as there are very few authentication schemes that would need to be handled. (user/pass, user/key, client certificate. I think that is all)

          Stephen Connolly added a comment - @Frédéric well we should be moving authentication credential storage to the credentials-plugin. I was discussing with @Nicolas what needs to be done to get, e.g. subversion-plugin using the credentials-plugin. There is a number of things that I would like to see in terms of additions to the credentials-plugin API that should take place before wiring it up to more plugins. The point of getting credentials 1.0 out was to let the API soak some more. After over a year of soaking I am mostly happy with the core of the API. Where I feel there are gaps are with respect to grouping credentials and providing an easy way for consumers to retrieve credentials. I am leaning towards adding Hamcrest matchers as a dependency, with methods for using those matchers to select a sub-set of credentials. Additionally I am thinking of adding some grouping extension point to allow grouping related credentials together. Finally, there needs to be a credential mapper extension point, thus you could have A grouping that reflects, e.g. an Active Directory / LDAP domain. Within the group a User/Password pair A credential mapper from User/Password to SSH User/Password A credential mapper from User/Password to HTTP(s) User/Password A credential mapper from User/Password to SVNSERVE User/Password Then the Credential consumer can ask for credentials within a Jenkins scope (i.e. specific job) for a specific user, against a specific host (which the grouping then matches) and for a specific protocol The user, on the other hand, sees just the one credential in the one group, and changing that password updates all of that credential's use by Jenkins in one go. The bits I need to refine are: Credential mapper - and when a credential mapper should not be employed, e.g. if the SSH password is different in that scope for the same user name and there already is a SSH credential defined in the group Credential group extension point - how to handle matching which group is appropriate... maybe not needed, if the group is just a "name" but it would be nice to have a better solution. Whether to expose hamcrest matchers directly (pushing the API into Jenkins Core effectively as credentials is now in core) or whether to copy the "nice" bits of Hamcrest and do an NIH... leaning towards exposing as Hamcrest seems to be good at maintaining backwards compatibility (unlike some others... I'm looking at you BouncyCastle) Then the final bits should actually be mostly trivial as there are very few authentication schemes that would need to be handled. (user/pass, user/key, client certificate. I think that is all)

          @Nicolas & @Stephen I created JENKINS-18129 which represent a different concern than current issue to my POV
          It will be better to discuss there instead of noising current comments

          Frédéric Camblor added a comment - @Nicolas & @Stephen I created JENKINS-18129 which represent a different concern than current issue to my POV It will be better to discuss there instead of noising current comments

          I think split into multiple plugins is a good solution.

          I think this refactor must address credentials topic. May be the maven-scm plugin can be remove for a simplest one, I had bad memories about patches to add git support.

          I'm ok to move the git support to the new architecture.

          Bertrand Paquet added a comment - I think split into multiple plugins is a good solution. I think this refactor must address credentials topic. May be the maven-scm plugin can be remove for a simplest one, I had bad memories about patches to add git support. I'm ok to move the git support to the new architecture.

          Mark Waite added a comment -

          The scm sync configuration plugin is deprecated. No further work is expected on the plugin.

          Users of this plugin are encouraged to migrate their jobs to Pipeline Plugin, so that their jobs will be specified in pipeline code that can be checked into an SCM, all without using the SCM Sync configuration plugin.

          Users of this plugin are also encouraged to use the Configuration as Code Plugin to specify the configuration of their Jenkins server in a file which can be checked into an SCM, all without using the SCM Sync configuration plugin.

          Mark Waite added a comment - The scm sync configuration plugin is deprecated. No further work is expected on the plugin. Users of this plugin are encouraged to migrate their jobs to Pipeline Plugin, so that their jobs will be specified in pipeline code that can be checked into an SCM, all without using the SCM Sync configuration plugin. Users of this plugin are also encouraged to use the Configuration as Code Plugin to specify the configuration of their Jenkins server in a file which can be checked into an SCM, all without using the SCM Sync configuration plugin.

            Unassigned Unassigned
            fcamblor Frédéric Camblor
            Votes:
            2 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: