• Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • mercurial-plugin
    • None
    • Platform: All, OS: All

      In addition to supporting the Forest extension, it would be good to support the
      new subrepo system in Hg 1.3+.

          [JENKINS-4838] Subrepo support

          Jesse Glick added a comment -

          .

          Jesse Glick added a comment - .

          Jesse Glick added a comment -

          I will probably not work on this any time soon since it seems NetBeans cannot
          use subrepo: the exist of the .hgsub file causes all operations to work on it,
          whereas our main/contrib subrepo is optional.

          Jesse Glick added a comment - I will probably not work on this any time soon since it seems NetBeans cannot use subrepo: the exist of the .hgsub file causes all operations to work on it, whereas our main/contrib subrepo is optional.

          Netbeans probably supports this, albeit in a roundabout way. The real issue here is that if the CI server doesn't, then that limits the choice for the entire team. Since Forest is no longer actively maintained and subrepos are here to stay, I would say that this should become a strong candidate for development.

          Ioannis Mavroukakis added a comment - Netbeans probably supports this, albeit in a roundabout way. The real issue here is that if the CI server doesn't, then that limits the choice for the entire team. Since Forest is no longer actively maintained and subrepos are here to stay, I would say that this should become a strong candidate for development.

          davidmc24 added a comment -

          On the technical side, does anyone have thoughts on how to accomplish this?

          The polling logic should already be somewhat subrepo-compatible, in that a change to the .hgsubstate in the top level repo would trigger a build. This doesn't currently play nicely with the ability to specify "modules" that should be included.

          The main area that I would like to see improvement in is support for the changelog including the paths that were changed in the nested repositories. Additionally, there's likely various issues with repository browsers.

          So, how do we get information about what files were changed in subrepos in each changeset? Unfortunately, the "hg log" command doesn't currently have any knowledge of subrepos (that I can see). The best approach I've thought up is to use the output of "hg log" to get a list of the changesets that are relevant, then use "hg diff -S -c CHANGESET" to get subrepo-aware information from each changeset, and parse the diff output to get the list of files.

          davidmc24 added a comment - On the technical side, does anyone have thoughts on how to accomplish this? The polling logic should already be somewhat subrepo-compatible, in that a change to the .hgsubstate in the top level repo would trigger a build. This doesn't currently play nicely with the ability to specify "modules" that should be included. The main area that I would like to see improvement in is support for the changelog including the paths that were changed in the nested repositories. Additionally, there's likely various issues with repository browsers. So, how do we get information about what files were changed in subrepos in each changeset? Unfortunately, the "hg log" command doesn't currently have any knowledge of subrepos (that I can see). The best approach I've thought up is to use the output of "hg log" to get a list of the changesets that are relevant, then use "hg diff -S -c CHANGESET" to get subrepo-aware information from each changeset, and parse the diff output to get the list of files.

          Jesse Glick added a comment -

          You might be able to hg cat --rev $rev .hgsubstate using both the old main repo version (as saved in SCM state) and the new tip; parse the .hgsubstate contents; and use that to compute the start and end points for a computation of csets in subrepos. This would let you produce a precise changelog (without parsing diff output), and would also solve the polling support for the modules field (if anyone really cares). Seems like a fair amount of work, though.

          Jesse Glick added a comment - You might be able to hg cat --rev $rev .hgsubstate using both the old main repo version (as saved in SCM state) and the new tip; parse the .hgsubstate contents; and use that to compute the start and end points for a computation of csets in subrepos. This would let you produce a precise changelog (without parsing diff output), and would also solve the polling support for the modules field (if anyone really cares). Seems like a fair amount of work, though.

          GlennZ added a comment - - edited

          Any activity on this? We use hg subrepos, and I've got a partial work-around that allows us to include the subrepo changes on the jenkins "Changes" screens, but it doesn't work for the email notifications:

          Our build script runs a similar "hg incoming" command to the one that the jenkins mercurial plug-in already runs for the top level hg repo, and it modifies/augments the jenkins changelog.xml file in the builds directory with the additional changes for each subrepo.

          Here's a partial snippet from that script:

          TEMPLATE="<changeset node='{node}' author='{author|xmlescape}' rev='{rev}' date='{date}'><msg>{desc|xmlescape}</msg><added>{file_adds|stringify|xmlescape}</added><deleted>{file_dels|stringify|xmlescape}</deleted><files>{files|stringify|xmlescape}</files><parents>{parents}</parents></changeset>\n" 
          
          {
              grep -v '</changesets>' $CHANGELOG
          
              # Run the hg incoming report to pick up the changes for each project.
              # (The main/parent project changelog is already captured by jenkins.)
              for PROJECT in $@; do
                 hg -R $PROJECT incoming --quiet --template "$TEMPLATE"
              done
          
              echo "</changesets>"
          } > $CHANGELOG.new  && mv $CHANGELOG.new $CHANGELOG
          

          Note that the above "hg in" will include all changesets up to tip since the last build. To more generally support the .hgsubstate file, the "hg in" should include "-r $rev", where $rev is set from the entry in .hgsubstate.

          The build script then proceeds to update the workspace with the incoming subrepo changesets and complete the build. The final kludge to get jenkins to see the modified changelog.xml file is to trigger a jenkins reload.

          It would be better to get this functionality into the mercurial plugin and avoid needing to do that kludge, which would likely also allow the added changes to be included with the build notification emails. I haven't yet delved into jenkins plug-in module development but might look into it for this.

          GlennZ added a comment - - edited Any activity on this? We use hg subrepos, and I've got a partial work-around that allows us to include the subrepo changes on the jenkins "Changes" screens, but it doesn't work for the email notifications: Our build script runs a similar "hg incoming" command to the one that the jenkins mercurial plug-in already runs for the top level hg repo, and it modifies/augments the jenkins changelog.xml file in the builds directory with the additional changes for each subrepo. Here's a partial snippet from that script: TEMPLATE="<changeset node='{node}' author='{author|xmlescape}' rev='{rev}' date='{date}'><msg>{desc|xmlescape}</msg><added>{file_adds|stringify|xmlescape}</added><deleted>{file_dels|stringify|xmlescape}</deleted><files>{files|stringify|xmlescape}</files><parents>{parents}</parents></changeset>\n" { grep -v '</changesets>' $CHANGELOG # Run the hg incoming report to pick up the changes for each project. # (The main/parent project changelog is already captured by jenkins.) for PROJECT in $@; do hg -R $PROJECT incoming --quiet --template "$TEMPLATE" done echo "</changesets>" } > $CHANGELOG.new && mv $CHANGELOG.new $CHANGELOG Note that the above "hg in" will include all changesets up to tip since the last build. To more generally support the .hgsubstate file, the "hg in" should include "-r $rev", where $rev is set from the entry in .hgsubstate. The build script then proceeds to update the workspace with the incoming subrepo changesets and complete the build. The final kludge to get jenkins to see the modified changelog.xml file is to trigger a jenkins reload. It would be better to get this functionality into the mercurial plugin and avoid needing to do that kludge, which would likely also allow the added changes to be included with the build notification emails. I haven't yet delved into jenkins plug-in module development but might look into it for this.

          Jesse Glick added a comment - - edited

          I do not know of anyone actively working on this. Since my organization does not use subrepos I am not inclined to work on it myself.

          I would expect it to be a relatively difficult change, i.e. you would need to understand all aspects of the current plugin; handle (or explicitly forbid) interaction with various existing features such as caching & sharing, polling exclusions, and browsers, which are all written to assume that the portion of the workspace under consideration is the checkout of one node of one repository; and write a full JUnit suite detailing the expected behavior, ideally before starting to make changes to main sources.

          Note that for jobs formerly using the Forest extension the recommendation is to use the Multi-SCM plugin instead - basically configuring a separate instance of the Mercurial plugin for each repo in the "forest". This seems to work fine, if perhaps a little more verbose to configure. But subrepos work differently due to .hgsubstate, so I doubt you could use the Multi-SCM plugin for this purpose.

          Also beware of subrepos using foreign SCMs like Subversion; it is not clear whether or how the Mercurial plugin (in cooperation with other SCM plugins) could behave correctly with such a setup.

          Jesse Glick added a comment - - edited I do not know of anyone actively working on this. Since my organization does not use subrepos I am not inclined to work on it myself. I would expect it to be a relatively difficult change, i.e. you would need to understand all aspects of the current plugin; handle (or explicitly forbid) interaction with various existing features such as caching & sharing, polling exclusions, and browsers, which are all written to assume that the portion of the workspace under consideration is the checkout of one node of one repository; and write a full JUnit suite detailing the expected behavior, ideally before starting to make changes to main sources. Note that for jobs formerly using the Forest extension the recommendation is to use the Multi-SCM plugin instead - basically configuring a separate instance of the Mercurial plugin for each repo in the "forest". This seems to work fine, if perhaps a little more verbose to configure. But subrepos work differently due to .hgsubstate , so I doubt you could use the Multi-SCM plugin for this purpose. Also beware of subrepos using foreign SCMs like Subversion; it is not clear whether or how the Mercurial plugin (in cooperation with other SCM plugins) could behave correctly with such a setup.

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            10 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: