Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38679

Constant builds when svn url contains parameter

      I see a wierd bug when using global libraries on pipeline.

      We have a groovy class with static defines, which lives in a legacy place in subversion and can't be moved.

      I've worked around that by configuring a global library:

          <org.jenkinsci.plugins.workflow.libs.LibraryConfiguration>
            <name>mylib</name>
            <retriever class="org.jenkinsci.plugins.workflow.libs.SCMRetriever">
              <scm class="hudson.scm.SubversionSCM" plugin="subversion@2.7.1">
                <locations>
                  <hudson.scm.SubversionSCM_-ModuleLocation>
                    <remote>https://svn/project/data${library.ladok3.version}/legacy/path/mylib</remote>
                    <credentialsId>xxx</credentialsId>
                    <local>src/mylib</local>
                    <depthOption>infinity</depthOption>
                    <ignoreExternalsOption>true</ignoreExternalsOption>
                  </hudson.scm.SubversionSCM_-ModuleLocation>
                </locations>
                <excludedRegions>/.*</excludedRegions>
                <includedRegions></includedRegions>
                <excludedUsers></excludedUsers>
                <excludedRevprop></excludedRevprop>
                <excludedCommitMessages></excludedCommitMessages>
                <workspaceUpdater class="hudson.scm.subversion.UpdateUpdater"/>
                <ignoreDirPropChanges>false</ignoreDirPropChanges>
                <filterChangelog>false</filterChangelog>
              </scm>
            </retriever>
            <defaultVersion>/trunk</defaultVersion>
            <implicit>false</implicit>
            <allowVersionOverride>true</allowVersionOverride>
          </org.jenkinsci.plugins.workflow.libs.LibraryConfiguration>
      

      Every scm poll call finishes like:
      Workspace doesn't contain https://svn/project/data${library.mylib.version}/legacy/path/mylib. Need a new build.

      Everything loads just file when i use:
      @Library("mylib@/trunk")
      import mylib

      but the polling bug makes it re-build every poll call.

      I started out using the Legacy SCM api to declare the lib, because the modern scm dropdown is empty, and due to the nice workaround checking out the library to src/library i can't easily switch to the modern SCM api.

      The library is checked out just fine in workspace/pipeline@libs/src/mylib , but it looks like the polling is confused due to this.

          [JENKINS-38679] Constant builds when svn url contains parameter

          Anton Lundin added a comment -

          Sorry for the simpe typo. What i ment was “every SCM poll call”. Or don't you know what scm polling is?

          Anyhow, if you're not planning to support the "Legacy SCM api" I'd suggest you completly remove the support for it, so others don't need to waste their time trying to get workflow-cps-global-lib-plugin working with it.

          Anyhow, i've solved my problems with the pipeline-classpath plugin instead. Not as pretty, but it works, way better.

          Anton Lundin added a comment - Sorry for the simpe typo. What i ment was “every SCM poll call”. Or don't you know what scm polling is? Anyhow, if you're not planning to support the "Legacy SCM api" I'd suggest you completly remove the support for it, so others don't need to waste their time trying to get workflow-cps-global-lib-plugin working with it. Anyhow, i've solved my problems with the pipeline-classpath plugin instead. Not as pretty, but it works, way better.

          Jesse Glick added a comment -

          Legacy SCM is supported. The “legacy place in Subversion” sounds dubious.

          Yes I know what SCM polling is, but not what it has to do with this issue. Shared libraries do not participate in polling.

          Do not what pipeline-classpath is but it is not supported and sounds like a possible security hole.

          Jesse Glick added a comment - Legacy SCM is supported. The “legacy place in Subversion” sounds dubious. Yes I know what SCM polling is, but not what it has to do with this issue. Shared libraries do not participate in polling. Do not what pipeline-classpath is but it is not supported and sounds like a possible security hole.

          Anton Lundin added a comment -

          Shared libraries do not participate in polling.

          , yes, they do and they cause every scm poll call to trigger a build, due to the scm poll function not finding the library in the jobs workspace, so it things its something new and thus a new build is required.

          Thats the bug I've reported all a long.

          With pipeline-chasspath i referrer to pipeline-classpath-step-plugin, its mentioned under different names in different places. Why are you saying that its a possible security hole?

          Anton Lundin added a comment - Shared libraries do not participate in polling. , yes, they do and they cause every scm poll call to trigger a build, due to the scm poll function not finding the library in the jobs workspace, so it things its something new and thus a new build is required. Thats the bug I've reported all a long. With pipeline-chasspath i referrer to pipeline-classpath-step-plugin, its mentioned under different names in different places. Why are you saying that its a possible security hole?

          Jesse Glick added a comment -

          they do

          If reproducible, that is a bug. According to sources they do not.

          Jesse Glick added a comment - they do If reproducible, that is a bug. According to sources they do not.

          Brian Ray added a comment - - edited

          jglick, I think I'm experiencing shared libs participating in polling that glance is mentioning.

          My little test pipeline is using two shared global libs and one folder-specific shared lib; as well as using a polling checkout of the actual project codebase the pipeline is to build. The shared pipeline libs and the project codebase are in SVN servers. I switched over to the Modern Subversion config facility but experienced this at one point with Legacy as well. The pipeline is configured to poll SCM every few hours.

          If I'm not mistaken the poll log screenshot

          reveals that the shared libs are participating in SCM polling. The first three log messages there are for those libs.

          Setup:

          LTS 2.19.1 on Win7 Pro
          Zulu OpenJDK 1.8.0_102-b14
          Pipeline: SCM Step 2.2
          Pipeline: Shared Groovy Libraries 2.4
          SCM API 1.3
          Subversion 2.7.1

          I'm wondering if this is somehow related to the business with multiple shared libs you recently submitted a patch for in the course of JENKINS-38517.

          On further thought, that seems like a reach. But I think the shared libs are still getting caught up in the SCM poll.

          Brian Ray added a comment - - edited jglick , I think I'm experiencing shared libs participating in polling that glance is mentioning. My little test pipeline is using two shared global libs and one folder-specific shared lib; as well as using a polling checkout of the actual project codebase the pipeline is to build. The shared pipeline libs and the project codebase are in SVN servers. I switched over to the Modern Subversion config facility but experienced this at one point with Legacy as well. The pipeline is configured to poll SCM every few hours. If I'm not mistaken the poll log screenshot reveals that the shared libs are participating in SCM polling. The first three log messages there are for those libs. Setup: LTS 2.19.1 on Win7 Pro Zulu OpenJDK 1.8.0_102-b14 Pipeline: SCM Step 2.2 Pipeline: Shared Groovy Libraries 2.4 SCM API 1.3 Subversion 2.7.1 I'm wondering if this is somehow related to the business with multiple shared libs you recently submitted a patch for in the course of JENKINS-38517 . On further thought, that seems like a reach. But I think the shared libs are still getting caught up in the SCM poll.

          Anton Lundin added a comment - - edited

          Subversion Polling Log

          Started on Oct 27, 2016 5:05:23 PM
          Received SCM poll call on master for pipeline-trunk on Oct 27, 2016 5:05:23 PM
          https://svn/project/data/pipeline-job is at revision 654
          Workspace doesn't contain https://svn/global-library-repo\$\{library.ladok3.version}/legacy/path/mylib. Need a new build.
          Received SCM poll call on master for pipeline-trunk on Oct 27, 2016 5:05:23 PM
          https://svn/project is at revision 877
          Ignored revision 877: Found no included paths
          Done. Took 0.2 sec
          Changes found

          So, pipeline code haven't changed, The code haven't changed, but because it can't find the library in the workspace, it triggers a build.

          Anton Lundin added a comment - - edited Subversion Polling Log Started on Oct 27, 2016 5:05:23 PM Received SCM poll call on master for pipeline-trunk on Oct 27, 2016 5:05:23 PM https://svn/project/data/pipeline-job is at revision 654 Workspace doesn't contain https://svn/global-library-repo\$\ {library.ladok3.version}/legacy/path/mylib. Need a new build. Received SCM poll call on master for pipeline-trunk on Oct 27, 2016 5:05:23 PM https://svn/project is at revision 877 Ignored revision 877: Found no included paths Done. Took 0.2 sec Changes found So, pipeline code haven't changed, The code haven't changed, but because it can't find the library in the workspace, it triggers a build.

          Jesse Glick added a comment -

          Dunno, will have to see if it is reproducible in a functional test when I have some time.

          Jesse Glick added a comment - Dunno, will have to see if it is reproducible in a functional test when I have some time.

          Ben M added a comment -

          The problem is actually that the variable does not get resolved when checking for existence. This is still an issue.

          Ben M added a comment - The problem is actually that the variable does not get resolved when checking for existence. This is still an issue.

          Ben M added a comment -

          Problematic code is in SubversionSCM.java.

          Lines 1435 following:

           

                          try {
                              url = loc.getSVNURL().toDecodedString();
                          } catch (SVNException ex) {
                              listener.error(Messages.SubversionSCM_pollChanges_exception(loc.getURL()));
                              return BUILD_NOW;
                          }
                          if (!baseline.revisions.containsKey(url)) {
                              listener.getLogger().println(Messages.SubversionSCM_pollChanges_locationNotInWorkspace(url));
                              return BUILD_NOW;
                          }
          

          The if checks on the variable url, when it is not yet expanded.

          Shouldn't be too hard to fix?

           

           

          Ben M added a comment - Problematic code is in  SubversionSCM.java . Lines 1435 following:   try { url = loc.getSVNURL().toDecodedString(); } catch (SVNException ex) { listener.error(Messages.SubversionSCM_pollChanges_exception(loc.getURL())); return BUILD_NOW; } if (!baseline.revisions.containsKey(url)) { listener.getLogger().println(Messages.SubversionSCM_pollChanges_locationNotInWorkspace(url)); return BUILD_NOW; } The if checks on the variable url , when it is not yet expanded. Shouldn't be too hard to fix?    

          Morten Breum added a comment -

          I observe the same issue, and I agree with Ben M's comment.

          My polling log continuously says (base URL replaced)

          Started on Mar 30, 2020 5:38:00 PM
          Workspace doesn't contain https://<base_url>/products/product_a/${CODE_REPO_SUB_LOC}. Need a new build.
          Workspace doesn't contain https://<base_url>/products/product_a/${CODE_REPO_SUB_LOC}. Need a new build.
          Done. Took 0 ms
          Changes found
          

          My svn repository contains products/product_a/trunk/ and products/product_a/trunk/tags/1.0.0. There is no difference between building trunk and a tag, so there is one job, Build_ProductA, to do that. That job is invoked from either:

          • job Release_ProductA
            with the parameter CODE_REPO_SUB_LOC set to /tags/1.0.0
            And the build output is uploaded to a public nexus server
          • job Snapshot_ProductA
            with the parameter CODE_REPO_SUB_LOC set to /trunk
            And the build output is uploaded to a private stash
          • SCM polling
            with the parameter set to /trunk (from default)

          Because of this bug, I have to have an extra separate job to avoid building when there are no changes.

          Morten Breum added a comment - I observe the same issue, and I agree with Ben M's comment . My polling log continuously says (base URL replaced) Started on Mar 30, 2020 5:38:00 PM Workspace doesn't contain https://<base_url>/products/product_a/${CODE_REPO_SUB_LOC}. Need a new build. Workspace doesn't contain https://<base_url>/products/product_a/${CODE_REPO_SUB_LOC}. Need a new build. Done. Took 0 ms Changes found My svn repository contains products/product_a/trunk/ and products/product_a/trunk/tags/1.0.0 . There is no difference between building trunk and a tag, so there is one job, Build_ProductA , to do that. That job is invoked from either: job Release_ProductA with the parameter CODE_REPO_SUB_LOC set to /tags/1.0.0 And the build output is uploaded to a public nexus server job Snapshot_ProductA with the parameter CODE_REPO_SUB_LOC set to /trunk And the build output is uploaded to a private stash SCM polling with the parameter set to /trunk (from default) Because of this bug, I have to have an extra separate job to avoid building when there are no changes.

            Unassigned Unassigned
            glance Anton Lundin
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: