• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • maven-plugin
    • None
    • Platform: All, OS: All

      We're using Maven 2 to build our modules. I've created the projects in
      Hudson and set a build trigger to build whenever a SNAPSHOT dependency
      is built. Unfortunately, it doesn't appear to do this in a
      breadth-first dependency order.

      i.e.

      All of our projects bar one have as a parent our company wide pom,
      which defines repositories, profiles, etc.

      If this project is built, it kicks of a build for the world, but lots
      of them are built repeatedly.

      e.g. Project Common contains our common POM.

      Project Alpha depends on Project Gamma and Project Common.
      Project Gamma depends on Project Common only.

      If Project Common gets built, Project Alpha seems to get built next
      (due to alphabetic ordering somewhere?), then Project Gamma is built,
      and the Project Alpha is built again, since Project Gamma has just
      been built.

      When we have 20 projects, this gets a little inefficient, with certain
      projects being built 3 or 4 times when they really only should be
      built once.

        1. maven-plugin.hpi
          2.20 MB
        2. patch V1.3.diff
          20 kB
        3. patch V1.2.diff
          14 kB
        4. patch.txt
          19 kB
        5. patch.txt
          18 kB

          [JENKINS-2736] Inefficient build order for Maven2 projects

          jabley created issue -

          jabley added a comment -

          jabley added a comment - As raised on the mailing list: https://hudson.dev.java.net/servlets/ReadMsg?list=users&msgNo=14841

          dmosses added a comment -

          +1 for this.

          I've also been asking about this on the mailing list. The volume of triggered
          builds in these kind of scenarios is just crazy.

          I'm not using a parent pom, but have many interdependent snapshot projects (no
          cyclic relations). A small update to one of the base projects without many
          dependencies can lead to some of the others being build 10-15 times.

          Not only does this consume vast unnecessary resouces, but I think would make
          file fingerprints useless?

          dmosses added a comment - +1 for this. I've also been asking about this on the mailing list. The volume of triggered builds in these kind of scenarios is just crazy. I'm not using a parent pom, but have many interdependent snapshot projects (no cyclic relations). A small update to one of the base projects without many dependencies can lead to some of the others being build 10-15 times. Not only does this consume vast unnecessary resouces, but I think would make file fingerprints useless?

          r_m_manning added a comment -

          I believe I see the same issue as well.

          Rob

          r_m_manning added a comment - I believe I see the same issue as well. Rob

          jeffjensen added a comment -

          This issue could prevent us moving to Hudson from CC. Too many false build
          failures occur when dependencies don't build first.

          jeffjensen added a comment - This issue could prevent us moving to Hudson from CC. Too many false build failures occur when dependencies don't build first.

          riksmith added a comment -

          We have currently aproximately 12 projects in hudson. When a snapshot dependency
          is build, the number of builds before everything is correctly build is insane.
          Some projects build more than 6 times. In total that means more at least 30 - 40
          builds to complete one cycle. That is unacceptable.

          I would like to suggest to raise the priority. I am just new here, so I am not
          right away going to change it myself.

          riksmith added a comment - We have currently aproximately 12 projects in hudson. When a snapshot dependency is build, the number of builds before everything is correctly build is insane. Some projects build more than 6 times. In total that means more at least 30 - 40 builds to complete one cycle. That is unacceptable. I would like to suggest to raise the priority. I am just new here, so I am not right away going to change it myself.

          jeffjensen added a comment -

          For my team eval of Hudson (we currently use CC successfully), I did these two
          Hudson config changes to help minimize this:

          • Turned off "Build whenever a SNAPSHOT dependency is built", and configured
            "Build after other projects are built" - explicitly defining the deps. While
            this works to prevent quite a bit of the randomness, it causes us to reproduce
            the POM dependencies in two places: POM and Hudson. This is the same as CC.
          • Reduced to only 1 build queue to prevent parallel builds. That's ok for now,
            with only 9 components configured, but that won't scale...

          However, we still have far too many false build failures, as one component can
          detect it needs a build before a dependent one does; we have to determine a good
          solution to this problem, or I don't know how we (and others) can use it!

          CC has the concept of a "build veto", which checks if dependent projects need to
          build, and if so, prevents the current project from building. This can lead to
          some starvation of downstream components, but we find that far preferable to
          false build failures (which removes the conditioning to quick response to build
          failures!). I think Hudson needs this kind of feature to solve this issue.

          jeffjensen added a comment - For my team eval of Hudson (we currently use CC successfully), I did these two Hudson config changes to help minimize this: Turned off "Build whenever a SNAPSHOT dependency is built", and configured "Build after other projects are built" - explicitly defining the deps. While this works to prevent quite a bit of the randomness, it causes us to reproduce the POM dependencies in two places: POM and Hudson. This is the same as CC. Reduced to only 1 build queue to prevent parallel builds. That's ok for now, with only 9 components configured, but that won't scale... However, we still have far too many false build failures, as one component can detect it needs a build before a dependent one does; we have to determine a good solution to this problem, or I don't know how we (and others) can use it! CC has the concept of a "build veto", which checks if dependent projects need to build, and if so, prevents the current project from building. This can lead to some starvation of downstream components, but we find that far preferable to false build failures (which removes the conditioning to quick response to build failures!). I think Hudson needs this kind of feature to solve this issue.

          riksmith added a comment -

          I was just looking through the code, and if i am not mistaking hudson already
          has the functionality in place for this.

          There is a check in DependencyGraph.java:

          /**

          • Returns true if a project has a non-direct dependency to another project.
          • <p>
          • A non-direct dependency is a path of dependency "edge"s from the source
            to the destination,
          • where the length is greater than 1.
            */
            public boolean hasIndirectDependencies(AbstractProject src, AbstractProject dst)
            {
            Set<AbstractProject> visited = new HashSet<AbstractProject>();
            Stack<AbstractProject> queue = new Stack<AbstractProject>();

          queue.addAll(getDownstream(src));
          queue.remove(dst);

          while(!queue.isEmpty())

          { AbstractProject p = queue.pop(); if(p==dst) return true; if(visited.add(p)) queue.addAll(getDownstream(p)); }

          return false;
          }

          this method will never return true, because:
          AbstractProject p = queue.pop();
          if(p==dst)
          dst will never be in the list.
          It is removed by calling: queue.remove(dst);

          What this method should recursively check is: Does a downstream project of this
          project have dst as dependency.

          riksmith added a comment - I was just looking through the code, and if i am not mistaking hudson already has the functionality in place for this. There is a check in DependencyGraph.java: /** Returns true if a project has a non-direct dependency to another project. <p> A non-direct dependency is a path of dependency "edge"s from the source to the destination, where the length is greater than 1. */ public boolean hasIndirectDependencies(AbstractProject src, AbstractProject dst) { Set<AbstractProject> visited = new HashSet<AbstractProject>(); Stack<AbstractProject> queue = new Stack<AbstractProject>(); queue.addAll(getDownstream(src)); queue.remove(dst); while(!queue.isEmpty()) { AbstractProject p = queue.pop(); if(p==dst) return true; if(visited.add(p)) queue.addAll(getDownstream(p)); } return false; } this method will never return true, because: AbstractProject p = queue.pop(); if(p==dst) dst will never be in the list. It is removed by calling: queue.remove(dst); What this method should recursively check is: Does a downstream project of this project have dst as dependency.

          riksmith added a comment -

          Sorry I missed a line,

          if(visited.add(p))
          queue.addAll(getDownstream(p));

          So this seems not to be the problem. But wat is?

          riksmith added a comment - Sorry I missed a line, if(visited.add(p)) queue.addAll(getDownstream(p)); So this seems not to be the problem. But wat is?

          riksmith added a comment -

          I have been working on a possible fix for this issue.

          What i have seen so far: There are two types of build: A MavenModuleBuild and a
          MavenModuleSetBuild.
          A MavenModuleBuild only triggers a downstream project when there is no non-
          direct path to the downstream project. A MavenModuleSetBuild does not do this,
          this does just trigger everything.

          Secondly the dependency graph is not build for MavenModuleSets.

          My solution consists of two parts:
          1. Introduce an AbstractMavenBuild base class for MavenModuleSetBuild and
          MavenModuleBuild. This class handles the scheduling of downstream projects. The
          code for scheduling is moved (and only slightly adapted) from MavenModuleBuild.

          2. Fix the dependency graph, a MavenModuleSet now request its modules to fill
          the graph.

          3. This is not directly a fix for the triggering: I implemented the
          isBuildingBlocked for AbstractMavenProject to return true, when an upstream
          project is in the queue.

          I have attached a patch file for this (First time I have created a patch file t
          so I hope it is working OK)

          riksmith added a comment - I have been working on a possible fix for this issue. What i have seen so far: There are two types of build: A MavenModuleBuild and a MavenModuleSetBuild. A MavenModuleBuild only triggers a downstream project when there is no non- direct path to the downstream project. A MavenModuleSetBuild does not do this, this does just trigger everything. Secondly the dependency graph is not build for MavenModuleSets. My solution consists of two parts: 1. Introduce an AbstractMavenBuild base class for MavenModuleSetBuild and MavenModuleBuild. This class handles the scheduling of downstream projects. The code for scheduling is moved (and only slightly adapted) from MavenModuleBuild. 2. Fix the dependency graph, a MavenModuleSet now request its modules to fill the graph. 3. This is not directly a fix for the triggering: I implemented the isBuildingBlocked for AbstractMavenProject to return true, when an upstream project is in the queue. I have attached a patch file for this (First time I have created a patch file t so I hope it is working OK)

            abayer Andrew Bayer
            jabley jabley
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: