-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Platform: All, OS: All
-
Powered by SuggestiMate
We're using Maven 2 to build our modules. I've created the projects in
Hudson and set a build trigger to build whenever a SNAPSHOT dependency
is built. Unfortunately, it doesn't appear to do this in a
breadth-first dependency order.
i.e.
All of our projects bar one have as a parent our company wide pom,
which defines repositories, profiles, etc.
If this project is built, it kicks of a build for the world, but lots
of them are built repeatedly.
e.g. Project Common contains our common POM.
Project Alpha depends on Project Gamma and Project Common.
Project Gamma depends on Project Common only.
If Project Common gets built, Project Alpha seems to get built next
(due to alphabetic ordering somewhere?), then Project Gamma is built,
and the Project Alpha is built again, since Project Gamma has just
been built.
When we have 20 projects, this gets a little inefficient, with certain
projects being built 3 or 4 times when they really only should be
built once.
- maven-plugin.hpi
- 2.20 MB
- patch V1.3.diff
- 20 kB
- patch V1.2.diff
- 14 kB
- patch.txt
- 19 kB
- patch.txt
- 18 kB
- is blocking
-
JENKINS-4076 Maven2 Build no longer orders sub-module builds
-
- Closed
-
- is duplicated by
-
JENKINS-2306 upstream build queue
-
- Closed
-
-
JENKINS-1545 Strange build trigger behaviour
-
- Closed
-
-
JENKINS-2630 Maven2 job building with multiple executors
-
- Closed
-
[JENKINS-2736] Inefficient build order for Maven2 projects
+1 for this.
I've also been asking about this on the mailing list. The volume of triggered
builds in these kind of scenarios is just crazy.
I'm not using a parent pom, but have many interdependent snapshot projects (no
cyclic relations). A small update to one of the base projects without many
dependencies can lead to some of the others being build 10-15 times.
Not only does this consume vast unnecessary resouces, but I think would make
file fingerprints useless?
This issue could prevent us moving to Hudson from CC. Too many false build
failures occur when dependencies don't build first.
We have currently aproximately 12 projects in hudson. When a snapshot dependency
is build, the number of builds before everything is correctly build is insane.
Some projects build more than 6 times. In total that means more at least 30 - 40
builds to complete one cycle. That is unacceptable.
I would like to suggest to raise the priority. I am just new here, so I am not
right away going to change it myself.
For my team eval of Hudson (we currently use CC successfully), I did these two
Hudson config changes to help minimize this:
- Turned off "Build whenever a SNAPSHOT dependency is built", and configured
"Build after other projects are built" - explicitly defining the deps. While
this works to prevent quite a bit of the randomness, it causes us to reproduce
the POM dependencies in two places: POM and Hudson. This is the same as CC.
- Reduced to only 1 build queue to prevent parallel builds. That's ok for now,
with only 9 components configured, but that won't scale...
However, we still have far too many false build failures, as one component can
detect it needs a build before a dependent one does; we have to determine a good
solution to this problem, or I don't know how we (and others) can use it!
CC has the concept of a "build veto", which checks if dependent projects need to
build, and if so, prevents the current project from building. This can lead to
some starvation of downstream components, but we find that far preferable to
false build failures (which removes the conditioning to quick response to build
failures!). I think Hudson needs this kind of feature to solve this issue.
I was just looking through the code, and if i am not mistaking hudson already
has the functionality in place for this.
There is a check in DependencyGraph.java:
/**
- Returns true if a project has a non-direct dependency to another project.
- <p>
- A non-direct dependency is a path of dependency "edge"s from the source
to the destination, - where the length is greater than 1.
*/
public boolean hasIndirectDependencies(AbstractProject src, AbstractProject dst)
{
Set<AbstractProject> visited = new HashSet<AbstractProject>();
Stack<AbstractProject> queue = new Stack<AbstractProject>();
queue.addAll(getDownstream(src));
queue.remove(dst);
while(!queue.isEmpty())
{ AbstractProject p = queue.pop(); if(p==dst) return true; if(visited.add(p)) queue.addAll(getDownstream(p)); } return false;
}
this method will never return true, because:
AbstractProject p = queue.pop();
if(p==dst)
dst will never be in the list.
It is removed by calling: queue.remove(dst);
What this method should recursively check is: Does a downstream project of this
project have dst as dependency.
Sorry I missed a line,
if(visited.add(p))
queue.addAll(getDownstream(p));
So this seems not to be the problem. But wat is?
I have been working on a possible fix for this issue.
What i have seen so far: There are two types of build: A MavenModuleBuild and a
MavenModuleSetBuild.
A MavenModuleBuild only triggers a downstream project when there is no non-
direct path to the downstream project. A MavenModuleSetBuild does not do this,
this does just trigger everything.
Secondly the dependency graph is not build for MavenModuleSets.
My solution consists of two parts:
1. Introduce an AbstractMavenBuild base class for MavenModuleSetBuild and
MavenModuleBuild. This class handles the scheduling of downstream projects. The
code for scheduling is moved (and only slightly adapted) from MavenModuleBuild.
2. Fix the dependency graph, a MavenModuleSet now request its modules to fill
the graph.
3. This is not directly a fix for the triggering: I implemented the
isBuildingBlocked for AbstractMavenProject to return true, when an upstream
project is in the queue.
I have attached a patch file for this (First time I have created a patch file t
so I hope it is working OK)
Added new version of the patch. Updated to the SVN head.
Also added check if the project is already queued when triggering. If the
project is already queued, it is not triggered again.
Just wondering – if maven-reactor-plugin [1] was augmented to include a
complement of the make-scm-changes [2] goal, would that help some of the voters
of this Hudson enhancement?
make-scm-changes: "Goal to build all projects that you personally have changed
(according to SCM)"
The new complementary goal would:
1) Check SCM to see if any modules needed updates.
2) Update modules if necessary.
3) Build changed modules and their inter-reactor downstream modules in the
spirit of the build-sorting/ordering algorithm described in the "Guide to
Working with Multiple Modules" [3]. Maybe the implementation could just pass
the list of updated modules from step 2 to the make-dependents [4] goal.
If that new goal were created, I could more easily create efficient CI reactors
(using svn-externals in some circumstances).
Thoughts?
[1] http://maven.apache.org/plugins/maven-reactor-plugin/
[2] http://maven.apache.org/plugins/maven-reactor-plugin/make-scm-changes-mojo.html
[3]
http://svn.apache.org/viewvc/maven/site/trunk/src/site/apt/guides/mini/guide-multiple-modules.apt?view=markup&pathrev=740693
[4] http://maven.apache.org/plugins/maven-reactor-plugin/make-dependents-mojo.html
Luke,
I would like a fix for the issue. My druthers would be requiring zero
configuration from me as an end-user; it should Just Work and do the build as a
prioritized queue. In my view, this should fork-and-join within the available
number of build executors / jobs waiting to be built.
What you're talking about seems to be a solution on the maven level and while I
welcome any input / solutions to this issue, is solving it within Maven solving
it at the right level? Or is that the only possible solution?
Cheers,
James
James,
> I would like a fix for the issue. My druthers would be requiring zero
> configuration from me as an end-user; it should Just Work and do the build as a
> prioritized queue. In my view, this should fork-and-join within the available
> number of build executors / jobs waiting to be built.
I agree, that would be the ideal solution.
> is solving it within Maven solving it at the right level?
It would be nice to keep the core of the logic in one place. For sure, Maven
needs to be able to understand interdependencies and correct build order. It
would be nice if Hudson could just use some exposed (currently existing or new)
Maven API. I haven't studied the patches in detail so maybe that's what the
proposed fix is doing anyway.
Regards,
Luke
I don't have a clue what the last few comments have to do with this issue, but
is anybody going to do something about this? I have proposed two patches
already, but it seems nobody is actually doing something with it.
I asked again on the developer mailing list how we can drive this issue to conclusion. See the mailing list
archive to follow up:
I made some adjustments which i will test this week, after that i will add a new
patch (Also for the migration of maven to a plugin).
1. Made the "Block building if an upstream project is building" configurable.
This is by default on. By turning this off you can prevent starvation of this
specific project, at the cost of more (unneeded) builds.
2. Removed not triggering a project when an upstream dependency is building or
queued. By using option 1 a project can determine for itself if it wants to
build immediately or wait until its dependencies are ready.
I added a new patch containing the changes i mentioned earlier.
There is only one thing i cannot get to work:
The default on for "Block building if an upstream project is building".
When creating a new project it works, but for existing projects is does not.
So unless somebody knows how to fix this, you have to do this manually.
Wouldn't it be simpler to just sort the job queue in maven dependency order
every time new jobs are added to it?
Has anyone tried the Join plugin?
(http://wiki.jenkins-ci.org//display/JENKINS/Join+Plugin)
It looks as though this could be used to get round the problem, although it does
require manual configuration.
Why should a different plugin be needed to fix incorrect behaviour of the maven
plugin?
I don't think it should, but as this issue has been around for a long time (see
2072) I was wondering whether anyone had tried the Join plugin as a workaround.
I find this issue as frustrating as everyone else and am surprised that such an
obvious deficiency hasn't been addressed yet, especially as this defect has more
votes than any other.
I have tried both JoinPlugin and patch proposed by riksmith and JoinPlugin
didn't work for me. IMHO JoinPlugin is not good for maven projects as it
requires manual configuration contrary patch V1.3.
Attached patch works very well and I vote for including it in hudson source code
as it very increased performance of our hudson farm by avoiding useless builds
(we have one project split into about 50 maven modules).
I very thank to riksmith for the patch.
Tom
Any progress on this one? Due to this bug, a single commit build for our project can take over 3 hours,
whereas a full build only takes 15 minutes . Judging from the vote count, it seems we are not alone...
Ok, I've applied riksmith's changes. I'm committing the changes now, and will
also attach my local build of maven-plugin - I don't have time to do much
testing right now myself, but everything looks solid. If the many and various
people interested in this issue would be willing to test out the changes
themselves, well, it'd be much appreciated. =)
Created an attachment (id=777)
Patched 1.315-SNAPSHOT Maven plugin with fix for issue 2736
Code changed in hudson
User: : abayer
Path:
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/AbstractMavenBuild.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/AbstractMavenProject.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/MavenBuild.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/MavenModuleSet.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
trunk/hudson/main/maven-plugin/src/main/resources/hudson/maven/MavenModuleSet/configure-entries.jelly
trunk/hudson/main/war/resources/help/maven/block-upstream-building.html
http://fisheye4.cenqua.com/changelog/hudson/?cs=19614
Log:
JENKINS-2736 Application of patch submitted by riksmith.
I tried to build this revision for test purposes.
$ svn info
Path: .
URL: https://svn.dev.java.net/svn/hudson/trunk/hudson
Repository Root: https://svn.dev.java.net/svn/hudson
Repository UUID: 71c3de6d-444a-0410-be80-ed276b4c234a
Revision: 19614
Node Kind: directory
Schedule: normal
Last Changed Author: abayer
Last Changed Rev: 19614
Last Changed Date: 2009-07-11 16:24:42 +0100 (Sat, 11 Jul 2009)
$ mvn clean install
...
Results :
Failed tests:
testFooterContainsProfileWithinQuotation(hudson.plugins.google.analytics.FooterWebTest)
Tests in error:
testSubmittingConfigurationUpdatesProfileId(hudson.plugins.google.analytics.FooterWebTest)
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.
Please refer to
/home/jabley/work/eclipse/main/hudson/plugins/googleanalytics/target/surefire-reports
for the individual test results.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41 minutes 55 seconds
[INFO] Finished at: Sun Jul 12 10:20:01 BST 2009
[INFO] Final Memory: 111M/251M
[INFO] ------------------------------------------------------------------------
This is what stopped me from writing a patch in the first place - the fact that
I've never been able to checkout a revision of Hudson that builds successfully.
What am I missing to get the build to succeed?
jabley - I'd suggest just running mvn in hudson/main - you don't need to build
all the plugins.
If the patch was checked in on the 11th, does that mean Hudson 1.316 contains the
maven2 plugin with this patch?
Ok, I finally got a chance to put together a testbed for this, and it mostly
works. The "block build when dependency is building" logic doesn't seem to work
quite right, though, so we still end up with excessive builds - i.e., in my
testbed, hudson/main/pom.xml kicks off both hudson/main/cli and
hudson/main/core, which it should just kick off hudson/main/cli, since
hudson/main/core depends on cli. Trying to figure out what's going on in there now.
Code changed in hudson
User: : abayer
Path:
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/AbstractMavenBuild.java
trunk/hudson/main/maven-plugin/src/main/java/hudson/maven/AbstractMavenProject.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=19980
Log:
JENKINS-2736 Tweaked a couple things - should now check to see if (a) any upstream projects of a dependency are also in the list of builds to be kicked off, and if there are any, skip this dependency's build (since it'll get kicked off later), and (b) any upstream projects of a dependency are currently in queue or building, and if so, skips this dependency's build. These could possibly be folded in together, but I've got them separate for now.
I think I've got this actually fixed! It's still not entirely perfect (I'm
running into some strangeness in Hudson's logic for determining
upstream/downstream dependencies in the first place, but that's a separate
issue), but it's working. This'll be live in 1.317.
Code changed in hudson
User: : abayer
Path:
trunk/www/changelog.html
http://fisheye4.cenqua.com/changelog/hudson/?cs=19981
Log:
JENKINS-2736 Updating changelog for issue 2736
Additional testing on one Maven project with multiple modules, using the "build
modules in parallel" option, has shown it to still be a little overeager in
kicking off builds - I'm still trying to nail down exactly why.
Ah. It was because I copied in the wrong private build. Whoopsie. All is good.
As raised on the mailing list:
https://hudson.dev.java.net/servlets/ReadMsg?list=users&msgNo=14841