-
Bug
-
Resolution: Fixed
-
Major
-
None
-
We're running Jenkins on a RHEL 5.2, 32 bits box, with Java 1.6.0_04. The build agent runs on a RHEL 5.4, 64 bits box, with Java 1.6.0_23.
Hi. We upgraded our old Hudson server to Jenkins 1.409.2 on Sep, 20th. Since then we're facing bigger build times, like twice the time it used to take with Hudson.
We're running Jenkins on a RHEL 5.2, 32 bits box, with Java 1.6.0_04. The build agent runs on a RHEL 5.4, 64 bits box, with Java 1.6.0_23.
I'm attaching the System_Information_Jenkins.html
page which contains the configuration of our Jenkins server.
I'm also attaching the config.xml file of a particular job which shows the problem well.
I'm also attaching the etics-6.1.x-es_76_Console.html file with the Console output of a particular build. The output lines are prefixed with the time of day (via the timestamper plugin) so that we can see where most of the time was spent. As you can see, the build started at 10:47:17 and ended at 11:26:36, having taken 39 minutes and 19 seconds in the whole. However, as you can see at the end of the log, maven tells us that the build took only 11 minutes and 3 seconds. Most of the remaining 28 minutes and 16 seconds was spent between the lines prefixed with times 10:47:54 and 11:15:21.
I don't know what the Jenkins server or the agent were doing during this quiet period. I tried to see which processes were busy during this period and the only strange thing I noticed was that there was a process in the agent machine very busy, which was this:
/l/disk0/ipscate/tools/java/jdk1.6.0_23/bin/java -jar slave.jar
I straced it during a few seconds and saw that it was busy opening lots of files under the /l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd directory. In fact, in a short period of about 15 or 20 seconds it opened more than 40000 files, some of them being repeatedly opened thousands of times. These were the most often opened ones:
# sort /tmp/trace | uniq -c | sort -n | tail -5
3502 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", O_RDONLY) = 16
3502 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/cpqd-public-bom-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=45211, ...}) = 0
3793 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", O_RDONLY) = 16
3793 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=14792, ...}) = 0
7004 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", {st_mode=S_IFREG|0644, st_size=163, ...}) = 0
I'm not sure this behaviour is directly related with the delay I was seeing in the build, but it doesn't seem right to me.
Do you have any suggestion as to how can I go about investigating this?
- depends on
-
-
- Resolved
-
- is related to
-
-
- Closed
-
[JENKINS-11362] Strange delay during a maven job build
I also noticed that the /l/disk0/ipscate/ip/nexus/.m2 directory had 141GB of data below it!
I renamed it and created another empty .m2. Then I submitted another build, but the problem remains.
kutzi
added a comment - Sounds like this could be related to this maven issue https://jira.codehaus.org/browse/MNG-5125 (See http://vivin.net/2011/07/20/fixing-maven-3-0-3s-dependency-resolution-performance-regression/ for some more info)
Maven 3.0.3 is currently used by Jenkins internally to parse the POMs
kutzi
added a comment - Sounds like this could be related to this maven issue https://jira.codehaus.org/browse/MNG-5125 (See http://vivin.net/2011/07/20/fixing-maven-3-0-3s-dependency-resolution-performance-regression/ for some more info)
Maven 3.0.3 is currently used by Jenkins internally to parse the POMs Correcting my previous comment, when I renamed the .m2 directory the next build stalled forever just after the Parsing POMs log line. I waited a few hours and gave up. Now that I returned the previous .m2 directory it seems to be going as previously.
Another thing I tried was to kill all agent.jar processes that were running in the agent machine, but it seemed to have no effect.
@kutzi: it makes sense to me. I'm not a Java guy, so, please forgive me if this is a dumb question, but, what would it take to have that aether-1.12 being used inside Jenkins? Would I have to rebuild Jenkins from scratch after editing some POM file?
How can I know which version of maven does Jenkins use? Perhaps, if I go back to Jenkins 1.409.1 it uses a previous version of maven which doesn't have that problem.
Thank you very much.
kutzi
added a comment - That's not a dumb question at all and I'm not 100% sure how to get it work.
I think it could work, if you replace the aether-*.jar's in the plugins/maven-plugin/WEB-INF/lib directory with the 1.12 versions
You'll probably also have to 'pin' the maven-plugin so it won't be replaced on startup with the bundled one.
Going back to an 1.409.1 (or any other 1.40x version) won't work, as they use AFAIK the same version of maven/aether
kutzi
added a comment - That's not a dumb question at all and I'm not 100% sure how to get it work.
I think it could work, if you replace the aether-*.jar's in the plugins/maven-plugin/WEB-INF/lib directory with the 1.12 versions
You'll probably also have to 'pin' the maven-plugin so it won't be replaced on startup with the bundled one.
Going back to an 1.409.1 (or any other 1.40x version) won't work, as they use AFAIK the same version of maven/aether kutzi
added a comment - Olivier, wouldn't it make sense to bundle the aether 1.12 jars with the maven embedder?
kutzi
added a comment - Olivier, wouldn't it make sense to bundle the aether 1.12 jars with the maven embedder? kutzi
made changes -
| Assignee | New: Olivier Lamy [ olamy ] |
Olivier Lamy
added a comment - should fix that issue sure
But maybe can introduce some discrepancy between the pom parser and a build with mvn 3.x except if users change aether in their mvn installation too.
Olivier Lamy
added a comment - should fix that issue sure
But maybe can introduce some discrepancy between the pom parser and a build with mvn 3.x except if users change aether in their mvn installation too. 
I noticed that I had the maven-plugin pinned at version 1.401. So, I unpinned it and restarted the server, so that now it is at version 1.409.2. Unfortunatelly, the problem remains.