Hi. We upgraded our old Hudson server to Jenkins 1.409.2 on Sep, 20th. Since then we're facing bigger build times, like twice the time it used to take with Hudson.

We're running Jenkins on a RHEL 5.2, 32 bits box, with Java 1.6.0_04. The build agent runs on a RHEL 5.4, 64 bits box, with Java 1.6.0_23.

I'm attaching the System_Information_Jenkins.html page which contains the configuration of our Jenkins server.

I'm also attaching the config.xml file of a particular job which shows the problem well.

I'm also attaching the etics-6.1.x-es_76_Console.html file with the Console output of a particular build. The output lines are prefixed with the time of day (via the timestamper plugin) so that we can see where most of the time was spent. As you can see, the build started at 10:47:17 and ended at 11:26:36, having taken 39 minutes and 19 seconds in the whole. However, as you can see at the end of the log, maven tells us that the build took only 11 minutes and 3 seconds. Most of the remaining 28 minutes and 16 seconds was spent between the lines prefixed with times 10:47:54 and 11:15:21.

I don't know what the Jenkins server or the agent were doing during this quiet period. I tried to see which processes were busy during this period and the only strange thing I noticed was that there was a process in the agent machine very busy, which was this:

/l/disk0/ipscate/tools/java/jdk1.6.0_23/bin/java -jar	slave.jar

I straced it during a few seconds and saw that it was busy opening lots of files under the /l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd directory. In fact, in a short period of about 15 or 20 seconds it opened more than 40000 files, some of them being repeatedly opened thousands of times. These were the most often opened ones:

# sort /tmp/trace | uniq -c | sort -n | tail -5
   3502 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", O_RDONLY) = 16
   3502 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/cpqd-public-bom-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=45211, ...}) = 0
   3793 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", O_RDONLY) = 16
   3793 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=14792, ...}) = 0
   7004 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", {st_mode=S_IFREG|0644, st_size=163, ...}) = 0

I'm not sure this behaviour is directly related with the delay I was seeing in the build, but it doesn't seem right to me.

Do you have any suggestion as to how can I go about investigating this?