-
Bug
-
Resolution: Unresolved
-
Major
-
Jenkins 1.565.1
TeamConcert 1.1.8
Host: Red Hat Enterprise Linux Server 6.5
RTC 4.0.3
Jenkins running with 2 executors
Sometimes, when running personal builds from RTC, the job will hang indefinitely. The console output for the job shows that the job is in the team concert pluging and is fetching source from RTC.
Looking at the output from jenkins/ThreadDump, it would seem there is a deadlock situation occurring. Here is some example output:
Executor #1 for master : executing Mainline personal build #458
"Executor #1 for master : executing Mainline personal build #458" Id=945 Group=main TIMED_WAITING on java.util.ArrayList@5663b111
at java.lang.Object.wait(Native Method)
- waiting on java.util.ArrayList@5663b111
at com.ibm.team.filesystem.client.internal.copyfileareas.BatchingLock.acquire(BatchingLock.java:446)
at com.ibm.team.filesystem.client.internal.copyfileareas.CopyFileAreaManager.deregister(CopyFileAreaManager.java:350)
at com.ibm.team.filesystem.client.internal.SharingManager.deregister(SharingManager.java:1358)
at com.ibm.team.filesystem.client.internal.SharingManager.deregister(SharingManager.java:1316)
at com.ibm.team.build.internal.hjplugin.rtc.RepositoryConnection.checkout(RepositoryConnection.java:457)
at com.ibm.team.build.internal.hjplugin.rtc.RTCFacade.checkout(RTCFacade.java:390)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory$RTCFacadeWrapper.invoke(RTCFacadeFactory.java:115)
at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:166)
at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:32)
at hudson.FilePath.act(FilePath.java:920)
at hudson.FilePath.act(FilePath.java:893)
at com.ibm.team.build.internal.hjplugin.RTCScm.checkout(RTCScm.java:1079)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
at hudson.model.Run.execute(Run.java:1706)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:232)
I spoke with Scott Cowan over Sametime and he suggested that it was trying to access a lock on a file on the master filesystem and there might be some shared state between builds. I've had a quick look over our builds and I can't see anything obvious that would cause the problem but I will continue to investigate.
It should be noted that this only seems to occur for personal builds where a particular user's workspace is being used. Our normal builds have not seen this issue.
Thanks,
James
Hi James,
Sorry for the delay. Things have been a bit busy here and unfortunately I am away next week. I've looked over our code in this area but don't see anything obvious. The problem is at a lower level than us, but I don't quite understand what would trigger it.
So a bunch of questions...
Does this only happen for 1 user's personal build or is it randomly affecting all personal builds?
If it affects just 1 (or a given set of) user, can you check their workspace and see if its visible to the user that the Jenkins job is configured to use. Also see if there are missing or extra components from what is normally in the build workspace. If there are extra components, are they visible to the user that the Jenkins job is configured to use. I don't think this is the problem, but I want to rule it out too.
The sandbox is where the RTC repository workspace is to be loaded. Its defined on the Source Control tab of the build definition as the Load directory. If you set the path to ".", it will end up being in the Jenkins job's workspace.
I am interested in knowing if you use any special features wrt loading (i.e. in the build definition, do you have delete directory before loading checked? Is Create folders for components checked?)
When it happens, can you look at the job's log. I am interested in all the messages that are related to the check out (or possibly corrupted metadata). It will also help to understand how far it got during the load. I think that they would follow the "RTC Checkout : Source control setup" message.
Does the build result have an activity for Fetching Files... If so, was it completed?
Does trying to cancel the build help when this happens?