Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18293

Team Concert plugin cannot cope with symbolic links

      Loading a stream with 3 components, where 1 component has symbolic links.
      Loading the 2 components without symbolic links works fine, but as soon as the component with symbolic links is loaded, the Jenkins Job fails with the following messages:

      Jenkins Job log:
      ================
      Started by user Beerens, Danny
      [EnvInject] - Loading node environment variables.
      Building in workspace C:\Jenkins\Workspace
      RTC : checkout...
      RTC Checkout : Source control setup
      RTC Checkout : Accepting changes into workspace "sys24_Team Workspace" ...
      RTC Checkout : Fetching files to fetch destination "C:\Jenkins\Workspace\EPNav" ...
      FATAL: RTC : checkout failure: null
      java.lang.NullPointerException
      at com.ibm.team.filesystem.client.internal.core.SharingMetadata2.getFileItemInfo(SharingMetadata2.java:2539)
      at com.ibm.team.filesystem.client.internal.copyfileareas.CopyFileAreaStore.getLocalPathInternal(CopyFileAreaStore.java:944)
      at com.ibm.team.filesystem.client.internal.copyfileareas.CopyFileAreaStore.setConfigurationState(CopyFileAreaStore.java:2992)
      at com.ibm.team.filesystem.client.internal.load.MergeLoadMutator.load(MergeLoadMutator.java:825)
      at com.ibm.team.filesystem.client.internal.load.MergeLoadMutator.execute(MergeLoadMutator.java:312)
      at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.run(FileSystemOperation.java:89)
      at com.ibm.team.filesystem.client.internal.operations.LoadOperation.load(LoadOperation.java:1047)
      at com.ibm.team.filesystem.client.internal.operations.LoadOperation$1.run(LoadOperation.java:664)
      at com.ibm.team.filesystem.client.internal.SharingManager.runWithinFileSystemLock(SharingManager.java:935)
      at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:164)
      at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:144)
      at com.ibm.team.filesystem.client.internal.operations.LoadOperation.execute(LoadOperation.java:695)
      at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.run(FileSystemOperation.java:89)
      at com.ibm.team.build.internal.scm.SourceControlUtility.updateFileCopyArea(SourceControlUtility.java:678)
      at com.ibm.team.build.internal.hjplugin.rtc.RepositoryConnection.checkout(RepositoryConnection.java:367)
      at com.ibm.team.build.internal.hjplugin.rtc.RTCFacade.checkout(RTCFacade.java:321)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory$RTCFacadeWrapper.invoke(RTCFacadeFactory.java:92)
      at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:103)
      at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:30)
      at hudson.FilePath.act(FilePath.java:904)
      at hudson.FilePath.act(FilePath.java:877)
      at com.ibm.team.build.internal.hjplugin.RTCScm.checkout(RTCScm.java:836)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
      at hudson.model.Run.execute(Run.java:1575)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:241)
      ERROR: RTC : checkout failure: null
      Finished: FAILURE

          [JENKINS-18293] Team Concert plugin cannot cope with symbolic links

          Scott Cowan added a comment -

          I've adjusted the priority using the Jenkins guidelines,

          Blocker
          Blocks development and/or testing work, production could not run.
          Critical
          Crashes, loss of data, severe memory leak.
          Major
          Major loss of function.
          Minor
          Minor loss of function, or other problem where easy workaround is present.
          Trivial
          Cosmetic problem like misspelled words or misaligned text

          Scott Cowan added a comment - I've adjusted the priority using the Jenkins guidelines, Blocker Blocks development and/or testing work, production could not run. Critical Crashes, loss of data, severe memory leak. Major Major loss of function. Minor Minor loss of function, or other problem where easy workaround is present. Trivial Cosmetic problem like misspelled words or misaligned text

          Scott Cowan added a comment -

          Also Danny, could you indicate which platform and version of the build toolkit you're using?

          Thanks,
          Scott

          Scott Cowan added a comment - Also Danny, could you indicate which platform and version of the build toolkit you're using? Thanks, Scott

          RTC support for symbolic links requires one or two libraries (.dll/.so files).

          • Eclipse file system natives
          • RTC file system natives

          The reason is Java 6 and earlier don't have support for creating/looking at properties of symbolic links. So RTC uses some C code to manage symbolic links.

          Java 7 has symbolic link support that works on linux, but on Windows it has some limitations when creating links when the target has not yet been created (type is defaulted to file which is not good if its a directory).

          If you are running Linux and can use Java 7 then the only .so you need is for the Eclipse natives. Otherwise, you will need both.

          I'll describe what has to happen on Windows, but the steps are similar for linux.

          In the Build engine directory (buildengine\eclipse\plugins), look for the
          com.ibm.team.filesystem.client_3.1.600.v20130415_0257.jar
          org.eclipse.core.filesystem.win32.x86_1.1.201.R36x_v20100727-0745.jar or equivalent jars for your platform/release.

          From the com.ibm.team.filesystem.client jar you want to extract the winfsnatives.dll (libfsnatives.so on linux). Take all the .dll/.so from the org.eclipse.core.filesystem jar. Place them directly in a directory (eg. c:\natives\winfsnatives.dll).

          Now when you start Jenkins, we need to tell java about the directory so that it can load the libraries from it. This is done through the -Djava.library.path variable.
          eg. java -Djava.library.path="c:\natives;%Path%" -jar jenkins-1.509.1.war

          We are just adding to what the default would be for Java. You could also add the directory to your path instead.

          I realize the setup is a little convoluted, we could probably help by packaging the toolkit a bit differently. But this might be simpler than trying to load the component through the CLI.

          Heather Fraser-Dube added a comment - RTC support for symbolic links requires one or two libraries (.dll/.so files). Eclipse file system natives RTC file system natives The reason is Java 6 and earlier don't have support for creating/looking at properties of symbolic links. So RTC uses some C code to manage symbolic links. Java 7 has symbolic link support that works on linux, but on Windows it has some limitations when creating links when the target has not yet been created (type is defaulted to file which is not good if its a directory). If you are running Linux and can use Java 7 then the only .so you need is for the Eclipse natives. Otherwise, you will need both. I'll describe what has to happen on Windows, but the steps are similar for linux. In the Build engine directory (buildengine\eclipse\plugins), look for the com.ibm.team.filesystem.client_3.1.600.v20130415_0257.jar org.eclipse.core.filesystem.win32.x86_1.1.201.R36x_v20100727-0745.jar or equivalent jars for your platform/release. From the com.ibm.team.filesystem.client jar you want to extract the winfsnatives.dll (libfsnatives.so on linux). Take all the .dll/.so from the org.eclipse.core.filesystem jar. Place them directly in a directory (eg. c:\natives\winfsnatives.dll). Now when you start Jenkins, we need to tell java about the directory so that it can load the libraries from it. This is done through the -Djava.library.path variable. eg. java -Djava.library.path="c:\natives;%Path%" -jar jenkins-1.509.1.war We are just adding to what the default would be for Java. You could also add the directory to your path instead. I realize the setup is a little convoluted, we could probably help by packaging the toolkit a bit differently. But this might be simpler than trying to load the component through the CLI.

          Also, looking at the trace included leads me to think that your sandbox (place where the load is occurring) is corrupted. The lack of symbolic link support should have generated a nicer error message. So I would also recommend that you delete everything that has been loaded before you try it out.

          Heather Fraser-Dube added a comment - Also, looking at the trace included leads me to think that your sandbox (place where the load is occurring) is corrupted. The lack of symbolic link support should have generated a nicer error message. So I would also recommend that you delete everything that has been loaded before you try it out.

          Finally, if you are running on Windows, you need to be sure that you have permission to create symbolic links. The Symbolic links article in the jazz.net library https://jazz.net/library/article/970/ describes how.

          Heather Fraser-Dube added a comment - Finally, if you are running on Windows, you need to be sure that you have permission to create symbolic links. The Symbolic links article in the jazz.net library https://jazz.net/library/article/970/ describes how.

          Nick Edgar added a comment -

          Danny had responded by email (before Heather's responses above):
          >> This is a tricky one, since the ability to create a symbolic link can also be limited by the OS (Win 7 supports it, by requires updated user & application security settings. Win XP requires a patch). However the system and userids I’m using are set-up to allow the creation of symbolic links. Therefore it stands to reason it’s either the application security settings, or the ‘inability’ of the application.
          >> Regarding the ability of the application: If you’re utilizing the (l)scm load command, it usually does create the symbolic links when we start it from the command line in a CMD file ourselves. If run the same CMD file in a Jenkins Job, it usually doesn’t. We haven’t been able to figure out why…
          >> Regarding the security settings to allow creation of symlinks: Can you indicate what the (user/application security) settings should be, to allow the plugin to create symbolic links on Win7?
          >> You have a good suggestion, but it’s only a workaround. We’re currently looking into it.

          Hopefully the article Heather linked to above helps with the question of settings.

          Nick Edgar added a comment - Danny had responded by email (before Heather's responses above): >> This is a tricky one, since the ability to create a symbolic link can also be limited by the OS (Win 7 supports it, by requires updated user & application security settings. Win XP requires a patch). However the system and userids I’m using are set-up to allow the creation of symbolic links. Therefore it stands to reason it’s either the application security settings, or the ‘inability’ of the application. >> Regarding the ability of the application: If you’re utilizing the (l)scm load command, it usually does create the symbolic links when we start it from the command line in a CMD file ourselves. If run the same CMD file in a Jenkins Job, it usually doesn’t. We haven’t been able to figure out why… >> Regarding the security settings to allow creation of symlinks: Can you indicate what the (user/application security) settings should be, to allow the plugin to create symbolic links on Win7? >> You have a good suggestion, but it’s only a workaround. We’re currently looking into it. Hopefully the article Heather linked to above helps with the question of settings.

          Jan Wloka added a comment -

          Had the same problem on a Linux box using OpenJDK 1.7. As a workaround I replaced OpenJDK with the IBM JDK 6. That fixed the problem on my box.

          http://www.ibm.com/developerworks/java/jdk/linux/download.html

          Jan Wloka added a comment - Had the same problem on a Linux box using OpenJDK 1.7. As a workaround I replaced OpenJDK with the IBM JDK 6. That fixed the problem on my box. http://www.ibm.com/developerworks/java/jdk/linux/download.html

          Updated the help and the main page to indicate the setup for symbolic links

          Heather Fraser-Dube added a comment - Updated the help and the main page to indicate the setup for symbolic links

          bbonn added a comment -

          Hi There,

          Is there a work item for a more formal fix to this issue. We have a Jenkins farm with over 70 slaves. Which means I would have to do this workaround for all of them, then maintain it per RTC version. Let me know if I should create request on in RTC for Jazz dev.

          Thanks!

          bbonn added a comment - Hi There, Is there a work item for a more formal fix to this issue. We have a Jenkins farm with over 70 slaves. Which means I would have to do this workaround for all of them, then maintain it per RTC version. Let me know if I should create request on in RTC for Jazz dev. Thanks!

          I understand your problem. I wish we had a better solution. The problem is that code that the build toolkit uses to create/update symbolic links is in C libraries (even when using Java 7 on linux). Therefore in order for them to be loaded they need to be on the path for Java which can't be modified once Java starts. We could improve toolkit packaging to make the libraries more accessible. But you still need to specify the libraries when starting Java.

          Yes, create a request on RTC for this. Maybe other people might have some better ideas. Are your slaves running on Windows or Linux?

          Heather Fraser-Dube added a comment - I understand your problem. I wish we had a better solution. The problem is that code that the build toolkit uses to create/update symbolic links is in C libraries (even when using Java 7 on linux). Therefore in order for them to be loaded they need to be on the path for Java which can't be modified once Java starts. We could improve toolkit packaging to make the libraries more accessible. But you still need to specify the libraries when starting Java. Yes, create a request on RTC for this. Maybe other people might have some better ideas. Are your slaves running on Windows or Linux?

            hfraserdube Heather Fraser-Dube
            dbeerens Danny Beerens
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: