Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13202

Artifact archiving from an ssh slave fails if symlinks are present

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • Slave must be a "Unix via SSH" slave. I suspect the master must be Unix-based also. I tested with both the master and slave on Linux.

      When archiving artifacts from a job executed on a "Unix via SSH" slave, if a symlink is present and archived before its target, the archiving will fail. It looks like the master is trying to chmod the local symlink (because it has executable permissions stored in the tar used to do the remote copy), but that fails because the target doesn't exist yet, and the exception aborts the archiving. I think the solution is to just not chmod symlinks, since they don't have modes of their own. This was a regression from 1.455 to 1.456. I suspect the fixes for JENKINS-9118 are the cause.

      To reproduce this problem, install the Jenkins master on a Linux machine and add a "Unix via SSH" slave that is also a Linux machine. Create a job with the following build script:

      rm -rf stuff
      mkdir stuff
      cd stuff
      touch zzfile
      ln -s zzfile aafile
      ln -s zzfile bbfile
      

      Restrict this project to build on the slave. Set up archiving for "stuff/*". Run the job. The job will complete successfully, but archiving will abort partway though with a stack trace in the console.

      Archiving artifacts
      ERROR: Failed to archive artifacts: stuff/*
      hudson.util.IOException2: Failed to extract /tmp/jenkins-slave-home/workspace/create_symlinks/stuff/*
      	at hudson.FilePath.readFromTar(FilePath.java:1817)
      	at hudson.FilePath.copyRecursiveTo(FilePath.java:1729)
      	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
      	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
      	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
      	at hudson.model.Build$RunnerImpl.post2(Build.java:162)
      	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
      	at hudson.model.Run.run(Run.java:1435)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:238)
      Caused by: java.io.IOException: Failed to chmod /tmp/jenkins-home/jobs/create_symlinks/builds/2012-03-21_16-53-15/archive/stuff/aafile : No such file or directory
      	at hudson.FilePath._chmod(FilePath.java:1248)
      	at hudson.FilePath.readFromTar(FilePath.java:1813)
      	... 12 more
      

          [JENKINS-13202] Artifact archiving from an ssh slave fails if symlinks are present

          @Chris this is exactly our problem:

          The jobs themselves are fine - it's the huge amount of log messages (40MB within 2 hours) which are annoying and seem to make loading of the dashboard very slow.

          Matthias Fraass added a comment - @Chris this is exactly our problem: The jobs themselves are fine - it's the huge amount of log messages (40MB within 2 hours) which are annoying and seem to make loading of the dashboard very slow.

          Jesse Glick added a comment -

          @chrisgwarp:

          Can not some startup logic be used to test for the presence of readlink and record it's presence and then not attempt to use it if not present?

          Probably; file a pull request. Only someone with AIX can test.

          Jesse Glick added a comment - @chrisgwarp: Can not some startup logic be used to test for the presence of readlink and record it's presence and then not attempt to use it if not present? Probably; file a pull request. Only someone with AIX can test.

          Chris Graham added a comment -

          I don't get the pull request bit (not a git person).

          However, I have more than enough AIX hosts to test on; and more than willing too!

          Chris Graham added a comment - I don't get the pull request bit (not a git person). However, I have more than enough AIX hosts to test on; and more than willing too!

          Chris Graham added a comment - - edited

          AIX does have a readlink program, it's a part of the coreutils RPM package. It's located in /usr/freeware/bin, but by default, it does not get added to the default paths, so you'll need to do that yourself.

          Here are some hard timing facts:

          Jenkins 1.510:
          <<without readlink>>
          [10:36:46.966] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 98563ms]
          [10:39:50.486] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 96515ms]

          <<with readlink>>
          [10:58:11.697] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 43375ms]
          [10:59:11.871] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 37594ms]
          [11:00:20.580] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 40219ms]

          jenkins 1.500
          <<with readlink>>
          [11:30:52.488] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 7953ms]
          [11:32:05.379] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 4609ms]

          For 45ish jobs, 40 secs to generate the front page is not good. And as can be seen above, the presence (or lack) of presence of readlink is a major contributor to the performance problems.

          Has anyone opened a perf issue on the recent builds?

          Chris Graham added a comment - - edited AIX does have a readlink program, it's a part of the coreutils RPM package. It's located in /usr/freeware/bin, but by default, it does not get added to the default paths, so you'll need to do that yourself. Here are some hard timing facts: Jenkins 1.510: <<without readlink>> [10:36:46.966] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 98563ms] [10:39:50.486] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 96515ms] <<with readlink>> [10:58:11.697] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 43375ms] [10:59:11.871] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 37594ms] [11:00:20.580] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 40219ms] jenkins 1.500 <<with readlink>> [11:30:52.488] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 7953ms] [11:32:05.379] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 4609ms] For 45ish jobs, 40 secs to generate the front page is not good. And as can be seen above, the presence (or lack) of presence of readlink is a major contributor to the performance problems. Has anyone opened a perf issue on the recent builds?

          Jesse Glick added a comment -

          @chrisgwarp et al.: commenting on timing is fine but it does not help Jenkins developers make any progress since, again, we cannot test for you on AIX. You need to evaluate whether pull #770 makes a difference, e.g. by running https://jenkins.ci.cloudbees.com/job/core/job/jenkins_main_trunk/857/artifact/war/target/jenkins.war built from that branch.

          Jesse Glick added a comment - @chrisgwarp et al.: commenting on timing is fine but it does not help Jenkins developers make any progress since, again, we cannot test for you on AIX. You need to evaluate whether pull #770 makes a difference, e.g. by running https://jenkins.ci.cloudbees.com/job/core/job/jenkins_main_trunk/857/artifact/war/target/jenkins.war built from that branch.

          Hannes Kogler added a comment -

          same problem here:
          1.531

          Hannes Kogler added a comment - same problem here: 1.531

          What is the status of this problem. Getting a lot of readlink complains in Jenkins 1.565.3 running as master on HP-UX 11.31.

          Pawel Josefsson added a comment - What is the status of this problem. Getting a lot of readlink complains in Jenkins 1.565.3 running as master on HP-UX 11.31.

          Jesse Glick added a comment -

          pjos best to recheck in 1.586 with the fix of JENKINS-24521. Or if you can run Java 7, do so.

          Jesse Glick added a comment - pjos best to recheck in 1.586 with the fix of JENKINS-24521 . Or if you can run Java 7, do so.

          Daniel Beck added a comment -

          Is this still an issue in Jenkins 1.600+ / 1.596.x?

          Daniel Beck added a comment - Is this still an issue in Jenkins 1.600+ / 1.596.x?

          Hi ,

          I am also observing the "copyRecursiveTo" is not completely copying artifacts/files to master from slave.

          What i noticed that it had stopped copying one of the json file half. Because of same the next step which parse the json on master is failing.

          Same is working fine on master. Let me know any other information is required.

          I am writing this very first time on jenkins so let me know if i missed/not following any standard process here.

          There is relevant issue but this issue marked as duplicate issue hence writing the comment here.

          Thanks n Regards,
          Akhilkumar Patel

          Akhilkumar Patel added a comment - Hi , I am also observing the "copyRecursiveTo" is not completely copying artifacts/files to master from slave. What i noticed that it had stopped copying one of the json file half. Because of same the next step which parse the json on master is failing. Same is working fine on master. Let me know any other information is required. I am writing this very first time on jenkins so let me know if i missed/not following any standard process here. There is relevant issue but this issue marked as duplicate issue hence writing the comment here. Thanks n Regards, Akhilkumar Patel

            Unassigned Unassigned
            dreiss David Reiss
            Votes:
            23 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated: