Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26259

hudson.util.IOException2: Could not copy slave.jar into '/home/jenkins-build' on slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Minor Minor
    • core
    • None

      before upgrading to 1.50.1, this jenkins master was able to connect to slave.
      Slave configuration is vSphere, using vSphere Cloud Plugin, version 1.1.12.
      Master running 1.580.1, with ssh slaves plugin, version 1.9, fails to copy slave.jar to the slave.

      A second master running jenkins.war 1.580.1, vSphere Cloud Plugin, version 1.1.12, SSH Slave plugin, version 1.8, also fails to copy slave.jar to the slave.

      Note, this slave was successfully able to copy slave.jar to this slave, previously, most likely when master was running the prior LTS jenkins.war.

      slave permission of /dev/null are as

      ls -l /dev/null
      crw-rw-rw- 1 root root 1, 3 Dec 31 16:15 /dev/null

      The version of Java on both masters & slave is oracle 1.6, with slightly different minor version numbers.

      just before slave CI083 gets launched ...
      executing pre-launch scripts ...
      [CI083] Starting Virtual Machine...
      [CI083] Powering on VM
      [CI083] Waiting for VMTools
      [CI083] VM Tools are running
      [CI083] Finished wait for VMTools
      [12/31/14 16:17:06] [SSH] Opening SSH connection to 10.244.19.133:22.
      [12/31/14 16:17:06] [SSH] Authentication successful.
      [12/31/14 16:17:07] [SSH] The remote users environment is:
      ANT_HOME=/re/thirdparty_tools/ant/tags/Production
      BASH=/bin/bash
      BASH_ARGC=()
      BASH_ARGV=()
      BASH_EXECUTION_STRING=set
      BASH_LINENO=()
      BASH_SOURCE=()
      BASH_VERSINFO=([0]="3" [1]="2" [2]="51" [3]="1" [4]="release" [5]="x86_64-suse-linux-gnu")
      BASH_VERSION='3.2.51(1)-release'
      CCACHE_DIR=/var/cache/ccache
      CCACHE_UMASK=0
      CFG_BUILDROOT=/build
      COLORTERM=1
      CONSOLE_MAGIC='(K'
      CPU=x86_64
      CSHEDIT=emacs
      CVS_RSH=ssh
      DIRSTACK=()
      ENV=/etc/bash.bashrc
      EUID=1002
      FROM_HEADER=
      GROUPS=()
      G_BROKEN_FILENAMES=1
      G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252
      HISTSIZE=1000
      HOME=/home/c4dev
      HOST=CI-CPBE-slave-83
      HOSTNAME=CI-CPBE-slave-83
      HOSTTYPE=x86_64
      IFS=$' \t\n'
      INFODIR=/usr/local/info:/usr/share/info:/usr/info
      INFOPATH=/usr/local/info:/usr/share/info:/usr/info
      INPUTRC=/etc/inputrc
      INSTALL_DESKTOP_EXTENSIONS=yes
      JAVA_BINDIR=/usr/lib64/jvm/jre/bin
      JAVA_HOME=/re/thirdparty_tools/java/tags/linux/Production
      JAVA_ROOT=/usr/lib64/jvm/jre
      JRE_HOME=/usr/lib64/jvm/jre
      KDE_USE_IPV6=yes
      LANG=en_US.UTF-8
      LD_LIBRARY_PATH=/c4shares/auto/tools/lib/mysql::/opt/pegasus/lib
      LESS='-M -I'
      LESSCLOSE='lessclose.sh %s %s'
      LESSKEY=/etc/lesskey.bin
      LESSOPEN='lessopen.sh %s'
      LESS_ADVANCED_PREPROCESSOR=no
      LOGNAME=c4dev
      LS_COLORS=
      LS_OPTIONS='-N --color=none -T 0'
      MACHTYPE=x86_64-suse-linux
      MAIL=/var/mail/c4dev
      MANPATH=/usr/local/man:/usr/share/man
      MINICOM='-c on'
      MODSETUP_ARTIFACTS_DIR=/c4shares/Public/Artifacts
      MORE=-sl
      NFSDOWN=
      NNTPSERVER=news
      NXDIR=/usr/NX
      OPTERR=1
      OPTIND=1
      OSTYPE=linux
      PAGER=less
      PATH=/var/lib/ccache/bin:/c4shares/auto/tools/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/accurev/bin:/opt/CmTools/shbin:/usr/lib64/jvm/jre/bin:/re/bin:/re/thirdparty_tools/java/tags/linux/Production/bin:/opt/pegasus/bin:/re/thirdparty_tools/ant/tags/Production/bin:/c4shares/auto/devutils/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/NX/bin
      PEGASUS_HOME=/opt/pegasus
      PERL5LIB=/c4shares/auto/tools/lib/perl:/usr/lib/perl5/vendor_perl/5.10.0/:/usr/csx/c4global/c4perl/lib
      PIPESTATUS=([0]="0")
      PPID=4837
      PROFILEREAD=true
      PS4='+ '
      PWD=/home/c4dev
      PYTHONPATH=/c4shares/auto/tools/lib/python:
      PYTHONSTARTUP=/etc/pythonstart
      QT_SYSTEM_DIR=/usr/share/desktop-data
      SHELL=/bin/bash
      SHELLOPTS=braceexpand:hashall:interactive-comments
      SHLVL=1
      SSH_CLIENT='10.244.177.169 58607 22'
      SSH_CONNECTION='10.244.177.169 58607 10.244.19.133 22'
      TERM=dumb
      UID=1002
      USER=c4dev
      WINDOWMANAGER=/usr/bin/gnome
      XCURSOR_THEME=DMZ
      XDG_CONFIG_DIRS=/etc/xdg
      XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share
      XKEYSYMDB=/usr/share/X11/XKeysymDB
      XNLSPATH=/usr/share/X11/nls
      X_MOUSE_CURSOR=DMZ
      _=true
      is=bash
      japanese_legacy_encoding=EUC-JP
      tty=
      __gvfs_multiple_uris ()
      {
      local IFS='
      ';
      local cur="${COMP_WORDS[COMP_CWORD]}";
      COMPREPLY=($(compgen -W '$(gvfs-ls --show-completions "$cur")' – ""));
      case "$cur" in
      :)
      case "$COMP_WORDBREAKS" in
      :)
      colon_prefix=$(echo $cur | sed 's/:[^:]*$/:/' );
      COMPREPLY=${COMPREPLY##${colon_prefix}}
      ;;
      esac
      ;;
      esac
      }
      __polkit_action ()
      {
      local IFS='
      ';
      local cur="${COMP_WORDS[COMP_CWORD]}";
      case $COMP_CWORD in
      1)
      COMPREPLY=($(IFS=: compgen S' ' -W "action:reset-defaults:set-defaults-any:set-defaults-inactive:set-defaults-active:show-overrides:version:-help" – $cur))
      ;;
      2)
      case "${COMP_WORDS[1]}" in
      --action | --set-defaults-any | --set-defaults-inactive | --set-defaults-active)
      COMPREPLY=($(compgen -W "$(polkit-action)" – $cur))
      ;;
      --reset-defaults)
      COMPREPLY=($(compgen -W "$(polkit-action --show-overrides)" – $cur))
      ;;
      esac
      ;;
      3)
      case "${COMP_WORDS[1]}" in
      --set-defaults-any | --set-defaults-inactive | --set-defaults-active)
      COMPREPLY=($(IFS=: compgen -S' ' -W "yes:no:auth_admin_one_shot:auth_admin:auth_admin_keep_session:auth_admin_keep_always:auth_self_one_shot:auth_self:auth_self_keep_session:auth_self_keep_always" – $cur))
      ;;
      esac
      ;;
      esac
      }
      __polkit_auth ()
      {
      local IFS='
      ';
      local cur="${COMP_WORDS[COMP_CWORD]}";
      case $COMP_CWORD in
      1)
      COMPREPLY=($(IFS=: compgen S' ' -W "obtain:show-obtainable:explicit:explicit-detail:grant:block:revoke:user:version:-help" – $cur))
      ;;
      2)
      case "${COMP_WORDS[1]}" in
      --obtain)
      COMPREPLY=($(compgen -W "$(polkit-auth --show-obtainable)" – $cur))
      ;;
      --revoke)
      COMPREPLY=($(compgen -W "$(polkit-auth --explicit)" – $cur))
      ;;
      --grant | --block)
      COMPREPLY=($(compgen -W "$(polkit-action)" – $cur))
      ;;
      --user)
      COMPREPLY=($(compgen -u – $cur))
      ;;
      esac
      ;;
      3)
      case "${COMP_WORDS[1]}" in
      --user)
      COMPREPLY=($(IFS=: compgen S' ' -W "explicit:explicit-detail:grant:block:-revoke" – $cur))
      ;;
      --grant | --block)
      COMPREPLY=($(IFS=: compgen S' ' -W "-constraint" – $cur))
      ;;
      esac
      ;;
      4)
      case "${COMP_WORDS[3]}" in
      --revoke)
      case "${COMP_WORDS[1]}" in
      --user)
      local afou;
      afou=$(polkit-auth --user ${COMP_WORDS[2]} --explicit 2> /dev/null);
      if [ $? != 0 ]; then
      afou=$(polkit-action);
      fi;
      COMPREPLY=($(compgen -W "$afou" – $cur))
      ;;
      *)
      COMPREPLY=($(compgen -W "$(polkit-action)" – $cur))
      ;;
      esac
      ;;
      --grant | --block)
      COMPREPLY=($(compgen -W "$(polkit-action)" – $cur))
      ;;
      --constraint)
      COMPREPLY=($(IFS=: compgen -S' ' -W "local:active:exe\::selinux_context\:" – $cur))
      ;;
      esac
      ;;
      5)
      case "${COMP_WORDS[3]}" in
      --grant | --block)
      COMPREPLY=($(IFS=: compgen S' ' -W "-constraint" – $cur))
      ;;
      esac;
      case "${COMP_WORDS[1]}" in
      --grant | --block)
      COMPREPLY=($(IFS=: compgen S' ' -W "-constraint" – $cur))
      ;;
      esac
      ;;
      *)
      case "${COMP_WORDS[$(($COMP_CWORD - 1))]}" in
      --constraint)
      COMPREPLY=($(IFS=: compgen -S' ' -W "local:active:exe\::selinux_context\:" – $cur))
      ;;
      *)
      COMPREPLY=($(IFS=: compgen S' ' -W "-constraint" – $cur))
      ;;
      esac
      ;;
      esac
      }
      path ()
      {
      if test -x /usr/bin/$1; then
      ${1+"/usr/bin/$@"};
      else
      if test -x /bin/$1; then
      ${1+"/bin/$@"};
      fi;
      fi
      }
      test_lc ()
      {
      for lc in LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES LC_PAPER LC_NAMELC_ADDRESS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION LC_ALL;
      do
      eval val="\$$lc";
      if test -n "$val"; then
      unset lc val;
      return 0;
      fi;
      done;
      unset lc val;
      return 1
      }
      uniquefy_search_path ()
      {
      OIFS="$IFS";
      IFS='
      ';
      set – $(echo ${1+"$@"} | sed -r 's@/*:|([^\\]):@\1\n@g;H;x;s@/\n@\n@');
      IFS="$OIFS";
      _y="";
      for _x in "$@";
      do
      case ":${_y}:" in
      :"${_x}":)
      continue
      ;;
      esac;
      _y=${_y:+"$_y:"}${_x};
      done;
      echo "${_y}";
      unset _y _x
      }
      [12/31/14 16:17:07] [SSH] Checking java version of java
      [12/31/14 16:17:07] [SSH] java -version returned 1.6.0.
      [12/31/14 16:17:07] [SSH] Starting sftp client.
      [12/31/14 16:17:07] [SSH] Copying latest slave.jar...
      hudson.util.IOException2: Could not copy slave.jar into '/home/jenkins-build' on slave
      at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1076)
      at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:134)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:704)
      at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:691)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: hudson.util.IOException2: Could not copy slave.jar to '/home/jenkins-build/slave.jar' on slave
      at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1073)
      ... 7 more
      Caused by: com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.)
      at com.trilead.ssh2.SFTPv3Client.write(SFTPv3Client.java:1360)
      at hudson.plugins.sshslaves.SFTPClient$1.write(SFTPClient.java:102)
      at java.io.OutputStream.write(Unknown Source)
      at hudson.plugins.sshslaves.SSHLauncher.copySlaveJar(SSHLauncher.java:1067)
      ... 7 more
      [12/31/14 16:17:07] Launch failed - cleaning up connection
      [12/31/14 16:17:07] [SSH] Connection closed.

          [JENKINS-26259] hudson.util.IOException2: Could not copy slave.jar into '/home/jenkins-build' on slave

          Daniel Beck added a comment -

          Does /home/jenkins-build exist, or can it be created (permissions) by the user Jenkins authenticates as? Otherwise, does the user Jenkins authenticates as have write permissions to /home/jenkins-build?

          Daniel Beck added a comment - Does /home/jenkins-build exist, or can it be created (permissions) by the user Jenkins authenticates as? Otherwise, does the user Jenkins authenticates as have write permissions to /home/jenkins-build?

          Martin Buckley added a comment -

          On the slave /home/jenkins-build exists & contains workspace from prior job builds.

          bucklm2> ls -ld /home/jenkins-build/
          drwxr-xr-x 5 c4dev users 4096 Dec 31 16:53 /home/jenkins-build/
          bucklm2> ls !$
          ls /home/jenkins-build/
          slave.jar upc-gany-deveco2-usr upc-PFDC-cs workspace

          In addition, I have manually deleted /nome/jenkins-build/slave.jar. When the master starts the slave.jar file is being created.,

          Martin Buckley added a comment - On the slave /home/jenkins-build exists & contains workspace from prior job builds. bucklm2> ls -ld /home/jenkins-build/ drwxr-xr-x 5 c4dev users 4096 Dec 31 16:53 /home/jenkins-build/ bucklm2> ls !$ ls /home/jenkins-build/ slave.jar upc-gany-deveco2-usr upc-PFDC-cs workspace In addition, I have manually deleted /nome/jenkins-build/slave.jar. When the master starts the slave.jar file is being created.,

          Martin Buckley added a comment -

          Lack of disk space is causing the slave.jar copy to fail.

          Martin Buckley added a comment - Lack of disk space is causing the slave.jar copy to fail.

          adoudev viaccess-orca added a comment -

          read-only file system error can also causing this copy failure error.

          adoudev viaccess-orca added a comment - read-only file system error can also causing this copy failure error.

          Pravin Mundra added a comment -

          I faced the similar issue and lack of space was the actual problem. I had to delete the old workspaces created under /apps/jenkins/workspace and then it worked.

          Pravin Mundra added a comment - I faced the similar issue and lack of space was the actual problem. I had to delete the old workspaces created under /apps/jenkins/workspace and then it worked.

            martin_buckley Martin Buckley
            martin_buckley Martin Buckley
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: