Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62208

sshGet (scp mode) is showing the entire contents of the file in console output log

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ssh-steps-plugin
    • Jenkins ver. 2.190.3 on Windows 2012 with ssh versions below.
      ssh 2.6.1
      ssh-agent 1.17
      ssh-credentials 1.18
      ssh-steps 2.0.0

      I sent this to the mailing list and was advised that it might be a bug and to open a Jira ticket:

      When executing the following in a scripted pipeline:
       
      sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}", into: FileName, override: true
       
      The file is downloaded successfully from the remote server into the local directory on the Jenkins server.  However, the console output shows the contents of that file in its entirety, and the logs files in the build folder also contains the contents of the file.  The file is embedded within the rest of the log entries, and is not usable.  Therefore when I am downloading large files with sshGet, I end up consuming twice the space I should be, since I have a log file roughly the size of the file, and then the file itself which was downloaded.  
       
      This is not only a space issue, but also a security issue, since sometimes I am downloading files with sensitive information in them and don't want those details available in the jenkins logs.

        1. pipeline.txt
          1 kB
        2. scp-success.txt
          0.4 kB
        3. sftp-failed.txt
          0.7 kB
        4. consoletext.txt
          1 kB

          [JENKINS-62208] sshGet (scp mode) is showing the entire contents of the file in console output log

          Craig Webster created issue -

          A bit tricky one to reproduce, we are actively using this plugin and don't see this on our Jenkins servers, and probably could be due to Jenkins and plugin combination.

          Can you provide more information? like what is FileName is? Nothing pops up to me, can you make sure into: FileName is string vs a variable ? like into: "abc.txt"

          Naresh Rayapati added a comment - A bit tricky one to reproduce, we are actively using this plugin and don't see this on our Jenkins servers, and probably could be due to Jenkins and plugin combination. Can you provide more information? like what is FileName is? Nothing pops up to me, can you make sure into: FileName is string vs a variable ? like into: "abc.txt"

          Craig Webster added a comment -

          Not sure if this will format correctly, but hopefully gives you enough of an idea:

          import java.text.SimpleDateFormat

          node {
          def dateFormat = new SimpleDateFormat("yyyyMMddHHmm")
          sourcefile="server.log"
          String partialFileName = "
          " + sourcefile + "${MAL}" + dateFormat.format(new Date()) + "soc"

          servers = ['10.xxx.xxx.xxx',]

          def soc_remote = [:]
          soc_remote.name = "${MAL} SOC"
          soc_remote.allowAnyHosts = true
          soc_remote.fileTransfer = 'scp'

          withCredentials([usernamePassword(credentialsId: 'Credentials', usernameVariable: 'USER_ID', passwordVariable: 'USER_PASSWORD')]) {
          soc_remote.user = USER_ID
          soc_remote.password = USER_PASSWORD

          servers.each { server ->
          String FileName = partialFileName + server + ".txt"
          echo "${FileName}"
          soc_remote.host = server

          stage("Run log gather script on ${server}") {
          sshCommand remote: soc_remote, command: "ls -ltr /srv/jboss/server/${MAL}/log/${sourcefile}"
          }
          stage("Download ${MAL} logs from ${server}") {
          dir("D:\\Logs\\${MAL}\\logs
          ${server}") {
          try {
          def commandResult = sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}", into: FileName, override: true
          } catch (err) {
          echo "something failed"
          echo "${err}"
          }
          }
          }
          }
          }
          }

          Craig Webster added a comment - Not sure if this will format correctly, but hopefully gives you enough of an idea: import java.text.SimpleDateFormat node { def dateFormat = new SimpleDateFormat("yyyyMMddHHmm") sourcefile="server.log" String partialFileName = " " + sourcefile + " ${MAL} " + dateFormat.format(new Date()) + " soc " servers = ['10.xxx.xxx.xxx',] def soc_remote = [:] soc_remote.name = "${MAL} SOC" soc_remote.allowAnyHosts = true soc_remote.fileTransfer = 'scp' withCredentials( [usernamePassword(credentialsId: 'Credentials', usernameVariable: 'USER_ID', passwordVariable: 'USER_PASSWORD')] ) { soc_remote.user = USER_ID soc_remote.password = USER_PASSWORD servers.each { server -> String FileName = partialFileName + server + ".txt" echo "${FileName}" soc_remote.host = server stage("Run log gather script on ${server}") { sshCommand remote: soc_remote, command: "ls -ltr /srv/jboss/server/${MAL}/log/${sourcefile}" } stage("Download ${MAL} logs from ${server}") { dir("D:\\Logs\\${MAL}\\logs ${server}") { try { def commandResult = sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}", into: FileName, override: true } catch (err) { echo "something failed" echo "${err}" } } } } } }

          you could use the code block in JIRA for formatting. Like

          import java.text.SimpleDateFormat
          
          node {
              def dateFormat = new SimpleDateFormat("yyyyMMddHHmm")
              sourcefile="server.log"
              String partialFileName = "" + sourcefile + "${MAL}" + dateFormat.format(new Date()) + "soc"
              servers = ['10.xxx.xxx.xxx',]
              def soc_remote = [:]
              soc_remote.name = "${MAL} SOC"
              soc_remote.allowAnyHosts = true
              soc_remote.fileTransfer = 'scp'
          
              withCredentials([usernamePassword(credentialsId: 'Credentials', usernameVariable: 'USER_ID', passwordVariable: 'USER_PASSWORD')]) {
                  soc _remote.user = USER_ID
                  soc_remote.password = USER_PASSWORD
                  servers.each { server ->
                      String FileName = partialFileName + server + ".txt"
                      echo "${FileName}"
                      soc_remote.host = server
                      stage("Run log gather script on ${server}") {
                          sshCommand remote: soc_remote, command: "ls -ltr /srv/jboss/server/${MAL}/log/${sourcefile}"
                      }
                      stage("Download ${MAL} logs from ${server}") {
                          dir("D:\\Logs\\${MAL}\\logs${server}") {
                              try {
                                  def commandResult = sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}", into: FileName, override: true
                              } catch (err) {
                                  echo "something failed"
                                  echo "${err}"
                              }
                          }
                      }
                  }
              }
          }
          

          Naresh Rayapati added a comment - you could use the code block in JIRA for formatting. Like import java.text.SimpleDateFormat node { def dateFormat = new SimpleDateFormat( "yyyyMMddHHmm" ) sourcefile= "server.log" String partialFileName = "" + sourcefile + " ${MAL} " + dateFormat.format( new Date()) + " soc" servers = [ '10.xxx.xxx.xxx' ,] def soc_remote = [:] soc_remote.name = "${MAL} SOC" soc_remote.allowAnyHosts = true soc_remote.fileTransfer = 'scp' withCredentials([usernamePassword(credentialsId: 'Credentials' , usernameVariable: 'USER_ID' , passwordVariable: 'USER_PASSWORD' )]) { soc _remote.user = USER_ID soc_remote.password = USER_PASSWORD servers.each { server -> String FileName = partialFileName + server + ".txt" echo "${FileName}" soc_remote.host = server stage( "Run log gather script on ${server}" ) { sshCommand remote: soc_remote, command: "ls -ltr /srv/jboss/server/${MAL}/log/${sourcefile}" } stage( "Download ${MAL} logs from ${server}" ) { dir( "D:\\Logs\\${MAL}\\logs${server}" ) { try { def commandResult = sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}" , into: FileName, override: true } catch (err) { echo "something failed" echo "${err}" } } } } } }

          Okay couple of things: 

          • is the actual server that this step is trying to get the file from linux machine? 
          • Not sure def CommandResult is causing this issue, can we delete that assignment and see if that is the cause.
          • Looks like you are running Jenkins on Windows, unfortunately I didn't test this plugin on windows server, can you able test it on linux agent to see if that is the problem 

          Naresh Rayapati added a comment - Okay couple of things:  is the actual server that this step is trying to get the file from linux machine?  Not sure def CommandResult is causing this issue, can we delete that assignment and see if that is the cause. Looks like you are running Jenkins on Windows, unfortunately I didn't test this plugin on windows server, can you able test it on linux agent to see if that is the problem 

          Craig Webster added a comment -

          Yes, the remote server that it is getting the file from is a linux server, the source Jenkins server is Windows 2012
          I actually added the def CommandResult as part of troubleshooting but i took it out again and get the same results
          I am using a work provided Jenkins instance; not sure i have access to a linux box to install Jenkins on.

          Craig Webster added a comment - Yes, the remote server that it is getting the file from is a linux server, the source Jenkins server is Windows 2012 I actually added the def CommandResult as part of troubleshooting but i took it out again and get the same results I am using a work provided Jenkins instance; not sure i have access to a linux box to install Jenkins on.

          I don't have windows machine to test this right now, probably it might take more time for to me troubleshoot on this issue.

          Naresh Rayapati added a comment - I don't have windows machine to test this right now, probably it might take more time for to me troubleshoot on this issue.
          Craig Webster made changes -
          Attachment New: consoletext.txt [ 51156 ]

          Craig Webster added a comment -

          Ok, I spun up a micro instance on Google Cloud with Ubuntu and installed a fresh copy of Jenkins. I then took my pipeline and had it scp a file from that Jenkins server to that Jenkins server (didn't have anything else I could connect to). I attached the console log - it is still putting the contents of the file into the log itself. Code I used below, log is attached.

          node {
           [^consoletext.txt]     sourcefile="test.txt"
          
          servers = ['localhost',]
          
          	def soc_remote = [:]
          	soc_remote.name = "SOC"
          	soc_remote.allowAnyHosts = true
          	soc_remote.fileTransfer = 'scp'
             
          	withCredentials([usernamePassword(credentialsId: 'Credentials', usernameVariable: 'USER_ID', passwordVariable: 'USER_PASSWORD')]) {
          		soc_remote.user = USER_ID
          		soc_remote.password = USER_PASSWORD
          			
          		servers.each { server ->
                      String FileName = "test.txt"
                      echo "${FileName}"
                      soc_remote.host = server
          				
          				stage("Run log gather script on ${server}") {
          					sshCommand remote: soc_remote, command: "ls -ltr /tmp/test/${sourcefile}"
          				}
          				stage("Download logs from ${server}") {
          					dir("/tmp/") {
                                  try {
                                  sshGet remote: soc_remote, from: "/tmp/test/${sourcefile}", into: FileName, override: true
                                  } catch (err) {
                                    echo "something failed"
                                    echo "${err}"
                                    }
          					}
          				}
                  }
          	}
          }
          

          Craig Webster added a comment - Ok, I spun up a micro instance on Google Cloud with Ubuntu and installed a fresh copy of Jenkins. I then took my pipeline and had it scp a file from that Jenkins server to that Jenkins server (didn't have anything else I could connect to). I attached the console log - it is still putting the contents of the file into the log itself. Code I used below, log is attached. node { [^consoletext.txt] sourcefile= "test.txt" servers = [ 'localhost' ,] def soc_remote = [:] soc_remote.name = "SOC" soc_remote.allowAnyHosts = true soc_remote.fileTransfer = 'scp' withCredentials([usernamePassword(credentialsId: 'Credentials' , usernameVariable: 'USER_ID' , passwordVariable: 'USER_PASSWORD' )]) { soc_remote.user = USER_ID soc_remote.password = USER_PASSWORD servers.each { server -> String FileName = "test.txt" echo "${FileName}" soc_remote.host = server stage( "Run log gather script on ${server}" ) { sshCommand remote: soc_remote, command: "ls -ltr /tmp/test/${sourcefile}" } stage( "Download logs from ${server}" ) { dir( "/tmp/" ) { try { sshGet remote: soc_remote, from: "/tmp/test/${sourcefile}" , into: FileName, override: true } catch (err) { echo "something failed" echo "${err}" } } } } } }

          Apparently I didn't read your code fully before sorry about that.
          It is probably you are using the 'scp' as a fileTransfer mode, try to remove that line so that it is going to fall back to sftp or set it to 'sftp'

          ...
          soc_remote.fileTransfer = 'scp'
          ...
          

          Naresh Rayapati added a comment - Apparently I didn't read your code fully before sorry about that. It is probably you are using the 'scp' as a fileTransfer mode, try to remove that line so that it is going to fall back to sftp or set it to 'sftp' ... soc_remote.fileTransfer = 'scp' ...

            nrayapati Naresh Rayapati
            craigw Craig Webster
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: