Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62208

sshGet (scp mode) is showing the entire contents of the file in console output log

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ssh-steps-plugin
    • Jenkins ver. 2.190.3 on Windows 2012 with ssh versions below.
      ssh 2.6.1
      ssh-agent 1.17
      ssh-credentials 1.18
      ssh-steps 2.0.0

      I sent this to the mailing list and was advised that it might be a bug and to open a Jira ticket:

      When executing the following in a scripted pipeline:
       
      sshGet remote: soc_remote, from: "/srv/jboss/server/${MAL}/log/${sourcefile}", into: FileName, override: true
       
      The file is downloaded successfully from the remote server into the local directory on the Jenkins server.  However, the console output shows the contents of that file in its entirety, and the logs files in the build folder also contains the contents of the file.  The file is embedded within the rest of the log entries, and is not usable.  Therefore when I am downloading large files with sshGet, I end up consuming twice the space I should be, since I have a log file roughly the size of the file, and then the file itself which was downloaded.  
       
      This is not only a space issue, but also a security issue, since sometimes I am downloading files with sensitive information in them and don't want those details available in the jenkins logs.

        1. consoletext.txt
          1 kB
        2. pipeline.txt
          1 kB
        3. scp-success.txt
          0.4 kB
        4. sftp-failed.txt
          0.7 kB

          [JENKINS-62208] sshGet (scp mode) is showing the entire contents of the file in console output log

          Craig Webster added a comment -

          Yes, the remote server that it is getting the file from is a linux server, the source Jenkins server is Windows 2012
          I actually added the def CommandResult as part of troubleshooting but i took it out again and get the same results
          I am using a work provided Jenkins instance; not sure i have access to a linux box to install Jenkins on.

          Craig Webster added a comment - Yes, the remote server that it is getting the file from is a linux server, the source Jenkins server is Windows 2012 I actually added the def CommandResult as part of troubleshooting but i took it out again and get the same results I am using a work provided Jenkins instance; not sure i have access to a linux box to install Jenkins on.

          I don't have windows machine to test this right now, probably it might take more time for to me troubleshoot on this issue.

          Naresh Rayapati added a comment - I don't have windows machine to test this right now, probably it might take more time for to me troubleshoot on this issue.

          Craig Webster added a comment -

          Ok, I spun up a micro instance on Google Cloud with Ubuntu and installed a fresh copy of Jenkins. I then took my pipeline and had it scp a file from that Jenkins server to that Jenkins server (didn't have anything else I could connect to). I attached the console log - it is still putting the contents of the file into the log itself. Code I used below, log is attached.

          node {
           [^consoletext.txt]     sourcefile="test.txt"
          
          servers = ['localhost',]
          
          	def soc_remote = [:]
          	soc_remote.name = "SOC"
          	soc_remote.allowAnyHosts = true
          	soc_remote.fileTransfer = 'scp'
             
          	withCredentials([usernamePassword(credentialsId: 'Credentials', usernameVariable: 'USER_ID', passwordVariable: 'USER_PASSWORD')]) {
          		soc_remote.user = USER_ID
          		soc_remote.password = USER_PASSWORD
          			
          		servers.each { server ->
                      String FileName = "test.txt"
                      echo "${FileName}"
                      soc_remote.host = server
          				
          				stage("Run log gather script on ${server}") {
          					sshCommand remote: soc_remote, command: "ls -ltr /tmp/test/${sourcefile}"
          				}
          				stage("Download logs from ${server}") {
          					dir("/tmp/") {
                                  try {
                                  sshGet remote: soc_remote, from: "/tmp/test/${sourcefile}", into: FileName, override: true
                                  } catch (err) {
                                    echo "something failed"
                                    echo "${err}"
                                    }
          					}
          				}
                  }
          	}
          }
          

          Craig Webster added a comment - Ok, I spun up a micro instance on Google Cloud with Ubuntu and installed a fresh copy of Jenkins. I then took my pipeline and had it scp a file from that Jenkins server to that Jenkins server (didn't have anything else I could connect to). I attached the console log - it is still putting the contents of the file into the log itself. Code I used below, log is attached. node { [^consoletext.txt] sourcefile= "test.txt" servers = [ 'localhost' ,] def soc_remote = [:] soc_remote.name = "SOC" soc_remote.allowAnyHosts = true soc_remote.fileTransfer = 'scp' withCredentials([usernamePassword(credentialsId: 'Credentials' , usernameVariable: 'USER_ID' , passwordVariable: 'USER_PASSWORD' )]) { soc_remote.user = USER_ID soc_remote.password = USER_PASSWORD servers.each { server -> String FileName = "test.txt" echo "${FileName}" soc_remote.host = server stage( "Run log gather script on ${server}" ) { sshCommand remote: soc_remote, command: "ls -ltr /tmp/test/${sourcefile}" } stage( "Download logs from ${server}" ) { dir( "/tmp/" ) { try { sshGet remote: soc_remote, from: "/tmp/test/${sourcefile}" , into: FileName, override: true } catch (err) { echo "something failed" echo "${err}" } } } } } }

          Apparently I didn't read your code fully before sorry about that.
          It is probably you are using the 'scp' as a fileTransfer mode, try to remove that line so that it is going to fall back to sftp or set it to 'sftp'

          ...
          soc_remote.fileTransfer = 'scp'
          ...
          

          Naresh Rayapati added a comment - Apparently I didn't read your code fully before sorry about that. It is probably you are using the 'scp' as a fileTransfer mode, try to remove that line so that it is going to fall back to sftp or set it to 'sftp' ... soc_remote.fileTransfer = 'scp' ...

          Craig Webster added a comment -

          Thank you, i removed that line and it fixed my issue. scp vs sftp still confuses me; when i want to copy a file between linux servers the command I use is scp.... so probably that's why I had that in there.

          Curious though, why does the plugin echo the file contents to console output when using scp as transfer mode? Honestly in some ways it is handy because there are times when i actually do want the file echoed to the console.

          Craig Webster added a comment - Thank you, i removed that line and it fixed my issue. scp vs sftp still confuses me; when i want to copy a file between linux servers the command I use is scp.... so probably that's why I had that in there. Curious though, why does the plugin echo the file contents to console output when using scp as transfer mode? Honestly in some ways it is handy because there are times when i actually do want the file echoed to the console.

          I found a bug in the code, going to fix it soon to not to print the file in scp mode.

          But in cases like where you wanted to print the file contents explicitly, there are couple of other steps around reading file contents and printing those to console.
          https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#readfile-read-file-from-workspace

          def data = readFile(file: FileName)
          println(data)
          

          Few advanced steps to read specific files like yaml,csv or json here: https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/

          Naresh Rayapati added a comment - I found a bug in the code, going to fix it soon to not to print the file in scp mode. But in cases like where you wanted to print the file contents explicitly, there are couple of other steps around reading file contents and printing those to console. https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#readfile-read-file-from-workspace def data = readFile(file: FileName) println(data) Few advanced steps to read specific files like yaml,csv or json here: https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/

          Craig Webster added a comment -

          I have another similar (related?) issue. I have another server where removing the scp line results in me getting SSH_FX_NO_SUCH_FILE. If I leave everything else the same and add the scp line again, the file is downloaded (and output to the console as the original bug states). The file is there as evidenced by it working when using scp. Should I open a separate bug report for this? See attached files for pipeline steps, and resulting output for success and failure. pipeline.txt scp-success.txt sftp-failed.txt

          Craig Webster added a comment - I have another similar (related?) issue. I have another server where removing the scp line results in me getting SSH_FX_NO_SUCH_FILE. If I leave everything else the same and add the scp line again, the file is downloaded (and output to the console as the original bug states). The file is there as evidenced by it working when using scp. Should I open a separate bug report for this? See attached files for pipeline steps, and resulting output for success and failure. pipeline.txt scp-success.txt sftp-failed.txt

          craigw I think we are good with this JIRA, as I am going to fix that bug when we enable scp not to print the file contents.

          Naresh Rayapati added a comment - craigw I think we are good with this JIRA, as I am going to fix that bug when we enable scp not to print the file contents.

          Craig Webster added a comment -

          ok, you are referring to the secondary bug i was reporting with the SSH_FX_NO_SUCH_FILE error? You will fix both bugs under this JIRA ticket so no need to open another ticket?

          Craig Webster added a comment - ok, you are referring to the secondary bug i was reporting with the SSH_FX_NO_SUCH_FILE error? You will fix both bugs under this JIRA ticket so no need to open another ticket?

          I have the same issue. When I use SCP mode the files copied are printed to Jenkins console.

          This is a big issue especially if you want to copy secrets across.

          Can this be fixed please?

          Piotr Niebylski added a comment - I have the same issue. When I use SCP mode the files copied are printed to Jenkins console. This is a big issue especially if you want to copy secrets across. Can this be fixed please?

            nrayapati Naresh Rayapati
            craigw Craig Webster
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: