-
Bug
-
Resolution: Unresolved
-
Major
I am trying to copy contents inside of a folder(whcih is Jenkins workspace) to another reomte location using sshPut. put always copying full folder instead of contents.
In below example it always trying to copy test folder to destination location.
if the same command executes for the second time it giving below error.
Failed SFTP MKDIR: <host address>:$SERVER_PATH/test
example code:
sshPut remote: remote, from: "$BUILD_NUMBER/test/",into: "$SERVER_PATH"
Installed SSH Pipeline Steps version is 2.0.0
[JENKINS-58778] [ sshPut ] unable to copy only contents inside folder. put always copying full folder
Okay override option is only available for sshGet but for the put you would need to delete the folder before we create it again. or Use sshRemove step
Hokwang Lee
added a comment -
Hello.
Can you let me know more detail example for filterBy and filterRegex?
I want to scp some/folder/*.tar.gz to remote server's /output/ path.
In this case, how can I do that?
I can't find solution in docs and googling and so many test in my PC.
Many thanks,
Hokwang Lee
added a comment - nrayapati
Hello.
Can you let me know more detail example for filterBy and filterRegex?
I want to scp some/folder/*.tar.gz to remote server's /output/ path.
In this case, how can I do that?
I can't find solution in docs and googling and so many test in my PC.
Many thanks, 
Christian Ciach
added a comment - I second this request and I couldn't find a solution for this, too.
Christian Ciach
added a comment - I second this request and I couldn't find a solution for this, too. Maybe we could do something like that as a workaround:
findFiles(glob: 'Build/rpms/*.rpm').each { rpmFile -> sshPut remote: remote, from: rpmFile.path, into: '/data/mirror/rpm-repo' }
This feels a bit dirty this probably has negative performance implications, because this establishes a new SSH-connection for each file.
It would be great if this plugin could provide a "flatten" parameter for the sshPut-step, like the publish-over-ssh plugin: https://wiki.jenkins.io/display/JENKINS/Publish+Over#PublishOver-Flattenfiles
Or, alternatively, allow a wildcard for the "from"-parameter of the sshPut-step, so we could use "Build/rpms/*" instead of "Build/rpms" as the source.
The underlying library allows many kinds of "from" parameters (like Iterables), not just Strings. Maybe this jenkins plugin could be modified so that not only Strings are permitted as the "from" parameter? At least maybe allow Iterables, too?
Christian Ciach
added a comment - - edited Maybe we could do something like that as a workaround:
findFiles(glob: 'Build/rpms/*.rpm' ).each { rpmFile ->
sshPut remote: remote, from: rpmFile.path, into: '/data/mirror/rpm-repo'
}
This feels a bit dirty this probably has negative performance implications, because this establishes a new SSH-connection for each file.
It would be great if this plugin could provide a "flatten" parameter for the sshPut-step, like the publish-over-ssh plugin: https://wiki.jenkins.io/display/JENKINS/Publish+Over#PublishOver-Flattenfiles
Or, alternatively, allow a wildcard for the "from"-parameter of the sshPut-step, so we could use "Build/rpms/*" instead of "Build/rpms" as the source.
The underlying library allows many kinds of "from" parameters (like Iterables), not just Strings. Maybe this jenkins plugin could be modified so that not only Strings are permitted as the "from" parameter? At least maybe allow Iterables, too? 
Piotr S
added a comment - Im facing this problem now. This is a valid issue.
Exactly like described the workspace folder is forced and its creating very inconsistent behavior.
sshPut remote: remote, from: "${env.WORKSPACE}/"+"/", into: "/srv/backuproot"+"/"
above code would sometimes create:
/srv/backuproot/backupMake_dev2/<files here>
but if previus pipeline didnt finish (valid case from me) jenkins will create 'backupMake_dev2@tmp' and on target sometimes i end up with:
/srv/backuproot/backupMake_dev2@tmp/<files here>
Expected behavior:
/srv/backuproot/<files here>
Is it possible to do achieve the expected behavior in any way?
--------------------------------------------------------------------------------------
With filter its also not helping:
sshPut remote: remote, from: "${env.WORKSPACE}/"+"/", into: backuproot+"/", filterBy: "name", filterRegex : /\.json$/
above code creates workspace folder on target like so:
/srv/makeBackupAuto/backupMake_dev2/config.json
Expected:
/srv/makeBackupAuto/config.json
The only workaround i can think of is to mv the files in another step after "put" but i would like to avoid that.
Piotr S
added a comment - Im facing this problem now. This is a valid issue.
Exactly like described the workspace folder is forced and its creating very inconsistent behavior.
sshPut remote: remote, from: "${env.WORKSPACE}/" + "/" , into: "/srv/backuproot" + "/"
above code would sometimes create:
/srv/backuproot/backupMake_dev2/<files here>
but if previus pipeline didnt finish (valid case from me) jenkins will create 'backupMake_dev2@tmp' and on target sometimes i end up with:
/srv/backuproot/backupMake_dev2@tmp/<files here>
Expected behavior:
/srv/backuproot/<files here>
Is it possible to do achieve the expected behavior in any way?
--------------------------------------------------------------------------------------
With filter its also not helping:
sshPut remote: remote, from: "${env.WORKSPACE}/" + "/" , into: backuproot+ "/" , filterBy: "name" , filterRegex : /\.json$/
above code creates workspace folder on target like so:
/srv/makeBackupAuto/backupMake_dev2/config.json
Expected:
/srv/makeBackupAuto/config.json
The only workaround i can think of is to mv the files in another step after "put" but i would like to avoid that. I'm facing the same issue and the workaround that i found is shared pipeline libraries
sshUtils.groovy
#!/usr/bin/env groovy
Void call( Map map = [:], String quest, String host, String credsId, String path) {
def remote = [:]
remote.name = "${host}"
remote.host = "${host}"
remote.allowAnyHosts = true
remote.timeoutSec = 30
remote.logLevel = "SEVERE"
String target = "."
String filterRegex = ""
String script = ""
String command = ""
if (map.filterRegex) {
filterRegex = "${map.filterRegex}"
}
if (map.target) {
target = "${map.target}"
}
if (map.script) {
script = "${map.script}"
}
if (map.command) {
command = "${map.command}"
}
if (map.logLevel) {
remote.logLevel = "${map.logLevel}"
}
if (map.scp) {
remote.fileTransfer = "scp"
}
if (map.knownHosts) {
remote.allowAnyHosts = false
remote.knownHosts = "${map.knownHosts}"
}
if (map.retryCount) {
remote.retryCount= map.retryCount
remote.retryWaitSec = 10
}
def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(com.cloudbees.plugins.credentials.Credentials.class)
creds.each {
if (it.properties.id == credsId) {
if (it.properties.class =~ /.UsernamePasswordCredentialsImpl$/) {
withCredentials([usernamePassword(credentialsId: "${credsId}", passwordVariable: "pwd", usernameVariable: "user")]) {
remote.user = user
remote.password = pwd
doQuest( "${quest}", remote, "${path}", "${target}", "${filterRegex}", "${script}", "${command}")
}
}
else if (it.properties.class =~ /.BasicSSHUserPrivateKey$/) {
withCredentials([sshUserPrivateKey(credentialsId: "${credsId}", keyFileVariable: "privateKey", passphraseVariable: "passphrase", usernameVariable: "keyUser")]) {
remote.user = keyUser
remote.identityFile = privateKey
doQuest( "${quest}", remote, "${path}", "${target}", "${filterRegex}", "${script}", "${command}" )
}
}
else {
echo "Error: Credentials type not supported, Please choose one one this list [usernamePassword, BasicSSHUserPrivateKey]. Exiting..."
exit 1
}
}
}
}
Void doQuest( String quest, def remote, String path, String target, String filterRegex, String script, String command ) {
if (quest == "put") {
def flatten = path.tokenize('/')
String flattenPath = path - flatten.last()
String tmp = target + "/" + (path - flattenPath)
sshCommand remote: remote, command: "mkdir -p ${target}"
sshPut remote: remote, from: "${path}", into: "${target}", filterRegex: "${filterRegex}"
sshCommand remote: remote, command: "cp -R ${tmp}/* ${target} && rm -rf ${tmp}"
}
else if (quest == "execScript") {
if (script == "") {
echo "Error: Script file path need to be passed as optional parameter. Exiting..."
exit 1
}
else {
writeFile file: "script.sh", text: "${script}"
sshScript remote: remote, script: "script.sh"
}
}
else if (quest == "execCommand") {
sshCommand remote: remote, command: "${command}"
}
else if (quest == "get") {
sshGet remote: remote, from: "${path}", into: "${target}", override: true, filterRegex: "${filterRegex}"
}
else if (quest == "remove") {
sshRemove remote: remote, path: "${path}"
}
else {
echo "Error: Method not supported, Please choose one of this list [sshGet, sshPut, sshRemove, execScript, execCommand]. Exiting..."
exit 1
}
}
some calls examples
sshUtils("put", "host.domain.com", "creds-id", "path/local", target: "path/remote", filterRegex: /\.sh$/)
script {
fileContent = readFile(file: 'script.sh')
sshUtils("execScript", "host.domain.com", "private-ssh-key-creds-id", "path", logLevel: "FINEST", script: "${fileContent}")
}
Kader
added a comment - - edited I'm facing the same issue and the workaround that i found is shared pipeline libraries
sshUtils.groovy
#!/usr/bin/env groovy
Void call( Map map = [:], String quest, String host, String credsId, String path) {
def remote = [:]
remote.name = "${host}"
remote.host = "${host}"
remote.allowAnyHosts = true
remote.timeoutSec = 30
remote.logLevel = "SEVERE"
String target = "."
String filterRegex = ""
String script = ""
String command = ""
if (map.filterRegex) {
filterRegex = "${map.filterRegex}"
}
if (map.target) {
target = "${map.target}"
}
if (map.script) {
script = "${map.script}"
}
if (map.command) {
command = "${map.command}"
}
if (map.logLevel) {
remote.logLevel = "${map.logLevel}"
}
if (map.scp) {
remote.fileTransfer = "scp"
}
if (map.knownHosts) {
remote.allowAnyHosts = false
remote.knownHosts = "${map.knownHosts}"
}
if (map.retryCount) {
remote.retryCount= map.retryCount
remote.retryWaitSec = 10
}
def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(com.cloudbees.plugins.credentials.Credentials.class)
creds.each {
if (it.properties.id == credsId) {
if (it.properties.class =~ /.UsernamePasswordCredentialsImpl$/) {
withCredentials([usernamePassword(credentialsId: "${credsId}", passwordVariable: "pwd", usernameVariable: "user")]) {
remote.user = user
remote.password = pwd
doQuest( "${quest}", remote, "${path}", "${target}", "${filterRegex}", "${script}", "${command}")
}
}
else if (it.properties.class =~ /.BasicSSHUserPrivateKey$/) {
withCredentials([sshUserPrivateKey(credentialsId: "${credsId}", keyFileVariable: "privateKey", passphraseVariable: "passphrase", usernameVariable: "keyUser")]) {
remote.user = keyUser
remote.identityFile = privateKey
doQuest( "${quest}", remote, "${path}", "${target}", "${filterRegex}", "${script}", "${command}" )
}
}
else {
echo "Error: Credentials type not supported, Please choose one one this list [usernamePassword, BasicSSHUserPrivateKey]. Exiting..."
exit 1
}
}
}
}
Void doQuest( String quest, def remote, String path, String target, String filterRegex, String script, String command ) {
if (quest == "put") {
def flatten = path.tokenize('/')
String flattenPath = path - flatten.last()
String tmp = target + "/" + (path - flattenPath)
sshCommand remote: remote, command: "mkdir -p ${target}"
sshPut remote: remote, from: "${path}", into: "${target}", filterRegex: "${filterRegex}"
sshCommand remote: remote, command: "cp -R ${tmp}/* ${target} && rm -rf ${tmp}"
}
else if (quest == "execScript") {
if (script == "") {
echo "Error: Script file path need to be passed as optional parameter. Exiting..."
exit 1
}
else {
writeFile file: "script.sh", text: "${script}"
sshScript remote: remote, script: "script.sh"
}
}
else if (quest == "execCommand") {
sshCommand remote: remote, command: "${command}"
}
else if (quest == "get") {
sshGet remote: remote, from: "${path}", into: "${target}", override: true, filterRegex: "${filterRegex}"
}
else if (quest == "remove") {
sshRemove remote: remote, path: "${path}"
}
else {
echo "Error: Method not supported, Please choose one of this list [sshGet, sshPut, sshRemove, execScript, execCommand]. Exiting..."
exit 1
}
}
some calls examples
sshUtils("put", "host.domain.com", "creds-id", "path/local", target: "path/remote", filterRegex: /\.sh$/)
script {
fileContent = readFile(file: 'script.sh')
sshUtils("execScript", "host.domain.com", "private-ssh-key-creds-id", "path", logLevel: "FINEST", script: "${fileContent}")
}
Rich DiCroce
added a comment - I also encountered this. This seems to be a problem with the SFTP file transfer method. Setting fileTransfer to 'scp' on the remote object works around it. Note that the value is case-sensitive, so it must be lowercase, contrary to what the documentation says!
Rich DiCroce
added a comment - I also encountered this. This seems to be a problem with the SFTP file transfer method. Setting fileTransfer to 'scp' on the remote object works around it. Note that the value is case-sensitive, so it must be lowercase, contrary to what the documentation says! 
Please use filterBy, filterRegex options to copy contents of the folders instead of coping the entire folder, and also use
override: true option to override the contents.