I can confirm it. When copy 150MB WAR using scp the copy speed is 30MB/s, when archiving by Jenkins the rate is only about 3MB/s. I have tried steps described in JENKINS-3922 but no success. Running Jenkins v1.513 with slave v2.23.

We found that problem is not only copy of artifacts. Also upload artifacts to Nexus via Deploy Artifacts to Maven repository is slow only 3MB/s. When setup job on slave to run mvn deploy the upload speed is 30MB/s. So something must be wrong with master server. The master runs on Tomcat 6.0.36.

I have also look at Jenkins sources and what I understand that uploading artifacts is done via Maven native API, so I think that there should be no differences when calling maven in build and in artifacts upload on master.

Do you have any hints, where the problem may be? We are looking on tomcat, java, OS setup. But no luck yet.

Same problem with Jenkins 1.509.2 and Copy Artifacts 1.27

Same problem with LTS 1.509.2

Same problem with LTS 1.509.3

Same problem on LTS 1.509.3, any workaround for this major issue ? (avoiding a downgrade...)

Same problem on 1.517. The slave machine is windows xp.

We have some jobs that require archiving less than 1 MB of artifacts (with 1.535 on Windows 7 Slave), but they are stuck in "archiving artifacts" for HOURS!! Other projects produce much bigger artifact but get the job done significantly faster.

Does the archiving somehow depend on the number of files in the workspace or the size of the workspace?

We found the problem. Although the log file showed "archiving artifacts", jenkins was waiting for the previous concurrent build to finish.

Still reproducible on Jenkins ver. 1.537

Same problem on 1.532.2, master and slave are Linux

Same problem on 1.565. Master is linux, slave is Windows jnlp. Unlike what Marcel Beister reported, there is a transfer going on throughout the whole time for us. It's just extremely slow (250kB/s instead of 100mB/s).

We're seeing this issue too, on 1.554.2. Unix slaves can archive artifacts quickly, but it takes a factor of about 10 times greater to upload the artifacts from windows slaves.

Total 11GB artifacts copy takes 30 minutes from Windows slave to Linux master.
There is another critical bug https://issues.jenkins-ci.org/browse/JENKINS-10629 also.
I decided copy artifacts directly to $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive/.

• create public directory in Windows slave allowing anonymous read access
• kick network copy shell script using smbclient on Linux master from Windows batch script via ssh(putty's plink)

11GB copy using smbclient takes only 3 minutes.
Archive directories in matrix project has more complicated path so bothersome but possible too.

I'm seeing the same issue with Jenkins v. 1.581 (Master, Windows 7).
My slaves are Windows 7/Windows 2012 machines with jenkins running as a service on them. Retrieving artifacts from upstream job takes a very long time (approx. 7 min for 750 MB ~ 100MB/min).
The slaves are connected using JNLP agent
Archiving artifacts is about the same: 12 min. for 2 GB.
How can I provide additional information which might help in resolving this issue?

I do not even know where to start. On our build machines (similar setup) Jenkins has just finished a build and has been sitting there doing absolutely nothing for the last 1h, so I looked up the issue online. Hope for improvements.

Does this issue still occur on recent Jenkins versions? If so, what kind of slave (SSH, JNLP, …) is affected? I think there was a performance fix a few months ago…

I just upgraded to 1.638 with a Win7 master and Win2012 slaves connected through JNLP. The slaves are running Jenkins as a Windows service.
All machines run as virtual machines with non-dedicated disks. However, I would still assume the Network speed to be the bottleneck no matter what?

If I archive artifacts (3,95 GB) from my build job it takes approx. 15 min 30 sec. = 4.32 MBps. The artifacts consist of approx. 4100 files.

Copying artifacts from the master node (778MB) takes approx 8 min 50 sec. =1.47 MBps. The artifacts copied consisted of approx. 2.770 files.

I've not made elaborate statistical sampling but the above is more less the normal time for copying artifacts.
If I take a project with few files, I get 350 MB transferred in 17 sec. resulting in 20.6 MBps.

So the slow transfer times in my case seems related to the transfer of many small files whereas transferring a few large files seem to go sufficiently fast.
Would it make sense to zip artifacts prior to archiving? That sort of goes against the 'copy artifacts' step where it's possible to select/deselect files based on pattern? I don't assume this would be possible if I zip all the files prior to archiving.

I've introduced a zip/unzip step in our build proces, which has HALVED the time used spent on check-out and compile.

Our steps consists of check-out and archiving og 11500 files (approx. 350 MB) and subsequent compilation in a separate job. By zipping files prior to archiving and the unzipping after copying artifacts we've managed to reduce time spent checking out and compiling from 10 min (check-out/archiving/fingerprinting) + 30 min (copying, compiling, archiving) to 4 min + approx. 15 min.

I don't know if this is Windows specific but it might be relevant to make it an option of the copy-artifacts and archiving functionality that it should support zipping through check-boxes to reduce the amount of steps involved? I've used powershell to zip and unzip the files in between the jobs.

With Linux slaves our builds generates large artifacts (2GB+ after compressed with XZ). Copying these from and to builds takes a while obviously, the main reason seems to be that Jenkins archives via its control channel (e.g. ssh slave - using java SSH implementation JSCH). The java ssh just can't get anywhere near 1Gb/s network speed that native SSH can manage easily.

I don't think this plugin will ever be able to provide suitable speeds without a major redesign to support alternative copy methods - e.g. using native SCP, SSL etc where supported by the slave/master. If going to the effort, I would highly recommend also separating the master node from the archive storage node - e.g. we started with Jenkins in a small VM, it now requires one with 300+ of GB of storage and can only manage the last 48 hours of artifacts.
Delta-compression is useless for us as well, it works on only a certain type of data.

navlrac, can you clarify what you mean by "this plugin will never be able to provide suitable speeds..."? My understanding is that this bug is against Jenkins core. Do you mean the [ssh-slaves-plugin|https://github.com/jenkinsci/ssh-slaves-plugin]?

Also, has anyone achieved a reasonable workaround for archiving artifacts on jenkins agents?

We're using ssh-slaves-plugin for our linux build nodes, and experiencing abysmal agent-master throughput, despite their 10Gbps link. We could replace our usage of `archiveArtifacts` steps with a custom groovy call to use system scp to master, but I'd of course rather not have to reinvent this wheel. I found [the publish-over-ssh plugin|https://github.com/jenkinsci/publish-over-ssh-plugin], but it doesn't seem to be maintained anymore. 

FWIW, we're using Debian Buster, Jenkins 2.222.3, and this java version on all master/agents:

jenkins@jenkins-testing-agent-2:~$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Debian-1deb10u1, mixed mode, sharing)

Update: I created this Improvement issue over in ssh-slaves-plugin component, in hopes of getting some more ssh-specific eyes on this issue: https://issues.jenkins-ci.org/browse/JENKINS-63517

Master: Windows running 2.249.2

Slave: macOS 10.13.6

The slave was launched using JNLP.

In the process of investigating why the 'Copy Artifact' plugin was taking 12 mins to copy about 250 MB from the Master to the Slave, I incorrectly assumed that the plugin was slow.

I realized this the hard way by reconfiguring my jobs to use Azure Storage where the master uploaded to Azure and the slave downloaded from Azure. Before I embarked on the Azure Storage alternative, I ran a test on the Mac by downloading the necessary file from Azure - 250 MB in about 20 seconds. But when I finally ran the job, the download from Azure took about 13 minutes!

This is definitely not resolved.