Copying artifacts would be much faster if the plugin would only copy the changes using something along the lines of rsync -az --delete.
From build to build, the number of changed files is generally small. Even the built jars for intermediate steps will tend to be similar from run to run, differing only in the timestamps on the files inside.
In context: our project is relatively large. In the release build, we have to copy the installers from a matrix build which has the artifacts for each platform. Copying the installers takes 18 minutes in total and so far we have only set up the builds for about half the files we will ultimately be distributing.