[JENKINS-52404] Job fails randomly when using copyArtifact

Type: Bug
Resolution: Won't Fix
Priority: Minor
Component/s: copyartifact-plugin
Labels:
None
Environment:
Jenkins ver. 2.107.3
Windows 10x64 on master and slave-nodes
copy-artifact-plugin 1.39.1

Similar Issues:
Powered by SuggestiMate

Show

As the title states, the job fails randomly. Thus, I could not find out any more precise scenarios where the error occurs.

[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also:   java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
		at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
		at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
		at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
		at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
		at java.nio.file.Files.move(Unknown Source)
		at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
	at java.nio.file.Files.move(Unknown Source)
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
	at hudson.model.Fingerprint.save(Fingerprint.java:1295)
	at hudson.model.Fingerprint.save(Fingerprint.java:1245)
	at hudson.model.Fingerprint.add(Fingerprint.java:1031)
	at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Finished: FAILURE

relates to

JENKINS-52402 Job fails randomly when attempt to lock resources

Open

JENKINS-50824 "Excessive" exception logging on writing org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xm

Resolved

JENKINS-53187 Exception when using deleteDir() from pipeline

Closed

Marco Steffan created issue - 2018-07-06 06:50

Marco Steffan made changes - 2018-07-06 06:50

Link

New: This issue relates to JENKINS-52402 [ JENKINS-52402 ]

Marco Steffan made changes - 2018-07-06 06:51

Link

New: This issue relates to ~~JENKINS-50824~~ [ ~~JENKINS-50824~~ ]

Marco Steffan made changes - 2018-07-06 07:02

Description

Original: Jobs randomly fail when copying artifacts

{noformat}
[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also: java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
at hudson.model.Fingerprint.save(Fingerprint.java:1295)
at hudson.model.Fingerprint.save(Fingerprint.java:1245)
at hudson.model.Fingerprint.add(Fingerprint.java:1031)
at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Finished: FAILURE{noformat}

New: As the title states, the job fails randomly. Thus, I could not find out any more precise scenarios where the error occurs.

{noformat}
[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also: java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
at hudson.model.Fingerprint.save(Fingerprint.java:1295)
at hudson.model.Fingerprint.save(Fingerprint.java:1245)
at hudson.model.Fingerprint.add(Fingerprint.java:1031)
at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Finished: FAILURE{noformat}

Marco Steffan made changes - 2018-07-06 07:02

Environment

Original: Jenkins ver. 2.107.3 Win10x64
copy-artifact-plugin 1.39.1

New: Jenkins ver. 2.107.3
Windows 10x64 on master and slave-nodes
copy-artifact-plugin 1.39.1

Marco Steffan made changes - 2018-08-29 12:27

Link

New: This issue relates to ~~JENKINS-53187~~ [ ~~JENKINS-53187~~ ]

Mathias Helminger added a comment - 2018-10-19 09:39 - edited

In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded.

For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. (~~JENKINS-15331~~) (Turning kaspersky off temporarily helped but we do want to keep it running)

Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...

Mathias Helminger added a comment - 2018-10-19 09:39 - edited In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded. For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. ( JENKINS-15331 ) (Turning kaspersky off temporarily helped but we do want to keep it running) Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...

Marco Steffan added a comment - 2018-10-19 11:27

ilmarinen We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good...

https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311 might be of interest for you.

Marco Steffan added a comment - 2018-10-19 11:27 ilmarinen We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good... https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311 might be of interest for you.

ikedam added a comment - 2019-05-11 03:57

There’s no fundamental resolution for this issue as it comes from the OS level issue.
And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain.

Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

ikedam added a comment - 2019-05-11 03:57 There’s no fundamental resolution for this issue as it comes from the OS level issue. And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain. Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

ikedam made changes - 2019-05-11 03:57

Assignee		New: Marco Steffan [ marcosteffan ]
Resolution		New: Won't Fix [ 2 ]
Status	Original: Open [ 1 ]	New: Fixed but Unreleased [ 10203 ]

Assignee:: Marco Steffan

Reporter:: Marco Steffan

Votes:: 1 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2018-07-06 06:50

Updated:: 2019-05-11 03:57

Resolved:: 2019-05-11 03:57

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Mathias Helminger added a comment - 2018-10-19 09:39, Edited by Mathias Helminger - 2018-10-19 09:42

Expand comment: Mathias Helminger added a comment - 2018-10-19 09:39, Edited by Mathias Helminger - 2018-10-19 09:42

Collapse comment: Marco Steffan added a comment - 2018-10-19 11:27

Expand comment: Marco Steffan added a comment - 2018-10-19 11:27

Collapse comment: ikedam added a comment - 2019-05-11 03:57

Expand comment: ikedam added a comment - 2019-05-11 03:57

People

Dates