[JENKINS-52404] Job fails randomly when using copyArtifact - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Fixed but Unreleased (View Workflow)
Priority: Minor
Resolution: Won't Fix
Component/s: copyartifact-plugin
Labels:
None
Environment:
Jenkins ver. 2.107.3
Windows 10x64 on master and slave-nodes
copy-artifact-plugin 1.39.1

Similar Issues:

Show

Description

As the title states, the job fails randomly. Thus, I could not find out any more precise scenarios where the error occurs.

[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also:   java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
		at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
		at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
		at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
		at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
		at java.nio.file.Files.move(Unknown Source)
		at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
	at java.nio.file.Files.move(Unknown Source)
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
	at hudson.model.Fingerprint.save(Fingerprint.java:1295)
	at hudson.model.Fingerprint.save(Fingerprint.java:1245)
	at hudson.model.Fingerprint.add(Fingerprint.java:1031)
	at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Finished: FAILURE

Attachments

Issue Links

relates to

JENKINS-52402 Job fails randomly when attempt to lock resources

Open

JENKINS-50824 "Excessive" exception logging on writing org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xm

Resolved

JENKINS-53187 Exception when using deleteDir() from pipeline

Closed

Activity

Ascending order - Click to sort in descending order

Marco Steffan created issue - 2018-07-06 06:50

Marco Steffan made changes - 2018-07-06 06:50

Field	Original Value	New Value
Link		This issue relates to JENKINS-52402 [ JENKINS-52402 ]

Marco Steffan made changes - 2018-07-06 06:51

Link

This issue relates to ~~JENKINS-50824~~ [ ~~JENKINS-50824~~ ]

Marco Steffan made changes - 2018-07-06 07:02

Description

Jobs randomly fail when copying artifacts

{noformat}
[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also: java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
at hudson.model.Fingerprint.save(Fingerprint.java:1295)
at hudson.model.Fingerprint.save(Fingerprint.java:1245)
at hudson.model.Fingerprint.add(Fingerprint.java:1031)
at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Finished: FAILURE{noformat}

As the title states, the job fails randomly. Thus, I could not find out any more precise scenarios where the error occurs.

{noformat}
[Pipeline] [DSWZ10001] }
[Pipeline] [DSWZ10022] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Archive)
[Pipeline] copyArtifacts
Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
[Pipeline] copyArtifacts
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Also: java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
at hudson.model.Fingerprint.save(Fingerprint.java:1295)
at hudson.model.Fingerprint.save(Fingerprint.java:1245)
at hudson.model.Fingerprint.add(Fingerprint.java:1031)
at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Finished: FAILURE{noformat}

Marco Steffan made changes - 2018-07-06 07:02

Environment

Jenkins ver. 2.107.3 Win10x64
copy-artifact-plugin 1.39.1

Jenkins ver. 2.107.3
Windows 10x64 on master and slave-nodes
copy-artifact-plugin 1.39.1

Marco Steffan made changes - 2018-08-29 12:27

Link

This issue relates to ~~JENKINS-53187~~ [ ~~JENKINS-53187~~ ]

Mathias Helminger added a comment - 2018-10-19 09:39 - edited

In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded.

For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. (~~JENKINS-15331~~) (Turning kaspersky off temporarily helped but we do want to keep it running)

Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...

Mathias Helminger added a comment - 2018-10-19 09:39 - edited In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded. For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. ( JENKINS-15331 ) (Turning kaspersky off temporarily helped but we do want to keep it running) Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...

Marco Steffan added a comment - 2018-10-19 11:27

ilmarinen We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good...

https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311 might be of interest for you.

Marco Steffan added a comment - 2018-10-19 11:27 ilmarinen We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good... https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311 might be of interest for you.

ikedam added a comment - 2019-05-11 03:57

There’s no fundamental resolution for this issue as it comes from the OS level issue.
And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain.

Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

ikedam added a comment - 2019-05-11 03:57 There’s no fundamental resolution for this issue as it comes from the OS level issue. And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain. Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

ikedam made changes - 2019-05-11 03:57

Assignee		Marco Steffan [ marcosteffan ]
Resolution		Won't Fix [ 2 ]
Status	Open [ 1 ]	Fixed but Unreleased [ 10203 ]

People

Assignee:: Marco Steffan

Reporter:: Marco Steffan

Votes:: 1 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2018-07-06 06:50

Updated:: 2019-05-11 03:57

Resolved:: 2019-05-11 03:57