Details
-
Improvement
-
Status: Resolved (View Workflow)
-
Minor
-
Resolution: Fixed
-
Jenkins LTS 2.107.1 on Windows Server 2008 R2 x64,
ws-cleanup 0.34, resource-disposer 0.8
Description
Hi!
In our environment with 4 agents and many concurrent builds (of different jobs), the log contains many entries like:
Apr 13, 2018 5:26:09 AM WARNING hudson.util.AtomicFileWriter commit
Unable to move atomically, falling back to non-atomic move.
java.nio.file.AccessDeniedException: C:\webserver\Jenkins\atomic466027149674640611tmp -> C:\webserver\Jenkins\org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
at hudson.XmlFile.write(XmlFile.java:198)
at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:201)
at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$400(AsyncResourceDisposer.java:73)
at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:286)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
IMHO - I think, this issue is caused by (quasi) concurrent writes by the method persist() in AsyncResourceDisposer.java.
Therefore, I have built an private variant of this plugin to test, whether this issue can be fixed by simply adding the synchronized-keyword to the definition of the the persist-method (in the Java-class AsyncResourceDisposer). This has fixed this issue; I.e., there are no log-entries of this kind since about 2 days ...
Note: Maybe, my suggested fix should be used in a next release (after review of the maintainer)?
Best regards from Salzburg,
Markus
Attachments
Issue Links
- relates to
-
JENKINS-52402 Job fails randomly when attempt to lock resources
-
- Open
-
-
-
- Fixed but Unreleased
-
Activity
GMC Software Development B&R Corporate
added a comment - Hi Oliver!
This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!
At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master);
I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file;
and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism).
Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?!
Best regards from Salzburg,
Markus
P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).
GMC Software Development B&R Corporate
added a comment - Hi Oliver!
This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!
At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master);
I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file;
and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism).
Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?!
Best regards from Salzburg,
Markus
P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).
Failed to get the release out yesterday due to failing test. Corrected now: http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/resource-disposer/0.11/resource-disposer-0.11.hpi
The exactly same bits will appear in update center with delay.
olivergondza I stumbled into here by chance, I seem to understand this issue is actually resolved and hence could be marked as such isn't it?
Cheers
I presume the contention is now reduced to manageable levels. I am resolving this as there is noone else complaining since the improvement was released.
I am just releasing 0.11 that throttles concurrent disposer threads to 10 that should prevent such a high contention of the file and reduce the likelihood of this. Can you please report whether the problem persists or not? Thanks