Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50824

"Excessive" exception logging on writing org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xm

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Labels:
    • Environment:
      Jenkins LTS 2.107.1 on Windows Server 2008 R2 x64,
      ws-cleanup 0.34, resource-disposer 0.8
    • Similar Issues:

      Description

      Hi!

      In our environment with 4 agents and many concurrent builds (of different jobs), the log contains many entries like:

      Apr 13, 2018 5:26:09 AM WARNING hudson.util.AtomicFileWriter commit
      Unable to move atomically, falling back to non-atomic move.
      java.nio.file.AccessDeniedException: C:\webserver\Jenkins\atomic466027149674640611tmp -> C:\webserver\Jenkins\org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml
      at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
      at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      at java.nio.file.Files.move(Files.java:1395)
      at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
      at hudson.XmlFile.write(XmlFile.java:198)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:201)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$400(AsyncResourceDisposer.java:73)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:286)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      IMHO - I think, this issue is caused by (quasi) concurrent writes by the method persist() in AsyncResourceDisposer.java.

      Therefore, I have built an private variant of this plugin to test, whether this issue can be fixed by simply adding the synchronized-keyword to the definition of the the persist-method (in the Java-class AsyncResourceDisposer). This has fixed this issue; I.e., there are no log-entries of this kind since about 2 days ...
      Note: Maybe, my suggested fix should be used in a next release (after review of the maintainer)?

      Best regards from Salzburg,
      Markus

        Attachments

          Issue Links

            Activity

            Hide
            olivergondza Oliver Gondža added a comment -

            I am just releasing 0.11 that throttles concurrent disposer threads to 10 that should prevent such a high contention of the file and reduce the likelihood of this. Can you please report whether the problem persists or not? Thanks

            Show
            olivergondza Oliver Gondža added a comment - I am just releasing 0.11 that throttles concurrent disposer threads to 10 that should prevent such a high contention of the file and reduce the likelihood of this. Can you please report whether the problem persists or not? Thanks
            Hide
            gmc_devel GMC Software Development B&R Corporate added a comment -

            Hi Oliver!

            This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!

             

            At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master);
            I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file;
            and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism).

            Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?!

            Best regards from Salzburg,
            Markus

            P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).

             

            Show
            gmc_devel GMC Software Development B&R Corporate added a comment - Hi Oliver! This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!   At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master); I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file; and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism). Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?! Best regards from Salzburg, Markus P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).  
            Hide
            olivergondza Oliver Gondža added a comment -

            Failed to get the release out yesterday due to failing test. Corrected now: http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/resource-disposer/0.11/resource-disposer-0.11.hpi

            The exactly same bits will appear in update center with delay.

            Show
            olivergondza Oliver Gondža added a comment - Failed to get the release out yesterday due to failing test. Corrected now: http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/resource-disposer/0.11/resource-disposer-0.11.hpi The exactly same bits will appear in update center with delay.
            Hide
            batmat Baptiste Mathus added a comment -

            Oliver Gondža I stumbled into here by chance, I seem to understand this issue is actually resolved and hence could be marked as such isn't it?
            Cheers

            Show
            batmat Baptiste Mathus added a comment - Oliver Gondža I stumbled into here by chance, I seem to understand this issue is actually resolved and hence could be marked as such isn't it? Cheers
            Hide
            olivergondza Oliver Gondža added a comment -

            I presume the contention is now reduced to manageable levels. I am resolving this as there is noone else complaining since the improvement was released.

             

            Show
            olivergondza Oliver Gondža added a comment - I presume the contention is now reduced to manageable levels. I am resolving this as there is noone else complaining since the improvement was released.  

              People

              Assignee:
              olivergondza Oliver Gondža
              Reporter:
              gmc_devel GMC Software Development B&R Corporate
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: