Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50824

"Excessive" exception logging on writing org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xm

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • Jenkins LTS 2.107.1 on Windows Server 2008 R2 x64,
      ws-cleanup 0.34, resource-disposer 0.8

    Description

      Hi!

      In our environment with 4 agents and many concurrent builds (of different jobs), the log contains many entries like:

      Apr 13, 2018 5:26:09 AM WARNING hudson.util.AtomicFileWriter commit
      Unable to move atomically, falling back to non-atomic move.
      java.nio.file.AccessDeniedException: C:\webserver\Jenkins\atomic466027149674640611tmp -> C:\webserver\Jenkins\org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml
      at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
      at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      at java.nio.file.Files.move(Files.java:1395)
      at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
      at hudson.XmlFile.write(XmlFile.java:198)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:201)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$400(AsyncResourceDisposer.java:73)
      at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:286)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      IMHO - I think, this issue is caused by (quasi) concurrent writes by the method persist() in AsyncResourceDisposer.java.

      Therefore, I have built an private variant of this plugin to test, whether this issue can be fixed by simply adding the synchronized-keyword to the definition of the the persist-method (in the Java-class AsyncResourceDisposer). This has fixed this issue; I.e., there are no log-entries of this kind since about 2 days ...
      Note: Maybe, my suggested fix should be used in a next release (after review of the maintainer)?

      Best regards from Salzburg,
      Markus

      Attachments

        Issue Links

          Activity

            I am just releasing 0.11 that throttles concurrent disposer threads to 10 that should prevent such a high contention of the file and reduce the likelihood of this. Can you please report whether the problem persists or not? Thanks

            olivergondza Oliver Gondža added a comment - I am just releasing 0.11 that throttles concurrent disposer threads to 10 that should prevent such a high contention of the file and reduce the likelihood of this. Can you please report whether the problem persists or not? Thanks

            Hi Oliver!

            This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!

             

            At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master);
            I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file;
            and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism).

            Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?!

            Best regards from Salzburg,
            Markus

            P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).

             

            gmc_devel GMC Software Development B&R Corporate added a comment - Hi Oliver! This issue still exists for the origin resource-disposer release 0.10 and does not occur with "my" variant with the synchronized persist()-method!   At least (I think - based on the stack-trace), this will occur if more than on thread updates the file 'org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml' concurrently (on master); I.e., as I have determined the AtomicFileWriter will write to a temporary file, deletes the origin file and renames/moves the written file; and if more threads do the same in the meantime, there's a high probability of interruption and causing a failure (which will be solved by a fallback-mechanism). Additional note: If I remember correctly, this issue has begun with update of LTS core 2.89.4 to 2.107.1. There were some noticeable changes in File-I/O (e.g. 'Improve robustness and error handling of various file operations by switching to NIO. (issue 47324, issue 48405)') ?! Best regards from Salzburg, Markus P.S.: If you provide me a download for a snapshot-release of 0.11, I can check whether this issue has been fixed (or rarely occurs).  

            Failed to get the release out yesterday due to failing test. Corrected now: http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/resource-disposer/0.11/resource-disposer-0.11.hpi

            The exactly same bits will appear in update center with delay.

            olivergondza Oliver Gondža added a comment - Failed to get the release out yesterday due to failing test. Corrected now: http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/resource-disposer/0.11/resource-disposer-0.11.hpi The exactly same bits will appear in update center with delay.

            olivergondza I stumbled into here by chance, I seem to understand this issue is actually resolved and hence could be marked as such isn't it?
            Cheers

            batmat Baptiste Mathus added a comment - olivergondza I stumbled into here by chance, I seem to understand this issue is actually resolved and hence could be marked as such isn't it? Cheers

            I presume the contention is now reduced to manageable levels. I am resolving this as there is noone else complaining since the improvement was released.

             

            olivergondza Oliver Gondža added a comment - I presume the contention is now reduced to manageable levels. I am resolving this as there is noone else complaining since the improvement was released.  

            People

              olivergondza Oliver Gondža
              gmc_devel GMC Software Development B&R Corporate
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: