Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67624

FileAlreadyExistsException /root/.jenkins/updates

    • 2.366

      likely caused by
      https://github.com/jenkinsci/jenkins/pull/6025

      I've hit this twice recently with the core-pr-tester

      I assume two things are trying to load the update center data at the same time before the updates directory exists.

      1 time plugin data wasn't available and the other time it did load.

      This may also be found at: /root/.jenkins/secrets/initialAdminPassword
      
      *************************************************************
      *************************************************************
      *************************************************************
      
      2022-01-18 08:25:54.927+0000 [id=35]	WARNING	hudson.model.UpdateCenter#updateDefaultSite: Upgrading Jenkins. Failed to update the default Update Site 'default'. Plugin upgrades may fail.
      java.nio.file.FileAlreadyExistsException: /root/.jenkins/updates
      	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
      	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
      	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
      	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
      	at java.base/java.nio.file.Files.createDirectory(Files.java:690)
      	at hudson.Util.createDirectories(Util.java:1785)
      	at hudson.util.TextFile.write(TextFile.java:100)
      	at hudson.model.UpdateSite.updateData(UpdateSite.java:248)
      	at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:219)
      	at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:214)
      	at hudson.model.UpdateCenter.updateDefaultSite(UpdateCenter.java:2667)
      	at jenkins.install.SetupWizard.init(SetupWizard.java:206)
      	at jenkins.install.InstallState$InitialSecuritySetup.initializeState(InstallState.java:182)
      	at jenkins.model.Jenkins.setInstallState(Jenkins.java:1129)
      	at jenkins.install.InstallUtil.proceedToNextStateFrom(InstallUtil.java:98)
      	at jenkins.install.InstallState$Unknown.initializeState(InstallState.java:88)
      	at jenkins.model.Jenkins$15.run(Jenkins.java:3491)
      	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1156)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      

          [JENKINS-67624] FileAlreadyExistsException /root/.jenkins/updates

          Basil Crow added a comment -

          Do you have steps to reproduce the problem?

          Basil Crow added a comment - Do you have steps to reproduce the problem?

          Tim Jacomb added a comment -

          Run this https://github.com/jenkinsci/core-pr-tester#how-to-use

          2 out of the last 5 or so I ran had this happen. Just after the initial admin password.

          Running from a clean JENKINS_HOME should be the same thing as well

          Tim Jacomb added a comment - Run this https://github.com/jenkinsci/core-pr-tester#how-to-use 2 out of the last 5 or so I ran had this happen. Just after the initial admin password. Running from a clean JENKINS_HOME should be the same thing as well

          Tim Jacomb added a comment -

          Reproduced just by running:

          docker run -dt -p 8085:8080 jenkins/jenkins:jdk17-preview
          

          Tim Jacomb added a comment - Reproduced just by running: docker run -dt -p 8085:8080 jenkins/jenkins:jdk17-preview

          Basil Crow added a comment -

          I could never reproduce the problem as described above. I guess my computer is just too fast. I have amazing I/O on my desktop PC due to my use of a mirrored ZFS storage pool on M.2 NVMe SSDs.

          timja Pro-tip: consistently reproducing a race (rather than reproducing it by just getting lucky) almost always involves adding a sleep statement somewhere in the code to expose the race consistently.

          Basil Crow added a comment - I could never reproduce the problem as described above. I guess my computer is just too fast. I have amazing I/O on my desktop PC due to my use of a mirrored ZFS storage pool on M.2 NVMe SSDs. timja Pro-tip: consistently reproducing a race (rather than reproducing it by just getting lucky) almost always involves adding a sleep statement somewhere in the code to expose the race consistently.

          Basil Crow added a comment -

          I can consistently reproduce the race by flipping a coin, then with probability of 0.5 sleeping in between checking for the existence of a directory and creating it. A probability of 0.5 works because there are enough racing threads such that at least one of them will hit the sleep.

          --- a/core/src/main/java/hudson/Util.java
          +++ b/core/src/main/java/hudson/Util.java
          @@ -75,6 +75,7 @@ import java.nio.file.attribute.PosixFilePermissions;
           import java.security.DigestInputStream;
           import java.security.MessageDigest;
           import java.security.NoSuchAlgorithmException;
          +import java.security.SecureRandom;
           import java.text.NumberFormat;
           import java.text.ParseException;
           import java.time.LocalDate;
          @@ -1786,6 +1787,13 @@ public class Util {
                   for (Path name : parent.relativize(dir)) {
                       child = child.resolve(name);
                       if (!Files.isDirectory(child)) {
          +                if (new SecureRandom().nextInt(2) == 0) {
          +                    try {
          +                        Thread.sleep(1000L);
          +                    } catch (InterruptedException e) {
          +                        // ignore
          +                    }
          +                }
                           Files.createDirectory(child, attrs);
                       }
                   }
          

          Basil Crow added a comment - I can consistently reproduce the race by flipping a coin, then with probability of 0.5 sleeping in between checking for the existence of a directory and creating it. A probability of 0.5 works because there are enough racing threads such that at least one of them will hit the sleep. --- a/core/src/main/java/hudson/Util.java +++ b/core/src/main/java/hudson/Util.java @@ -75,6 +75,7 @@ import java.nio.file.attribute.PosixFilePermissions; import java.security.DigestInputStream; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; + import java.security.SecureRandom; import java.text.NumberFormat; import java.text.ParseException; import java.time.LocalDate; @@ -1786,6 +1787,13 @@ public class Util { for (Path name : parent.relativize(dir)) { child = child.resolve(name); if (!Files.isDirectory(child)) { + if ( new SecureRandom().nextInt(2) == 0) { + try { + Thread .sleep(1000L); + } catch (InterruptedException e) { + // ignore + } + } Files.createDirectory(child, attrs); } }

          Daniel Beck added a comment -

          Seems like a straightforward fix we could backport to prevent a bad first impression for folks choosing LTS.

          Daniel Beck added a comment - Seems like a straightforward fix we could backport to prevent a bad first impression for folks choosing LTS.

            basil Basil Crow
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: