Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56223

Jenkins discard-old-build caused open file handles

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Ubuntu 16.04
      Java 1.8.0_191
      Jenkins 2.138.4

       
      After Jenkins upgraded from 2.46.2 to 2.138.4 on Jan 5. 2019,

      I started to see many (deleted) open file handles.

      Those (deleted) open file handles seem caused by "discard old build" rotate.

      The "builds/.${old-build-number}" folder doesn't exist anymore,

      but when using "lsof" command to check, it showed up:

      sudo lsof -a -p $(pgrep -P $(sudo cat /var/run/jenkins/jenkins.pid)) | grep '(deleted)'

      eg

      java 34189 jenkins 5199r REG 252,0 1818111 17044616 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1394/log (deleted)

      java 34189 jenkins 5200r REG 252,0 1813394 17052273 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1388/log (deleted)

      java 34189 jenkins 5201r REG 252,0 1813394 17052273 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1388/log (deleted)

      java 34189 jenkins 5212r REG 252,0 1815212 17053470 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1398/log (deleted)

          [JENKINS-56223] Jenkins discard-old-build caused open file handles

          Rick Liu created issue -
          Rick Liu made changes -
          Description Original:  
          After Jenkins upgraded from 2.46.2 to 2.138.4 on Jan 5. 2019,
          I started to see many (deleted) open file handles.

          Those (deleted) open file handles seem caused by "discard old build" rotate.
          The "{{builds/.${old-build-number}}}" folder doesn't exist anymore,
          but when using "{{lsof}}" command to check,
          it showed up.

          {{sudo lsof -a -p $(pgrep -P $(sudo cat /var/run/jenkins/jenkins.pid)) | grep '(deleted)'}}
          New:  
          After Jenkins upgraded from 2.46.2 to 2.138.4 on Jan 5. 2019,

          I started to see many (deleted) open file handles.

          Those (deleted) open file handles seem caused by "discard old build" rotate.

          The "{{builds/.${old-build-number}}}" folder doesn't exist anymore,

          but when using "{{lsof}}" command to check, it showed up:

          {{sudo lsof -a -p $(pgrep -P $(sudo cat /var/run/jenkins/jenkins.pid)) | grep '(deleted)'}}


          eg

          bq. java 34189 jenkins 5199r REG 252,0 1818111 17044616 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1394/log (deleted)
          bq. java 34189 jenkins 5200r REG 252,0 1813394 17052273 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1388/log (deleted)
          bq. java 34189 jenkins 5201r REG 252,0 1813394 17052273 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1388/log (deleted)
          bq. java 34189 jenkins 5212r REG 252,0 1815212 17053470 /var/lib/jenkins/jobs/Projects/jobs/FortiClientEMS/jobs/br_6_Branch_Build/builds/.1398/log (deleted)
          Xavier Raynaud made changes -
          Comment [ I reproduce the issue here (Jenkins 2.235.1)

          {{I also see many entries like that in log:}}

           

          {{2020-07-04 02:07:56.265+0000 [id=40204] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build Discarder}}
           {{Also: java.nio.file.NoSuchFileException: /var/lib/jenkins/jobs/myjob/builds/3496 -> /var/lib/jenkins/jobs/myjob/builds/.3496}}
            at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
            at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
            at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
            at java.nio.file.Files.move(Files.java:1395)
            at hudson.model.Run.delete(Run.java:1645)
            at hudson.tasks.LogRotator.perform(LogRotator.java:181)

          {{Hope it helps...}} ]

            nkns165 nkns165
            totoroliu Rick Liu
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: