Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62985

Jenkins leaks a file handle when viewing a build log that contains an improperly encoded ConsoleNote (base64 error)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 2.222.3, 2.235.1
      java 1.8.0 201, java 1.8.0.251
      tomcat8 8.0.32-1ubuntu1.11
    • Similar Issues:
    • Released As:
      Jenkins 2.265 Stapler 1.261

      Description

      Repro:
      1. Create a pipeline build job
      2. Run a build
      3. Replace the log on (ex: /var/lib/jenkins/jobs/ThisIsATest/builds/55/log) with the attached bad.log file bad.log
      4. Browse to the build in Jenkins, view the log as plain text (https://jenkins/job/ThisIsATest/55/consoleText).
      5. Observe the attached error in the jenkins log. error.log
      6. Observe that Jenkins has leaked a file descriptor to the log on disk.

      For example:

      root@jenkins-test-local:/var/lib/jenkins/jobs/ThisIsATest/builds/55# lsof -p 10007|grep log
      java    10007 jenkins  mem       REG                8,1  1213761  408229 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-servicecatalog-1.11.791.jar
      java    10007 jenkins  mem       REG                8,1   563715  408224 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-logs-1.11.791.jar
      java    10007 jenkins  mem       REG                8,1   138625  408266 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-marketplacecatalog-1.11.791.jar
      java    10007 jenkins    1w      REG                8,1 24123228  395200 /var/log/tomcat8/catalina.out
      java    10007 jenkins    2w      REG                8,1 24123228  395200 /var/log/tomcat8/catalina.out
      java    10007 jenkins    3w      REG                8,1  1047507  392476 /var/lib/jenkins/gc/gc-2020-07-06_16-15-31.log.0.current
      java    10007 jenkins   17w      REG                8,1 10462062  408102 /var/log/tomcat8/catalina.2020-07-06.log
      java    10007 jenkins   18w      REG                8,1   348255  394397 /var/log/tomcat8/localhost.2020-07-06.log
      java    10007 jenkins   59w      REG                8,1  7486004  399682 /var/log/tomcat8/access.log
      java    10007 jenkins   66r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   68r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   69r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   70r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   71r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   72r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   73r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   74r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   75r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   76r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   77r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   78r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   79r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   80r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   82r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins   91r      REG               0,45      152  269033 /mnt/jenkins/jobs/ThisIsATest/builds/55/log (127.0.0.1:/exports/jenkins)
      java    10007 jenkins  361r      REG                8,1   138625  408266 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-marketplacecatalog-1.11.791.jar
      java    10007 jenkins  417r      REG                8,1   563715  408224 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-logs-1.11.791.jar
      java    10007 jenkins  441r      REG                8,1  1213761  408229 /var/lib/jenkins/plugins/aws-java-sdk/WEB-INF/lib/aws-java-sdk-servicecatalog-1.11.791.jar

        Attachments

        1. bad.log
          0.1 kB
        2. error.log
          24 kB

          Issue Links

            Activity

            Hide
            mikeryan_tan Mike Ryan added a comment -

            The file handle leak is the result of a bug that produces corrupt log lines.

            Show
            mikeryan_tan Mike Ryan added a comment - The file handle leak is the result of a bug that produces corrupt log lines.
            Hide
            mikeryan_tan Mike Ryan added a comment -

            This file handle leak is especially problematic in my environment, as prevents us from properly rotating logs/discarding builds. Our jenkins build logs sit on an NFS share, and we end up with `.nfs123412341234` files that cannot be deleted ("Device or resource busy" errors). Pretty much: https://support.cloudbees.com/hc/en-us/articles/115001371588-Device-or-resource-busy-error

            Show
            mikeryan_tan Mike Ryan added a comment - This file handle leak is especially problematic in my environment, as prevents us from properly rotating logs/discarding builds. Our jenkins build logs sit on an NFS share, and we end up with `.nfs123412341234` files that cannot be deleted ("Device or resource busy" errors). Pretty much: https://support.cloudbees.com/hc/en-us/articles/115001371588-Device-or-resource-busy-error
            Hide
            raihaan Raihaan Shouhell added a comment -

            Confirm I can reproduce this with a pipeline job and changing its logfile

            Show
            raihaan Raihaan Shouhell added a comment - Confirm I can reproduce this with a pipeline job and changing its logfile
            Hide
            raihaan Raihaan Shouhell added a comment -

            After testing with the linked stapler PR i can no longer reproduce the issue

            Show
            raihaan Raihaan Shouhell added a comment - After testing with the linked stapler PR i can no longer reproduce the issue

              People

              Assignee:
              raihaan Raihaan Shouhell
              Reporter:
              mikeryan_tan Mike Ryan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: