Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73090

OOME from LineTransformationOutputStream processing CR-delimited output

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • 2.459

      A CloudBees CI user reported an OutOfMemoryError on a particular controller. Examination of the heap dump revealed multiple byte arrays in the 1Gb range containing text from running builds; much of the text was of the form of thousands of lines like

      Checking out files:  25% (2345/34567)
      

      terminated by CR (ASCII 13). These seem to have been caused by a Windows agent cloning a gigantic Git repository; I was able to confirm in Windows 10 that

      node('windows') {
          bat 'dir'
          deleteDir()
          bat 'git clone https://github.com/jenkinsci/jenkins'
      }
      

      does produce a bunch of lines of this type. In a real terminal the CR would cause the progress bar to update live on a single line. While the manual page for git-clone says that --progress is implied only when Git is connected to a terminal, and that works as claimed on Linux, I suppose this code does not work in Windows, and you actually need to pass -q.

      Currently LineTransformationOutputStream does not treat CR specially (except for purposes of trimEOL) and so it buffers all of the output until the command completes. While the lack of live progress messages is at most an annoyance, if the output is large it can lead to heap exhaustion.

          [JENKINS-73090] OOME from LineTransformationOutputStream processing CR-delimited output

          Jesse Glick created issue -
          Jesse Glick made changes -
          Description New: A CloudBees CI user reported an {{OutOfMemoryError}} on a particular controller. Examination of the heap dump revealed multiple byte arrays in the 1Gb range containing text from running builds; much of the text was of the form of thousands of lines like

          {code:none}
          Checking out files: 25% (2345/34567)
          {code}

          terminated by CR (ASCII 13). These seem to have been caused by a Windows agent cloning a gigantic Git repository; I was able to confirm in Windows 10 that

          {code:groovy}
          node('windows') {
              bat 'dir'
              deleteDir()
              bat 'git clone https://github.com/jenkinsci/jenkins'
          }
          {code}

          does produce a bunch of lines of this type. In a real terminal the CR would cause the progress bar to update live on a single line. While the manual page for {{git-clone}} says that {{\-\-progress}} is implied only when Git is connected to a terminal, and that works as claimed on Linux, I suppose this code does not work in Windows, and you actually need to pass {{\-q}}.

          Currently {{LineTransformationOutputStream}} does not treat CR specially (except for purposes of {{trimEOL}}) and so it buffers all of the output until the command completes. While the lack of live progress messages is at most an annoyance, if the output is large it can lead to heap exhaustion.
          Jesse Glick made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "jenkins #9219 (Web Link)" [ 29640 ]
          Jesse Glick made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "CloudBees-internal issue (Web Link)" [ 29646 ]
          Jesse Glick made changes -
          Released As New: 2.459
          Resolution New: Fixed [ 1 ]
          Status Original: In Review [ 10005 ] New: Fixed but Unreleased [ 10203 ]
          Alexander Brandes made changes -
          Status Original: Fixed but Unreleased [ 10203 ] New: Closed [ 6 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: