Although FilePath.validateAntFileMask is typically bounded to check at most 10000 files before giving up and announcing that it cannot find a match, it turns out that DirectoryScanner collects the names of non-matches in filesNotIncluded and dirsNotIncluded in the meantime. In a big workspace, this can cause the JVM (usually a slave agent) to allocate gigabytes of heap, eventually causing the slave to become overwhelmed and crash.

          [JENKINS-25759] FilePath.validateAntFileMask sucks up heap

          Jesse Glick created issue -
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Link New: This issue is blocking JENKINS-7214 [ JENKINS-7214 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "Apache bug #57253 (Web Link)" [ 11918 ]
          Jesse Glick made changes -
          Labels Original: performance New: lts-candidate performance

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3833
          [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask. (Revision 534328b264f9338e48418d2bcc0d28daaf48b3a0)

          Result = SUCCESS
          jesse glick : 534328b264f9338e48418d2bcc0d28daaf48b3a0
          Files :

          • core/src/main/java/hudson/util/FormFieldValidator.java
          • core/src/main/java/hudson/FilePath.java
          • core/src/main/java/hudson/tasks/ArtifactArchiver.java
          • core/src/test/java/hudson/FilePathTest.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3833 [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask. (Revision 534328b264f9338e48418d2bcc0d28daaf48b3a0) Result = SUCCESS jesse glick : 534328b264f9338e48418d2bcc0d28daaf48b3a0 Files : core/src/main/java/hudson/util/FormFieldValidator.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/tasks/ArtifactArchiver.java core/src/test/java/hudson/FilePathTest.java changelog.html
          Jesse Glick made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/FilePath.java
          core/src/main/java/hudson/tasks/ArtifactArchiver.java
          core/src/main/java/hudson/util/FormFieldValidator.java
          core/src/test/java/hudson/FilePathTest.java
          http://jenkins-ci.org/commit/jenkins/534328b264f9338e48418d2bcc0d28daaf48b3a0
          Log:
          [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask.
          Not fully solved, since the scannedDirs field can still grow to be large, but at least clearing files/dirsNotIncluded.
          Also imposing a 5s timeout on the scan regardless of file count, and defining a user-customizable bound.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/tasks/ArtifactArchiver.java core/src/main/java/hudson/util/FormFieldValidator.java core/src/test/java/hudson/FilePathTest.java http://jenkins-ci.org/commit/jenkins/534328b264f9338e48418d2bcc0d28daaf48b3a0 Log: [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask. Not fully solved, since the scannedDirs field can still grow to be large, but at least clearing files/dirsNotIncluded. Also imposing a 5s timeout on the scan regardless of file count, and defining a user-customizable bound.
          Oliver Gondža made changes -
          Labels Original: lts-candidate performance New: 1.580.3-fixed performance

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/FilePath.java
          core/src/main/java/hudson/tasks/ArtifactArchiver.java
          core/src/main/java/hudson/util/FormFieldValidator.java
          core/src/test/java/hudson/FilePathTest.java
          http://jenkins-ci.org/commit/jenkins/23e0d6ad1964e0efdd1e0f1c90360321acc8a073
          Log:
          [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask.
          Not fully solved, since the scannedDirs field can still grow to be large, but at least clearing files/dirsNotIncluded.
          Also imposing a 5s timeout on the scan regardless of file count, and defining a user-customizable bound.
          (cherry picked from commit 534328b264f9338e48418d2bcc0d28daaf48b3a0)

          Conflicts:
          changelog.html

          Compare: https://github.com/jenkinsci/jenkins/compare/03600e925c3d...23e0d6ad1964

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/tasks/ArtifactArchiver.java core/src/main/java/hudson/util/FormFieldValidator.java core/src/test/java/hudson/FilePathTest.java http://jenkins-ci.org/commit/jenkins/23e0d6ad1964e0efdd1e0f1c90360321acc8a073 Log: [FIXED JENKINS-25759] Avoid consuming too much memory while running validateAntFileMask. Not fully solved, since the scannedDirs field can still grow to be large, but at least clearing files/dirsNotIncluded. Also imposing a 5s timeout on the scan regardless of file count, and defining a user-customizable bound. (cherry picked from commit 534328b264f9338e48418d2bcc0d28daaf48b3a0) Conflicts: changelog.html Compare: https://github.com/jenkinsci/jenkins/compare/03600e925c3d...23e0d6ad1964

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: