Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16301

Fingerprint performance

    XMLWordPrintable

Details

    Description

      A user working with files in the 100+ Mb range asked whether fingerprinting them was expensive. Probably it is not that expensive but it is likely the implementation could be optimized and at least measured.

      As far as the I/O goes, FilePath.digest uses FileInputStream rather than NIO—hmm, maybe OK. It does not use buffering, which may or may not be an issue here. Then it calls Util.getDigestOf.

      That uses the JRE’s MD5 implementation which AFAIK takes advantage of any cryptographic hardware acceleration when available. It does read into a 1024-byte buffer however which seems excessively small. (And uses a proxy stream rather than simply calling the update method, which is odd.) http://stackoverflow.com/questions/9321912/very-slow-when-generaing-md5-using-java-with-large-file suggests using a custom library though this may be overkill; others suggest DigestUtils from Commons Codec. A

      Attachments

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18178 All Maven 2 builds fail with java.lang.NoSuchMethodError DigestUtils.md5Hex

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19515 fingerprint are truncated

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18337 Fingerprint throws exceptions on 1.518

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-13154 Heavy thread congestion with FingerPrint.save

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-11814 Maven artifact fingerprints are computed and recorded twice

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-7813 Archiving artifacts very slow

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-17412 Multiple maven builds: Jenkins frequently dies

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18351 Fingerprinting seems to leak Files

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 15 older comments
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/hudson/Util.java
            test/src/test/java/hudson/maven/MavenBuildTest.java
            http://jenkins-ci.org/commit/jenkins/e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93
            Log:
            [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds.
            Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies;
            if you specify a build extension that can override dependencies used by Jenkins core classes!
            A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/Util.java test/src/test/java/hudson/maven/MavenBuildTest.java http://jenkins-ci.org/commit/jenkins/e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93 Log: [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/FilePath.java
            core/src/main/java/hudson/Util.java
            maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            http://jenkins-ci.org/commit/jenkins/11953750ab422bcec28d12a436867d0888313aaa
            Log:
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving.
            Introducing Util.getDigestOf(File) as a convenience.

            Compare: https://github.com/jenkinsci/jenkins/compare/d18a27e5bed4...11953750ab42

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Util.java maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java http://jenkins-ci.org/commit/jenkins/11953750ab422bcec28d12a436867d0888313aaa Log: [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. Introducing Util.getDigestOf(File) as a convenience. Compare: https://github.com/jenkinsci/jenkins/compare/d18a27e5bed4...11953750ab42
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2748
            [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. (Revision e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93)
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. (Revision 11953750ab422bcec28d12a436867d0888313aaa)

            Result = SUCCESS
            Jesse Glick : e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93
            Files :

            • core/src/main/java/hudson/Util.java
            • changelog.html
            • test/src/test/java/hudson/maven/MavenBuildTest.java

            Jesse Glick : 11953750ab422bcec28d12a436867d0888313aaa
            Files :

            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            • maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java
            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            • core/src/main/java/hudson/FilePath.java
            • core/src/main/java/hudson/Util.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2748 [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. (Revision e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93) [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. (Revision 11953750ab422bcec28d12a436867d0888313aaa) Result = SUCCESS Jesse Glick : e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93 Files : core/src/main/java/hudson/Util.java changelog.html test/src/test/java/hudson/maven/MavenBuildTest.java Jesse Glick : 11953750ab422bcec28d12a436867d0888313aaa Files : maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Util.java
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/maven/ExecutedMojo.java
            src/main/java/hudson/maven/reporters/MavenArtifact.java
            src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            http://jenkins-ci.org/commit/maven-plugin/db2b69174c511a1cba440ce3683de4cd6676734b
            Log:
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving.
            Introducing Util.getDigestOf(File) as a convenience.
            Originally-Committed-As: 11953750ab422bcec28d12a436867d0888313aaa

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/maven/ExecutedMojo.java src/main/java/hudson/maven/reporters/MavenArtifact.java src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java http://jenkins-ci.org/commit/maven-plugin/db2b69174c511a1cba440ce3683de4cd6676734b Log: [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. Introducing Util.getDigestOf(File) as a convenience. Originally-Committed-As: 11953750ab422bcec28d12a436867d0888313aaa
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/maven/MavenBuildTest.java
            http://jenkins-ci.org/commit/maven-plugin/1342c5f819d6672ce53238822a93dddc087f0bd8
            Log:
            Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/maven/MavenBuildTest.java http://jenkins-ci.org/commit/maven-plugin/1342c5f819d6672ce53238822a93dddc087f0bd8 Log: Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: