Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67602

java.lang.NullPointerException with archiveArtifacts

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None
    • Debian and Centos
      Jenkins 2.321
    • 2.340

      Our call to archiveArtifacts fails with a java.lang.NullPointerException.

      The call to archiveArtifacts  is from inside a Jenkins shared library file

      // File: libs/vars/doSomething.groovy
      archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false

       

      This is called from our pipeline script as follow:

      // File: Jenkinsfile.groovy
      pipeline {
         ...
         stage ('') {
             matrix { 
                 ...
                 stages {
                    ...
                     stage('archive stage') {
                         steps {
                             script {
                                 ... 
                                 doSomething( ... )
                             }
                         }
                     }
                 }
             }
         }
      }

      This worked for quite a long time until recently and then it started failing. 

      If I look at the Build Artifacts of the job it seems like it did archive a lot of "licenses".

      We have a preceding archiveArtifacts call that does not fail.

      For now I have wrapped the call in a try-catch block and at least it does not fail the build anymore and it reports the exception in the log:

      try {
          archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false
      }
      catch (Exception e) {
          echo 'Exception occurred with archive licenses: ' + e.toString()
      }
      

      The build log output

      [Pipeline] archiveArtifacts
      Archiving artifacts
      Recording fingerprints
      [Pipeline] echo
      Exception occurred with archive licenses: java.lang.NullPointerException
      

       

          [JENKINS-67602] java.lang.NullPointerException with archiveArtifacts

          Tim Jacomb added a comment -

          Hi no, but this will be useful for someone when they get the time

          Tim Jacomb added a comment - Hi no, but this will be useful for someone when they get the time

          Basil Crow added a comment -

          carelc I am unable to reproduce the problem with the provided .zip archive on Jenkins 2.333.

          I downloaded Jenkins 2.333 and started it against a clean home directory with java -jar jenkins.war. I then went through the wizard and installed the default suggested plugins. Then, I created the following Pipeline job:

          node {
            sh 'rm -rfv build/ licenses.zip && curl ${LICENSE_URL} --output licenses.zip'
            unzip 'licenses.zip'
            archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false
          }
          

          The job ran to completion on both Java 8 and Java 11.

          If you want help with this issue, you will need to provide steps to reproduce the issue from scratch on a minimal Jenkins installation.

          Basil Crow added a comment - carelc I am unable to reproduce the problem with the provided .zip archive on Jenkins 2.333. I downloaded Jenkins 2.333 and started it against a clean home directory with java -jar jenkins.war . I then went through the wizard and installed the default suggested plugins. Then, I created the following Pipeline job: node { sh 'rm -rfv build/ licenses.zip && curl ${LICENSE_URL} --output licenses.zip' unzip 'licenses.zip' archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false } The job ran to completion on both Java 8 and Java 11. If you want help with this issue, you will need to provide steps to reproduce the issue from scratch on a minimal Jenkins installation.

          Martin Karing added a comment -

          I got crashes when trying to checkout git and a log message in the error log referencing this issue.

          I was able to "fix" it, but it required me to delete all fingerprints (the entire directory). After doing so, everything started working again.

          Martin Karing added a comment - I got crashes when trying to checkout git and a log message in the error log referencing this issue. I was able to "fix" it, but it required me to delete all fingerprints (the entire directory). After doing so, everything started working again.

          Martin Karing added a comment -

          I checked all the fingerprint XML files and found two corrupted ones. One was just an 0 byte file, the other one was half way written. It was cut of in the middle of the <usages /> section of the file. Maybe the issue is caused during the deserialization with the facets field being initialized as null under some circumstance?

          Martin Karing added a comment - I checked all the fingerprint XML files and found two corrupted ones. One was just an 0 byte file, the other one was half way written. It was cut of in the middle of the <usages /> section of the file. Maybe the issue is caused during the deserialization with the facets field being initialized as null under some circumstance?

          Basil Crow added a comment -

          I got crashes when trying to checkout git and a log message in the error log referencing this issue.

          With what stack trace?

          Basil Crow added a comment - I got crashes when trying to checkout git and a log message in the error log referencing this issue. With what stack trace?

          Martin Karing added a comment -

          Ah sorry. Could have added this one right away.
          The pipeline did report this error:

          java.lang.NullPointerException
          	at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:151)
          	at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:140)
          	at hudson.model.Fingerprint.save(Fingerprint.java:1265)
          	at hudson.model.Fingerprint.add(Fingerprint.java:1027)
          	at hudson.model.Fingerprint.addFor(Fingerprint.java:1019)
          	at com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1456)
          	at com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1418)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:925)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:838)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1291)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	at java.base/java.lang.Thread.run(Unknown Source)
          

          There was no stack trace present in the log message referencing this issue.

          Also the Hash value of the finger print was not mentioned in the error log. Maybe adding this one would be helpful to track down the issue?

          Martin Karing added a comment - Ah sorry. Could have added this one right away. The pipeline did report this error: java.lang.NullPointerException at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:151) at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:140) at hudson.model.Fingerprint.save(Fingerprint.java:1265) at hudson.model.Fingerprint.add(Fingerprint.java:1027) at hudson.model.Fingerprint.addFor(Fingerprint.java:1019) at com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1456) at com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1418) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:925) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:838) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1291) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) There was no stack trace present in the log message referencing this issue. Also the Hash value of the finger print was not mentioned in the error log. Maybe adding this one would be helpful to track down the issue?

          Basil Crow added a comment -

          nitram I have proposed jenkinsci/jenkins#6385 to at least make the recovery process automatic. I am afraid it seems that fingerprint corruption is a long-standing issue in Jenkins.

          Basil Crow added a comment - nitram I have proposed jenkinsci/jenkins#6385 to at least make the recovery process automatic. I am afraid it seems that fingerprint corruption is a long-standing issue in Jenkins.

          I have not seen the NullPointerException, but I was looking at the fingerprint code for advice on synchronization in JENKINS-47748 and saw a reference to this issue. I did not find any obvious reason for corruption of fingerprint files.

          Kalle Niemitalo added a comment - I have not seen the NullPointerException, but I was looking at the fingerprint code for advice on synchronization in JENKINS-47748 and saw a reference to this issue. I did not find any obvious reason for corruption of fingerprint files. FileFingerprintStorage.save(Fingerprint fp, File file) uses AtomicFileWriter either directly or via XmlFile.write(Object) . That should prevent the file from being corrupted, although it might still lose an update if there are two writers. FileFingerprintStorage.save(Fingerprint fp) does synchronized (fp) , which protects against the Fingerprint instance being mutated while it is being saved. If there were multiple Fingerprint instances for the same hash, then this synchronization would not prevent them from being saved to the same file at the same time. If all Fingerprint instances are loaded or created using the methods of FingerprintMap (which extends KeyedDataStorage ), then there won't be more than one instance for the same hash. For example, Run.getBuildFingerprints() indirectly calls these. Fingerprint.load(String id) , FileFingerprintStorage.load(String id) , and FileFingerprintStorage.load(File file) create a new Fingerprint instance each time. These methods are called by FingerprintMap, but also by FileFingerprintStorage.cleanFingerprint , which then might delete a fingerprint file to which another thread is saving a separate Fingerprint instance.

          Basil Crow added a comment -

          I am going to mark this ticket as closed, since we delivered workarounds for the immediate pain points in jenkinsci/jenkins#6385 and jenkinsci/jenkins#6334. I think there remains more long-term work to be done to identify the root cause of this corruption and remove all of these workarounds. kon could you file a new ticket to track the long-term efforts, starting with the research you have begun? I think the main task would be to remove the workarounds in FileFingerprintStorage#load, which could only be done once we are sure that removing those workarounds won't re-expose the original problem.

          Basil Crow added a comment - I am going to mark this ticket as closed, since we delivered workarounds for the immediate pain points in jenkinsci/jenkins#6385 and jenkinsci/jenkins#6334 . I think there remains more long-term work to be done to identify the root cause of this corruption and remove all of these workarounds. kon could you file a new ticket to track the long-term efforts, starting with the research you have begun? I think the main task would be to remove the workarounds in FileFingerprintStorage#load , which could only be done once we are sure that removing those workarounds won't re-expose the original problem.

          I'd rather not, because I have never experienced the fingerprint file corruption.

          Kalle Niemitalo added a comment - I'd rather not, because I have never experienced the fingerprint file corruption.

            Unassigned Unassigned
            carelc Carel Combrink
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: