-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Debian and Centos
Jenkins 2.321
-
Powered by SuggestiMate -
2.340
Our call to archiveArtifacts fails with a java.lang.NullPointerException.
The call to archiveArtifacts is from inside a Jenkins shared library file
// File: libs/vars/doSomething.groovy archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false
This is called from our pipeline script as follow:
// File: Jenkinsfile.groovy pipeline { ... stage ('') { matrix { ... stages { ... stage('archive stage') { steps { script { ... doSomething( ... ) } } } } } } }
This worked for quite a long time until recently and then it started failing.
If I look at the Build Artifacts of the job it seems like it did archive a lot of "licenses".
We have a preceding archiveArtifacts call that does not fail.
For now I have wrapped the call in a try-catch block and at least it does not fail the build anymore and it reports the exception in the log:
try { archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false } catch (Exception e) { echo 'Exception occurred with archive licenses: ' + e.toString() }
The build log output
[Pipeline] archiveArtifacts Archiving artifacts Recording fingerprints [Pipeline] echo Exception occurred with archive licenses: java.lang.NullPointerException
- licenses.zip
- 7.05 MB
[JENKINS-67602] java.lang.NullPointerException with archiveArtifacts
From a quick look it doesn't seem like that should ever be null so I'm not sure what the root cause is here.
If you could provide a sample project that can reliably reproduce this it would be helpful.
Is this something you can reliably reproduce or just sometimes?
timja
It is reliable, it happens every time for every build.
I will try and see if I can reproduce but can't promise anything.
I could create a project that reproduces.
Find attached:
-
- Archive of the files that cause the archive to fail
- Created with zip zipFile: "licenses.zip", archive : true, glob: 'build/licenses/**'
- Must be uploaded to a place where the below Jenkinsfile can curl it from.
- Jenkinsfile.groovy
- The pipeline script
- NB: Replace the address of the curl command to where the zip archive can be found
The log file for the attached pipeline:
Obtained jenkins/Jenkinsfile.groovy from 2fffce729fe494ee4de288c20284ef28a92b3e36 [Pipeline] Start of Pipeline [Pipeline] stage [Pipeline] { (BuildAndDeploy) [Pipeline] parallel [Pipeline] { (Branch: Matrix - PLATFORM = 'centos8') [Pipeline] stage [Pipeline] { (Matrix - PLATFORM = 'centos8') [Pipeline] withEnv [Pipeline] { [Pipeline] node Running on slave-centos8-1 in /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce [Pipeline] { [Pipeline] stage [Pipeline] { (Get Zip) [Pipeline] sh + curl <URL HERE>/licenses.zip --output licenses.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7218k 100 7218k 0 0 587M 0 --:--:-- --:--:-- --:--:-- 587M [Pipeline] unzip Extracting from /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/licenses.zip Extracting: build/licenses/abseil/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/abseil/licenses/LICENSE Extracting: build/licenses/boost/licenses/LICENSE_1_0.txt -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/boost/licenses/LICENSE_1_0.txt Extracting: build/licenses/bzip2/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/bzip2/licenses/LICENSE Extracting: build/licenses/c-ares/licenses/LICENSE.md -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/c-ares/licenses/LICENSE.md Extracting: build/licenses/cppzmq/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/cppzmq/licenses/LICENSE Extracting: build/licenses/doctest/licenses/LICENSE.txt -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/doctest/licenses/LICENSE.txt Extracting: build/licenses/fftw_double/licenses/COPYING -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_double/licenses/COPYING Extracting: build/licenses/fftw_double/licenses/COPYRIGHT -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_double/licenses/COPYRIGHT Extracting: build/licenses/fftw_double/licenses/LICENSE.pdf -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_double/licenses/LICENSE.pdf Extracting: build/licenses/fftw_single/licenses/COPYING -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_single/licenses/COPYING Extracting: build/licenses/fftw_single/licenses/COPYRIGHT -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_single/licenses/COPYRIGHT Extracting: build/licenses/fftw_single/licenses/LICENSE.pdf -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fftw_single/licenses/LICENSE.pdf Extracting: build/licenses/flatbuffers/licenses/LICENSE.txt -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/flatbuffers/licenses/LICENSE.txt Extracting: build/licenses/flatc/licenses/LICENSE.txt -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/flatc/licenses/LICENSE.txt Extracting: build/licenses/fmt/licenses/LICENSE.rst -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/fmt/licenses/LICENSE.rst Extracting: build/licenses/grpc/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/grpc/licenses/LICENSE Extracting: build/licenses/libbacktrace/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/libbacktrace/licenses/LICENSE Extracting: build/licenses/libevent/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/libevent/licenses/LICENSE Extracting: build/licenses/openssl/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/openssl/licenses/LICENSE Extracting: build/licenses/protobuf/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/protobuf/licenses/LICENSE Extracting: build/licenses/re2/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/re2/licenses/LICENSE Extracting: build/licenses/spdlog/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/spdlog/licenses/LICENSE Extracting: build/licenses/thrift/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/thrift/licenses/LICENSE Extracting: build/licenses/zeromq/licenses/COPYING -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/zeromq/licenses/COPYING Extracting: build/licenses/zeromq/licenses/COPYING.LESSER -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/zeromq/licenses/COPYING.LESSER Extracting: build/licenses/zlib/licenses/LICENSE -> /home/jenkins/workspace/n_bugfix_jenkins_issue_reproduce/build/licenses/zlib/licenses/LICENSE Extracted: 26 files [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Package) [Pipeline] catchError [Pipeline] { [Pipeline] archiveArtifacts Archiving artifacts Recording fingerprints [Pipeline] } ERROR: Archive licenses failed java.lang.NullPointerException at hudson.model.Fingerprint.initFacets(Fingerprint.java:1418) at hudson.model.Fingerprint.load(Fingerprint.java:1361) at hudson.model.FingerprintMap.load(FingerprintMap.java:92) at hudson.model.FingerprintMap.load(FingerprintMap.java:47) at hudson.util.KeyedDataStorage.get(KeyedDataStorage.java:161) at hudson.model.FingerprintMap.get(FingerprintMap.java:82) at hudson.model.FingerprintMap.get(FingerprintMap.java:47) at hudson.util.KeyedDataStorage.getOrCreate(KeyedDataStorage.java:111) at hudson.model.FingerprintMap.getOrCreate(FingerprintMap.java:72) at hudson.tasks.Fingerprinter$Record.addRecord(Fingerprinter.java:271) at hudson.tasks.Fingerprinter.record(Fingerprinter.java:326) at hudson.tasks.Fingerprinter.perform(Fingerprinter.java:189) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:262) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:100) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:70) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [Pipeline] // catchError [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] End of Pipeline [Bitbucket] Notifying commit build result [Bitbucket] Build result notified Finished: FAILURE
- On our side the script fails on Centos 8 and Debian 9
I hope this helps in finding the issue.
PS: Since initially reporting the issue we upgraded our Jenkins controller to version 2.332. It still fails after the upgrade.
Maybe related: On the Jenkins "Manage Old Data" page we get the following entries:
Type ↓ | Name | Error |
---|---|---|
hudson.model.Fingerprint | LICENSE_1_0.txt | ConversionException: ---- Debugging information ---- cause-exception : java.lang.RuntimeException cause-message : null class : java.util.Hashtable required-type : java.util.Hashtable converter-type : hudson.util.RobustMapConverter path : /fingerprint/usages/entry[33] line number : 141 ------------------------------- |
carelc I am unable to reproduce the problem with the provided .zip archive on Jenkins 2.333.
I downloaded Jenkins 2.333 and started it against a clean home directory with java -jar jenkins.war. I then went through the wizard and installed the default suggested plugins. Then, I created the following Pipeline job:
node { sh 'rm -rfv build/ licenses.zip && curl ${LICENSE_URL} --output licenses.zip' unzip 'licenses.zip' archiveArtifacts artifacts: 'build/licenses/**', fingerprint: true, allowEmptyArchive: true, followSymlinks: false }
The job ran to completion on both Java 8 and Java 11.
If you want help with this issue, you will need to provide steps to reproduce the issue from scratch on a minimal Jenkins installation.
I got crashes when trying to checkout git and a log message in the error log referencing this issue.
I was able to "fix" it, but it required me to delete all fingerprints (the entire directory). After doing so, everything started working again.
I checked all the fingerprint XML files and found two corrupted ones. One was just an 0 byte file, the other one was half way written. It was cut of in the middle of the <usages /> section of the file. Maybe the issue is caused during the deserialization with the facets field being initialized as null under some circumstance?
I got crashes when trying to checkout git and a log message in the error log referencing this issue.
With what stack trace?
Ah sorry. Could have added this one right away.
The pipeline did report this error:
java.lang.NullPointerException at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:151) at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:140) at hudson.model.Fingerprint.save(Fingerprint.java:1265) at hudson.model.Fingerprint.add(Fingerprint.java:1027) at hudson.model.Fingerprint.addFor(Fingerprint.java:1019) at com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1456) at com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1418) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:925) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:838) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1291) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)
There was no stack trace present in the log message referencing this issue.
Also the Hash value of the finger print was not mentioned in the error log. Maybe adding this one would be helpful to track down the issue?
nitram I have proposed jenkinsci/jenkins#6385 to at least make the recovery process automatic. I am afraid it seems that fingerprint corruption is a long-standing issue in Jenkins.
I have not seen the NullPointerException, but I was looking at the fingerprint code for advice on synchronization in JENKINS-47748 and saw a reference to this issue. I did not find any obvious reason for corruption of fingerprint files.
- FileFingerprintStorage.save(Fingerprint fp, File file) uses AtomicFileWriter either directly or via XmlFile.write(Object). That should prevent the file from being corrupted, although it might still lose an update if there are two writers.
- FileFingerprintStorage.save(Fingerprint fp) does synchronized (fp), which protects against the Fingerprint instance being mutated while it is being saved. If there were multiple Fingerprint instances for the same hash, then this synchronization would not prevent them from being saved to the same file at the same time.
- If all Fingerprint instances are loaded or created using the methods of FingerprintMap (which extends KeyedDataStorage), then there won't be more than one instance for the same hash. For example, Run.getBuildFingerprints() indirectly calls these.
- Fingerprint.load(String id), FileFingerprintStorage.load(String id), and FileFingerprintStorage.load(File file) create a new Fingerprint instance each time. These methods are called by FingerprintMap, but also by FileFingerprintStorage.cleanFingerprint, which then might delete a fingerprint file to which another thread is saving a separate Fingerprint instance.
I am going to mark this ticket as closed, since we delivered workarounds for the immediate pain points in jenkinsci/jenkins#6385 and jenkinsci/jenkins#6334. I think there remains more long-term work to be done to identify the root cause of this corruption and remove all of these workarounds. kon could you file a new ticket to track the long-term efforts, starting with the research you have begun? I think the main task would be to remove the workarounds in FileFingerprintStorage#load, which could only be done once we are sure that removing those workarounds won't re-expose the original problem.
I'd rather not, because I have never experienced the fingerprint file corruption.
Instead of try-catch I used the `catchError` step and now I see the full exception in the log:
I hope this helps