-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Powered by SuggestiMate -
4.8.1 released July 25, 2021
After the 4.8.0 upgrade, any jobs that have "Poll SCM" as the only build trigger are dumping. Jenkins LTS 2.277.4. Confirmed this behaviour on 3 different Jenkins servers. RHEL7.9 is the OS for the servers.
2021-07-22 09:42:01.162+0000 [id=4766] SEVERE h.triggers.SCMTrigger$Runner#runPolling: Failed to record SCM polling for hudson.matrix.MatrixProject@641cb4d8[job_name]
java.lang.NullPointerException
at hudson.plugins.git.util.BuildData.getLastBuild(BuildData.java:168)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:768)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:685)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:401)
at hudson.scm.SCM.poll(SCM.java:418)
at org.jenkinsci.plugins.multiplescms.MultiSCM.compareRemoteRevisionWith(MultiSCM.java:114)
at hudson.scm.SCM.poll(SCM.java:418)
at hudson.model.AbstractProject._poll(AbstractProject.java:1383)
at hudson.model.AbstractProject.poll(AbstractProject.java:1289)
at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:143)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:606)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:652)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[JENKINS-66212] Null pointer exception "h.triggers.SCMTrigger$Runner#runPolling: Failed to record SCM polling..."
A cat of one of the scm-polling.log files:
Started on Jul 23, 2021 9:48:00 AM Using strategy: Default [poll] Last Built Revision: Revision null (refs/remotes/origin/main) The recommended git tool is: NONE No credentials specified > git --version # timeout=10 > git --version # 'git version 1.8.3.1' Setting http proxy: <proxy server redacted>:80 > git ls-remote -h https://github.com/Project/repo # timeout=10 Found 3 remote heads on https://github.com/Project/repo ERROR: Failed to record SCM polling for hudson.matrix.MatrixProject@1d7042f7[job name] java.lang.NullPointerException
markewaite – Thanks, Mark. Let me know any further information I can provide. I would like to add that if I kick a build off manually, it appears the error gets resolved. Unfortunately, I don't know enough about the 47 jobs that are showing this error to be able to kick them off as a workaround and I'm loathe to figure out who the job owners are and contact them if I don't have to.
I can't duplicate the problem from any of my polling cases. I don't see any changes in that area of code that were different between 4.7.2 and 4.8.0.
I've uploaded a test build that outputs a message to report which assumption is being violated that is causing the null pointer exception. If you could install git-4.8.1-SNAPSHOT-a707434c.jpi and enable a custom logger from http://your-jenkins/log/) for level FINEST on the class hudson.plugins.git.util.BuildData, then perform a poll that usually reports a null pointer exception. Refresh the log page and it will tell us which item is null that was previously not null.
That test build will probably also resolve the null pointer exception.
Thanks Mark. I'll work on my sandbox to duplicate the issue (fingers crossed) and then install the SNAPSHOT build and enable logging. I will do this later this afternoon. Thanks again.
I have reproduced the error by copying a failing jobs config.xml to my dev server. Sadly, this server is running git plugin 4.7.2
$ grep "<version" ../../config.xml <version>2.277.4</version> $ grep Plugin-Version ../../plugins/git/META-INF/MANIFEST.MF Plugin-Version: 4.7.2 $ cat builds/1/polling.log Started on Jul 23, 2021 1:37:00 PM No existing build. Scheduling a new one. Done. Took 0 ms Changes found $ cat scm-polling.log Started on Jul 23, 2021 1:47:00 PM Using strategy: Default [poll] Last Built Revision: Revision null (refs/remotes/origin/main) The recommended git tool is: NONE No credentials specified > git --version # timeout=10 > git --version # 'git version 1.8.3.1' Setting http proxy: <redacted proxy name> > git ls-remote -h https://github.com/Project/repo # timeout=10 Found 3 remote heads on https://github.com/Project/repo ERROR: Failed to record SCM polling for hudson.matrix.MatrixProject@1b148e26[job name] java.lang.NullPointerException at hudson.plugins.git.util.BuildData.getLastBuild(BuildData.java:168) at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:764) at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:681) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:401) at hudson.scm.SCM.poll(SCM.java:418) at org.jenkinsci.plugins.multiplescms.MultiSCM.compareRemoteRevisionWith(MultiSCM.java:114) at hudson.scm.SCM.poll(SCM.java:418) at hudson.model.AbstractProject._poll(AbstractProject.java:1383) at hudson.model.AbstractProject.poll(AbstractProject.java:1289) at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:143) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:606) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:652) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
I am going to upgrade to the git-4.8.1-SNAPSHOT now on this dev server.
I am still at 2.277.4. Plugin install failed:
Failed to load: Jenkins Git plugin (4.8.1-SNAPSHOT (private-a707434c-mwaite))
- Jenkins (2.289.2) or higher required
Mark, since this problem could be related to something tangential to git-plugin (since it is also occurring on 4.7.2), I have asked one of the developers with a bunch of failing jobs to run his builds. I'll report back, but we may just want to close this ticket afterwards since the error is not reproducible for you, and I cannot get details on the failure.
I did verify that the errors started after the plugin updates I did on Wednesday. (My logs go back to Nov 2020.)
Thanks for your investigation. Sorry that I made the mistake of building the diagnostic plugin for Jenkins 2.289.2. I've built another copy of the diagnostic build that will run with 2.263.1 and later. It is attached as git-4.8.1-SNAPSHOT-6d9582a1.jpi .
Since the null pointer exception is in the git plugin, that's likely the best place to fix the null pointer exception. The diagnostics will help me understand which of the assumptions that were true previously are no longer true. If you're willing to use the diagnostic plugin with your case that duplicates the issue, it would help me know how to safeguard the code.
If you're willing to upload the config.xml of one of the failing jobs, that might also allow me to duplicate the problem.
Ugh. I just discovered that the diagnostic build I uploaded will likely fail with the same null pointer exception before it provides the diagnostics. I'm preparing a new build. You can ignore the git-4.8.1-SNAPSHOT-6d9582a1.jpi file, it is no help for isolating what has changed.
I've submitted PR-1112 that proposes a fix for the problem and provides additional diagnostics to help us understand what changed that caused the null pointer exception to become visible. The null pointer exception is a git plugin problem, so the fix makes a change to the git plugin. The build git-4.8.1-rc4427.cb4506c003ee.hpi includes the additional logging that I described earlier.
Oh nice! Thanks.
I will have to take a look at a config.xml for a failing job. I'm not sure I'll have one I can share.
I'll check into the latest build this weekend.
I've merged the change that should resolve this issue. You can pull the latest build (build 803 or later) from the master branch on ci.jenkins.io at https://ci.jenkins.io/job/Plugins/job/git-plugin/job/master/
I would still like confirmation that the change fully resolves the issue you were seeing. Since I can't duplicate the issue, I'm not 100% confident that the changes are all that are needed to resolve the problem.
I can't force either of my dev servers to throw the error.
I'll update the plugin on my production server. Do you want me to use build 803 or the git-4.8.1-r4427... that has additional logging?
Additional logging is included in both of them and is disabled by default. You'll need to enable the additional logging through the http://your-jenkins/log log configuration to enable FINEST logging for the class hudson.plugins.git.util.BuildData.
I'd prefer that you use the latest successful hpi artifact from https://ci.jenkins.io/job/Plugins/job/git-plugin/job/master/ , since that's the code that will be included in the next release of the git plugin. At the moment, that is https://ci.jenkins.io/job/Plugins/job/git-plugin/job/master/lastSuccessfulBuild/artifact/org/jenkins-ci/plugins/git/4.8.1-rc4439.d80c8e8df69d/git-4.8.1-rc4439.d80c8e8df69d.hpi
Thanks, Mark. I'll install and enable logging, and then I'll let you know what happens.
markewaite – There is joy. I upgraded to 4.8.1 and my troublesome instance is now responsive and not erroring out. Jobs are being triggered:
2021-07-26 17:41:01.711+0000 [id=782] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in <L_job>. Triggering #794 2021-07-26 17:41:01.799+0000 [id=784] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in <S_job> Triggering #496 2021-07-26 17:41:03.111+0000 [id=785] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in <SP_job>. Triggering #595 2021-07-26 17:42:01.431+0000 [id=806] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in <BIG_job> Triggering #114 2021-07-26 17:42:03.715+0000 [id=833] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in <P_job>. Triggering #246
The other three instances are working as well.
I appreciate so much your engagement, response and fix for this.
I'm glad to hear it.
I would appreciate if you could briefly enabled the FINEST logging for hudson.plugins.git.util.BuildData and capture the log output after a few polls. I'd like to understand if there are more places in the BuildData class that need to be more rigorously checked for null pointer safety.
Here you go. I monitored this for a while after installation.
Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:900a803eb3c8a015906e705bf5f4788468e197cc, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:b10a79adbdfcc30c3b7ad3b8f884f4cdaa5bf56d, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:9d8f03e55dc834d1b118be1dcfbd60fe74bd108f, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:9d8f03e55dc834d1b118be1dcfbd60fe74bd108f, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:b10a79adbdfcc30c3b7ad3b8f884f4cdaa5bf56d, returning null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:d5ba7b25784cc6c3406e5c93cc33e132c9ed1e54, returning null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:a1dbd55408a94488689834cc1ee9b698b049c19b, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:a1dbd55408a94488689834cc1ee9b698b049c19b, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:8ba0500b8d533ccd05778010f37e27ad2812d1f8, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:8ba0500b8d533ccd05778010f37e27ad2812d1f8, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:8ba0500b8d533ccd05778010f37e27ad2812d1f8, returning lastBuild Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:8ba0500b8d533ccd05778010f37e27ad2812d1f8, returning null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:56:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:476fa029f81c5fcd30c295edde8996306a4b2c50, returning null Jul 26, 2021 11:56:02 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:5339dc1f29af06492d5943846783d6cca657e61d, returning lastBuild Jul 26, 2021 11:56:02 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:8ced643058db275dd72a0d87ebd1071232bb0069, returning lastBuild Jul 26, 2021 11:56:03 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:96e60bcf25fd3867f94e4080c133d6c424fed0f8, returning lastBuild Jul 26, 2021 11:56:03 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:7afd638338475298a4a6196107e894b172705002, returning lastBuild Jul 26, 2021 11:56:04 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:0d28c21248927578c848879d0f561ccb5de12500, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:0d28c21248927578c848879d0f561ccb5de12500, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:0d28c21248927578c848879d0f561ccb5de12500, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:1f666782ffe65274885273d0a136073239c47bce, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:b10a79adbdfcc30c3b7ad3b8f884f4cdaa5bf56d, returning null Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:cd1e4bae96ccff7e7612cf2a92b23d3a748cbe37, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:b10a79adbdfcc30c3b7ad3b8f884f4cdaa5bf56d, returning null Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:a1dbd55408a94488689834cc1ee9b698b049c19b, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:a1dbd55408a94488689834cc1ee9b698b049c19b, returning lastBuild Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.revision.getSha1() is null, checking lastBuild.marked Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuild.marked.getSha1() is null Jul 26, 2021 11:57:01 AM FINEST hudson.plugins.git.util.BuildDataNo match found in getLastBuild for sha1:8ba0500b8d533ccd05778010f37e27ad2812d1f8, returning null Jul 26, 2021 11:57:02 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:d99406f2af1fb62268c34453a2fbe6bd4a7348f0, returning lastBuild Jul 26, 2021 11:57:03 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:c208d0aba97d9f7e9500a7dfb3e90a73b302399c, returning lastBuild Jul 26, 2021 11:57:03 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:46aa2ab9730b9bcf51503ff1d8fa6b1fd97900fe, returning lastBuild Jul 26, 2021 11:58:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:0d28c21248927578c848879d0f561ccb5de12500, returning lastBuild Jul 26, 2021 11:58:01 AM FINEST hudson.plugins.git.util.BuildDatalastBuildRevisionSha1 matches sha1:5339dc1f29af06492d5943846783d6cca657e61d, returning lastBuild
Let me know if you're not seeing all that you want.
That's perfect. Some code is now creating Revision objects that have a null SHA1. That's surprising and will need a little verification to confirm that the rest of the BuildData code is safe when getSha1() returns null.
Is it allowed for getSha1() to return null now? We have an NPE in a local plugin that currently expects the sha1 to be non-null.
In addition to "MatrixProject", this happens for these: