Hmm, will see if I can reproduce in a test. WorkflowRun.checkouts is not transient and is only ever set to a non-null value when the build begins, but I am guessing onLoad fails to set it, implying that this should actually be a RunAction2 or something. 81a052f dealt with a variant NPE.

Independently, GitStatus.doNotifyCommit ought to catch RuntimeException in a listener, warn, and continue.

If reproducible, this on its own merits a 1.3 release.

Was not able to reproduce. Using Linux and Java 8, I ran 1.598 from the WAR with a fresh home directory, installed Workflow plugins (1.2), added the Git plugin (2.3.5), restarted Jenkins to complete the installation. Created a new Git repository /tmp/testrepo, added one file to it and committed. Defined a job

<?xml version='1.0' encoding='UTF-8'?>
<flow-definition plugin="workflow-job@1.2">
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <definition class="org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition" plugin="workflow-cps@1.2">
    <script>node {
  git &apos;/tmp/testrepo&apos;
  sh &apos;ls&apos;
}</script>
    <sandbox>false</sandbox>
  </definition>
  <triggers>
    <hudson.triggers.SCMTrigger>
      <spec>@daily</spec>
      <ignorePostCommitHooks>false</ignorePostCommitHooks>
    </hudson.triggers.SCMTrigger>
  </triggers>
</flow-definition>

Ran a build; it checked out the repo as expected and listed the one file.

From a shell, ran

$ curl -i http://localhost:8080/jenkins/git/notifyCommit?url=/tmp/testrepo
HTTP/1.1 200 OK
Triggered: http://localhost:8080/jenkins/job/<JOBNAME>/
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 92
Server: Jetty(winstone-2.8)

Scheduled polling of <JOBNAME>
No Git consumers using SCM API plugin for: /tmp/testrepo

as expected.

Restarted Jenkins (/restart), and when it was back up, ran the same curl command again, with the same result.

Also committed a change to the Git repo and ran the notify command. Again the same curl output, and now build #2 of the job was triggered, with the expected changelog.

If you can still reproduce this problem, please attach $JENKINS_HOME/jobs/$JOBNAME/builds/lastSuccessfulBuild/build.xml. There should be a section beginning

<checkouts class="linked-list">

Your stack trace suggests that it is missing for some reason, but I am not sure how that could have happened.

See PR 57; could not reproduce from a functional test either.

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepTest.java
http://jenkins-ci.org/commit/workflow-plugin/646b3d58647012025e1f94081a7ce5be7ded65b3
Log:
JENKINS-26761 Verifying that polling still works after a restart.

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepTest.java
http://jenkins-ci.org/commit/workflow-plugin/3bc29d627efbdc3f9863f0a88e41b84aac2331e0
Log:
Merge pull request #57 from jglick/WorkflowRun.checkouts-JENKINS-26761

JENKINS-26761 Verifying that polling still works after a restart

Compare: https://github.com/jenkinsci/workflow-plugin/compare/7309ed1a6e2b...3bc29d627efb

This problem went away mysteriously and so when you could not reproduce, I let it go. Now it is back and I have dug in a little deeper. Your request to attach the build.xml for the last successful build may point to the crux of the issue: this job has four failed builds and no successful ones. In lieu of the nonexistent build.xml, here is some console output that illumines the situation:

[root@chef-jenkins-01 jenkins]# cd /var/lib/jenkins/jobs/couchbase-alpha-cluster-1/
[root@chef-jenkins-01 couchbase-alpha-cluster-1]# ll
total 12
drwxr-xr-x. 6 jenkins jenkins 4096 Mar 10 15:28 builds
-rw-r--r--. 1 jenkins jenkins  800 Feb 19 19:26 config.xml
lrwxrwxrwx. 1 jenkins jenkins   22 Mar 10 15:25 lastStable -> builds/lastStableBuild
lrwxrwxrwx. 1 jenkins jenkins   26 Mar 10 15:25 lastSuccessful -> builds/lastSuccessfulBuild
-rw-r--r--. 1 jenkins jenkins    2 Mar 10 15:25 nextBuildNumber
[root@chef-jenkins-01 couchbase-alpha-cluster-1]# ll builds/
total 16
drwxr-xr-x. 3 jenkins jenkins 4096 Feb 19 19:27 1
drwxr-xr-x. 3 jenkins jenkins 4096 Mar 10 12:54 2
drwxr-xr-x. 3 jenkins jenkins 4096 Mar 10 13:12 3
drwxr-xr-x. 3 jenkins jenkins 4096 Mar 10 15:28 4
lrwxrwxrwx. 1 jenkins jenkins    1 Mar 10 15:28 lastFailedBuild -> 4
lrwxrwxrwx. 1 jenkins jenkins    2 Feb 19 19:37 lastStableBuild -> -1
lrwxrwxrwx. 1 jenkins jenkins    2 Feb 19 19:27 lastSuccessfulBuild -> -1
lrwxrwxrwx. 1 jenkins jenkins    2 Feb 19 19:37 lastUnstableBuild -> -1
lrwxrwxrwx. 1 jenkins jenkins    1 Mar 10 15:28 lastUnsuccessfulBuild -> 4
-rw-r--r--. 1 jenkins jenkins    0 Feb 21 20:38 legacyIds

I should further note two things: First, the aforementioned behavior still holds. If I manually run the couchbase-alpha-cluster-01 job, though it still fails, the git-hook thereafter works. However, if I restart Jenkins, the problem then comes back.

Second, there may be an additional variable or more at play here in that this never-yet-succeeded job has been on this server for a few weeks w/o causing problems. I suppose it is possible that the Jenkins server has never been restarted since the job was created but that seems unlikely in our environment.

I realized that the question of lastSuccessfulBuild was spurious because what really matters is the most recent build, regardless of status. I checked all the builds and they all have the requisite linked-list entry. However, I have attached the xml for the most recent failed build. It is odd though because with a debugger attached to the server, the getSCMs method is called for several other workflow jobs before this one and they all have a non-null checkouts property for their last completed build. Its only when it gets to this, never-before-succeeded job, that the checkouts property is null. Inspecting the b variable (type WorkflowRun) for the failing case, it is indeed the last of the failed builds (build #4 in the above example), so that part looks ok. Moreover, if I run this job manually and then try the git hook, it does get past this job, cycles through several more jobs and then fails again on the next never-before-succeeded job. So this lastSuccessfulBuild question actually does seem relevant empirically.

I have made a little more progress on the issue and figured out why (superficially) the problem only re-appeared yesterday. The jobs runs that are missing their checkouts are all from yesterday. The aforementioned job which had failed 4 times, for example has 1 failed build from early February and the remaining failed builds were from yesterday. If I move the builds from yesterday out of the builds dir and restart Jenkins, the build from February is fine and checkouts. None of the builds from yesterday are good though. So I am now attaching build-1.xml as well. The only diff of significance seems to be an extra entry beneath flow-build/hudson.plugins.git.util.BuildData/buildsByBranchName and the related change to flow-build/hudson.plugins.git.util.BuildData/lastBuild. I have tried to follow what is going on in the deserialization process of this file into a WorkflowRun object but I am afraid I am lost.

I upgraded the workflow plugin to 1.3 (from 1.2) and this resolved the issue. Upgrading Jenkins to 1.602 did not resolve the issue.

Experienced the same 500 stack trace with the same repro steps. My instance is running 1.580 Jenkins Enterprise, 1.4 Workflow, and 2.3.5 Git Plugin.

kbaltrinic I am afraid I have lost you. kkelly not sure which steps to reproduce you are referring to.

If anyone can reproduce the problem from scratch, please reopen with details.

Jesse, I was referring to the repro steps described in the original description. (create and run and it works fine, restart Jenkins and /git/notifyCommit 500s until you run the job successfully manually once).

As noted on Feb 21 I tried to reproduce using the somewhat vague instructions originally given, and failed, taking note of the exact steps I used. It is possible this is only reproducible under some more specific circumstances that have not been described yet. I would certainly like to fix the issue, but I see nothing wrong by code inspection, so without a way to reproduce locally I cannot debug further.

@jglick, sorry I lost you with my comments and am not doing a good job of watching this thread. Here is the bottom line from what I can tell:

As the stacktrace shows, the problem is that b.checkouts is null on line 419 (here). (Note this seem to have moved to around line 423 on master).

I have identified two different build.xml files for the same job wherein for the one, when it hits this line, checkouts is non-null and for the other when it hits, checkouts is null. I am hoping that you can feed these two attached files into Jenkins' deserializer and figure out why the one is yielding a null checkout property when the other does not. As I recall the first file ...-build.xml is the one that yeilds the null value and ...-build-1.xml is the one that works as expected.

Hope that helps.

Well I already knew that the problems involved the checkouts field being null. The question is how this happens, since both build.xml files contain a definition for this field. That is why I want to know if there is some way to reproduce the problem from scratch, so I can actually debug it.

The only noteworthy difference I see between the two files is that JENKINS-26761-build-1.xml lacks a queueId. This field was added in 1.601, matching your comment that you upgraded to 1.602 at some point. Anyway it is unlikely that has any relationship.

I can easily suppress the exception; the underlying bug would remain, and polling would not work on this job.

Given that the problem seems to be in the deserialization of those files, I was thinking that given the files, and your knowledge of the code base, you could run them through the deserialization process and hopefully reproduce the problem. Not the case?

I tried to trace through the deserialization process with my debugger attached to the affected machine but, in the time I had, I was not able to get my head around the deserialization code sufficient to understand what was going on.

Would a comprehensive list of all plugins installed on the box help? A copy of its config.xml file? Not sure what are the relevant details.

If you are able to confirm that it is actually some aspect of the content of the build.xml files that triggers the problem, that would give me a place to start. But I doubt that is the case.

XStream is indeed complex, though it is not clear that this is actually the source of the problem anyway. That hypothesis would be strengthened if a debugger shows that Run.reload is calling unmarshal(this) on the correct build.xml with correct <checkouts> yet after the call the checkouts field remains null.

My current hypothesis is that two copies of the same WorkflowRun are getting loaded, one with all fields restored, one without, and that the webhook request is getting routed to the bad one. You can create a custom logger on hudson.model.Run recording messages at FINER, looking for the reload message to see if the same build (myproject #123) is being loaded twice (@abc123 vs. @def456).

Scratch that, turns out I can reproduce the NPE given the attached build.xml files.

Hmm, every field defined in Run is loaded, but nothing defined in WorkflowRun: checkouts and execution are both null. (logsToCopy is supposed to be null, since the build is finished.) Trying to debug why.

OK, go to http://<jenkins>/administrativeMonitor/OldData/manage and let me know if you see some errors reported there relating to this job. The fact that I could reproduce the error given your build.xml does not mean anything, because for a workflow build, build.xml cannot in general be loaded properly without some context such as the workflow/ subdirectory and so on (though to make this clearer WorkflowRun.reload needs to throw an IOException in case these fields were not successfully loaded, or XStream2.addCriticalField should be opened up for plugins).

A cleaner fix of JENKINS-27531 would be to make CpsFlowExecution.ConverterImpl.unmarshal less brittle, by just loading the XML as written and doing any follow-up work later. That might help here as well.

BTW kbaltrinic it is not a bad idea to install the Support Core plugin and attach a generated support.zip to an issue, as that lets you give the developer lots of useful information at once: Jenkins and plugin versions, logs, etc. In this case I would have been able to see any Manage Old Data errors without having to ask for them (they would show up in the file listing administrative warnings).

We have the same issue, but with using SVN instead of GIT.

After manually running the Job the SVN commit trigger started working again.

FYI, this problem has yet again disappeared for us, but I will add the loggers and take the other debug steps mentioned should it re-appear.

A user reported that this is reproducible iff multiple jobs share a clone URL, but I have not yet managed to reproduce via the existing test:

diff --git a/aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java b/aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
index bcf6da9..84276e9 100644
--- a/aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
+++ b/aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
@@ -76,6 +76,16 @@ public class GitStepRestartTest {
                 p.save();
                 WorkflowRun b = r.j.assertBuildStatusSuccess(p.scheduleBuild2(0));
                 r.j.assertLogContains("Cloning the remote Git repository", b);
+                p = r.j.jenkins.createProject(WorkflowJob.class, "p2");
+                p.addTrigger(new SCMTrigger(""));
+                p.setDefinition(new CpsFlowDefinition(
+                    "node('remote') {\n" +
+                    "    ws {\n" +
+                    "        git($/" + sampleRepo + "/$)\n" +
+                    "    }\n" +
+                    "}"));
+                p.save();
+                b = r.j.assertBuildStatusSuccess(p.scheduleBuild2(0));
             }
         });
         r.addStep(new Statement() {

I filed https://github.com/jenkinsci/workflow-plugin/pull/160 so if anyone is consistently affected by this, you may want to install a snapshot build from that and see if it helps. (Requires 1.609.x+; could be backported to 1.580.x as needed.)

Code changed in jenkins
User: Jesse Glick
Path:
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-plugin/7d21f2d187a9f8973df403b47b5efd66e6b00bd2
Log:
JENKINS-26761 Recover gracefully from null WorkflowRun.checkouts, printing some diagnostics.

Code changed in jenkins
User: Jesse Glick
Path:
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-plugin/ad9b6caf0428b9c85601ef9d349ca2691b7055be
Log:
JENKINS-26761 Try using a PersistedList rather than a LinkedList for new WorkflowRun.checkouts.
(Existing build records will continue to use LinkedList.) Possible advantages:
· If the bug was caused by an unreported failure to serialize/deserialize an SCM instance,
this would create a polite “old data” record instead.
(CopyOnWriteArrayList would also work, via RobustCollectionConverter.)
· The backing store is CopyOnWriteList, solving a hypothetical ConcurrentModificationException.
· Additions from onCheckout should be saved to disk immediately, rather than waiting for some other save trigger.

Code changed in jenkins
User: Jesse Glick
Path:
CHANGES.md
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-plugin/f6f21ae0f0bf0d7970b3ae2372b1501717938ec4
Log:
JENKINS-26761 Merging #160.

Compare: https://github.com/jenkinsci/workflow-plugin/compare/121889f5cf4c...f6f21ae0f0bf

Code changed in jenkins
User: Jesse Glick
Path:
CHANGES.md
http://jenkins-ci.org/commit/workflow-plugin/fe932961dc7bae5f3aad2930fdf61642ef317ec5
Log:
JENKINS-26761 Noting backport to 1.4.1.

Should this still be open?

Yes, because the underlying problem was never resolved. The NPE is gone, but there is still something seriously wrong during deserialization of which the NPE was just a symptom.

For what its worth we just got bit by this again on a machine w/an older version for workflow. As we now have Cloudbees support I opened an internal ticket (ironically failing to recognize the problem as being this one ), and that ticket has an attached support bundle in case it helps run down the problem. See Cloudbees ZenDesk ticket #32944.

I suspect this is actually just one of many symptoms of JENKINS-22767. If so, it should quietly disappear in 1.646+.

Closing therefore.

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/scm/GitStepRestartTest.java
http://jenkins-ci.org/commit/workflow-scm-step-plugin/9c10531b3956312f7463a7d66ce45ad2b3f5c3fd
Log:
JENKINS-26761 Verifying that polling still works after a restart.
Originally-Committed-As: 646b3d58647012025e1f94081a7ce5be7ded65b3

Code changed in jenkins
User: Jesse Glick
Path:
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-job-plugin/34802c5bfbccdf481794dce9409deeb2dfcfa9b1
Log:
JENKINS-26761 Recover gracefully from null WorkflowRun.checkouts, printing some diagnostics.
Originally-Committed-As: 7d21f2d187a9f8973df403b47b5efd66e6b00bd2

Code changed in jenkins
User: Jesse Glick
Path:
job/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
http://jenkins-ci.org/commit/workflow-job-plugin/c1eb2e285c8264bf05dfc9b64ba1750c339818fe
Log:
JENKINS-26761 Try using a PersistedList rather than a LinkedList for new WorkflowRun.checkouts.
(Existing build records will continue to use LinkedList.) Possible advantages:
· If the bug was caused by an unreported failure to serialize/deserialize an SCM instance,
this would create a polite “old data” record instead.
(CopyOnWriteArrayList would also work, via RobustCollectionConverter.)
· The backing store is CopyOnWriteList, solving a hypothetical ConcurrentModificationException.
· Additions from onCheckout should be saved to disk immediately, rather than waiting for some other save trigger.
Originally-Committed-As: ad9b6caf0428b9c85601ef9d349ca2691b7055be