Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30975

Jobs not firing; NPE sending event to GerritMissedEventsPlaybackManager

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • Jenkins 1.609.3
      gerrit-trigger 2.16.0
      Gerrit 2.11.3 w/ events-log v2.11.3-8-g318953f
      OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
      Ubuntu 14.04.3 LTS

    Description

      We installed the events-log and enabled the REST integration this morning. Almost immediately, jobs stopped firing on gerrit events. We noted high CPU usage on the Jenkins process, and looking at the javamelody monitoring we saw that 2 of the 3 GerritMissedEventsPlaybackManager receive threads were blocked. In our jenkins.log we saw entries similar to the following:

      Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
      SEVERE: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@7a9a6d56 about event: ManualPatchsetCreated Change: Change-Id for #36488: Ia70accbcf0ef07628b6ea861af546235adc2c1fb PatchSet: PatchSet: 1
      Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
      SEVERE: Notify-error:
      java.lang.NullPointerException
      at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:411)
      at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:256)
      at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:317)
      at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:285)
      at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77)
      at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
      at com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:48)
      at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
      at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)

      Looking at persist(), it looks like the passed-in Gerrit event is null or missing GetEventCreatedOn(), but that doesn't make sense.

      We've disabled REST for the time being, because this behavior effectively breaks our Jenkins deployment.

      Attachments

        Activity

          wsaxon Will Saxon created issue -
          wsaxon Will Saxon made changes -
          Field Original Value New Value
          Description We installed the events-log and enabled the REST integration this morning. Almost immediately, jobs stopped firing on gerrit events. We noted high CPU usage on the Jenkins process, and looking at the javamelody monitoring we saw that 2 of the 3 GerritMissedEventsPlaybackManager receive threads were blocked. In our jenkins.log we saw entries similar to the following:

          {{Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@7a9a6d56 about event: [ManualPatchsetCreated Change: Change-Id for #36488: Ia70accbcf0ef07628b6ea861af546235adc2c1fb PatchSet: PatchSet: 1]
          Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: Notify-error:
          java.lang.NullPointerException
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:411)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:256)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:317)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:285)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:48)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
          }}

          Looking at persist(), it looks like the passed-in Gerrit event is null or missing GetEventCreatedOn(), but that doesn't make sense.
          We installed the events-log and enabled the REST integration this morning. Almost immediately, jobs stopped firing on gerrit events. We noted high CPU usage on the Jenkins process, and looking at the javamelody monitoring we saw that 2 of the 3 GerritMissedEventsPlaybackManager receive threads were blocked. In our jenkins.log we saw entries similar to the following:

          {quote}Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@7a9a6d56 about event: [ManualPatchsetCreated Change: Change-Id for #36488: Ia70accbcf0ef07628b6ea861af546235adc2c1fb PatchSet: PatchSet: 1]
          Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: Notify-error:
          java.lang.NullPointerException
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:411)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:256)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:317)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:285)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:48)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
          {quote}

          Looking at persist(), it looks like the passed-in Gerrit event is null or missing GetEventCreatedOn(), but that doesn't make sense.
          wsaxon Will Saxon made changes -
          Description We installed the events-log and enabled the REST integration this morning. Almost immediately, jobs stopped firing on gerrit events. We noted high CPU usage on the Jenkins process, and looking at the javamelody monitoring we saw that 2 of the 3 GerritMissedEventsPlaybackManager receive threads were blocked. In our jenkins.log we saw entries similar to the following:

          {quote}Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@7a9a6d56 about event: [ManualPatchsetCreated Change: Change-Id for #36488: Ia70accbcf0ef07628b6ea861af546235adc2c1fb PatchSet: PatchSet: 1]
          Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: Notify-error:
          java.lang.NullPointerException
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:411)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:256)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:317)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:285)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:48)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
          {quote}

          Looking at persist(), it looks like the passed-in Gerrit event is null or missing GetEventCreatedOn(), but that doesn't make sense.
          We installed the events-log and enabled the REST integration this morning. Almost immediately, jobs stopped firing on gerrit events. We noted high CPU usage on the Jenkins process, and looking at the javamelody monitoring we saw that 2 of the 3 GerritMissedEventsPlaybackManager receive threads were blocked. In our jenkins.log we saw entries similar to the following:

          {quote}Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@7a9a6d56 about event: [ManualPatchsetCreated Change: Change-Id for #36488: Ia70accbcf0ef07628b6ea861af546235adc2c1fb PatchSet: PatchSet: 1]
          Oct 15, 2015 10:41:04 AM com.sonymobile.tools.gerrit.gerritevents.GerritHandler notifyListeners
          SEVERE: Notify-error:
          java.lang.NullPointerException
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:411)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:256)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:317)
                  at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:285)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:48)
                  at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
                  at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
          {quote}

          Looking at persist(), it looks like the passed-in Gerrit event is null or missing GetEventCreatedOn(), but that doesn't make sense.

          We've disabled REST for the time being, because this behavior effectively breaks our Jenkins deployment.
          scoheb Scott Hebert made changes -
          Assignee jyrkiput [ jyrkiput ] Scott Hebert [ scoheb ]
          scoheb Scott Hebert added a comment -

          Can you run a:

          ssh -p 29418 <your gerrit server hostname> gerrit stream-events

          and see if events do in fact contain eventCreatedOn attributes?

          I added protection in that method by checking if eventCreatedOn.getTime() was 0. That was supposed to handle Gerrit Servers that did not support that attribute.

          I can further protect it by doing some null checks.

          scoheb Scott Hebert added a comment - Can you run a: ssh -p 29418 <your gerrit server hostname> gerrit stream-events and see if events do in fact contain eventCreatedOn attributes? I added protection in that method by checking if eventCreatedOn.getTime() was 0. That was supposed to handle Gerrit Servers that did not support that attribute. I can further protect it by doing some null checks.
          scoheb Scott Hebert made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          scoheb Scott Hebert added a comment - Created PR https://github.com/jenkinsci/gerrit-trigger-plugin/pull/261

          Code changed in jenkins
          User: Scott Hebert
          Path:
          src/main/java/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/GerritMissedEventsPlaybackManager.java
          src/test/java/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/GerritMissedEventsLoadPersistTest.java
          src/test/resources/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/DeserializeEventCreatedOnTest.json
          http://jenkins-ci.org/commit/gerrit-trigger-plugin/7fd089691116567def282f47b8059d647e5834f9
          Log:
          Protect PlaybackManager from null EventCreatedOn

          It is possible that Gerrit might return a null eventCreated attribute for some events.

          This protects the persistence of events from this situation.

          [FIXED JENKINS-30975]

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Scott Hebert Path: src/main/java/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/GerritMissedEventsPlaybackManager.java src/test/java/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/GerritMissedEventsLoadPersistTest.java src/test/resources/com/sonyericsson/hudson/plugins/gerrit/trigger/playback/DeserializeEventCreatedOnTest.json http://jenkins-ci.org/commit/gerrit-trigger-plugin/7fd089691116567def282f47b8059d647e5834f9 Log: Protect PlaybackManager from null EventCreatedOn It is possible that Gerrit might return a null eventCreated attribute for some events. This protects the persistence of events from this situation. [FIXED JENKINS-30975]
          scm_issue_link SCM/JIRA link daemon made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 166127 ] JNJira + In-Review [ 197917 ]

          People

            scoheb Scott Hebert
            wsaxon Will Saxon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: