Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17590

Builds fail because of "slave went offline during the build"

    XMLWordPrintable

Details

    Description

      Several times now I have builds that fail because of a "Looks like the node went offline during the build. Check the slave log for the details." message.

      That slave (a swarm slave) is still connected, but we want to reboot it's host. We have switched it to offline because it still had builds running, and we wanted to wait till they finish, but not have it accept new builds (that's the purpose of Offline, yes?)

      Of course that whole purpose is defeated if switching the slave to Offline also causes running builds to fail.

      Expected:

      • Have a way to cleanly shut down and disconnect an existing slave that has builds running, without disturbing its running builds in any way. Currently that is not possible.

      Attachments

        Issue Links

          Activity

            Steven,

            Do you have any ideas on this?

            I've used Timestamper for awhile before this... so something seems to have changed in core that brought this about.

            Thanks,
            Andy

            aje Andrew Erickson added a comment - Steven, Do you have any ideas on this? I've used Timestamper for awhile before this... so something seems to have changed in core that brought this about. Thanks, Andy
            marc_guenther Marc Günther added a comment - - edited

            I don't know if this is related to my original report. But I just tried Andrews example on Jenkins 1.501, and it also happened there, but with a different stacktrace:

            + sleep 15
            Looks like the node went offline during the build. Check the slave log for the details.FATAL: null
            java.lang.NullPointerException
            	at hudson.plugins.timestamper.TimestampFormatter.<init>(TimestampFormatter.java:71)
            	at hudson.plugins.timestamper.TimestamperConfig$1.get(TimestamperConfig.java:59)
            	at hudson.plugins.timestamper.TimestamperConfig$1.get(TimestamperConfig.java:54)
            	at hudson.plugins.timestamper.TimestamperConfig.formatter(TimestamperConfig.java:148)
            	at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.newInstance(TimestampAnnotatorFactory.java:51)
            	at hudson.console.ConsoleAnnotator._for(ConsoleAnnotator.java:143)
            	at hudson.console.ConsoleAnnotator.initial(ConsoleAnnotator.java:133)
            	at hudson.console.AnnotatedLargeText.createAnnotator(AnnotatedLargeText.java:140)
            	at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:157)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:605)
            	at hudson.model.Run.execute(Run.java:1568)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:236)
            

            In that test job, the "Add timestamps to the Console Output" is disabled.

            marc_guenther Marc Günther added a comment - - edited I don't know if this is related to my original report. But I just tried Andrews example on Jenkins 1.501, and it also happened there, but with a different stacktrace: + sleep 15 Looks like the node went offline during the build. Check the slave log for the details.FATAL: null java.lang.NullPointerException at hudson.plugins.timestamper.TimestampFormatter.<init>(TimestampFormatter.java:71) at hudson.plugins.timestamper.TimestamperConfig$1.get(TimestamperConfig.java:59) at hudson.plugins.timestamper.TimestamperConfig$1.get(TimestamperConfig.java:54) at hudson.plugins.timestamper.TimestamperConfig.formatter(TimestamperConfig.java:148) at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.newInstance(TimestampAnnotatorFactory.java:51) at hudson.console.ConsoleAnnotator._for(ConsoleAnnotator.java:143) at hudson.console.ConsoleAnnotator.initial(ConsoleAnnotator.java:133) at hudson.console.AnnotatedLargeText.createAnnotator(AnnotatedLargeText.java:140) at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:157) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:605) at hudson.model.Run.execute(Run.java:1568) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) In that test job, the "Add timestamps to the Console Output" is disabled.
            marc_guenther Marc Günther added a comment -

            Ah, my last comment was using the Plugin version 1.5.2. I just upgraded to 1.5.3, and now my stack trace looks identical to Andrews:

            Looks like the node went offline during the build. Check the slave log for the details.FATAL: null
            java.lang.NullPointerException
            	at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.getOffset(TimestampAnnotatorFactory.java:65)
            	at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.newInstance(TimestampAnnotatorFactory.java:52)
            	at hudson.console.ConsoleAnnotator._for(ConsoleAnnotator.java:143)
            	at hudson.console.ConsoleAnnotator.initial(ConsoleAnnotator.java:133)
            	at hudson.console.AnnotatedLargeText.createAnnotator(AnnotatedLargeText.java:140)
            	at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:157)
            	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:605)
            	at hudson.model.Run.execute(Run.java:1568)
            	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
            	at hudson.model.ResourceController.execute(ResourceController.java:88)
            	at hudson.model.Executor.run(Executor.java:236)
            
            marc_guenther Marc Günther added a comment - Ah, my last comment was using the Plugin version 1.5.2. I just upgraded to 1.5.3, and now my stack trace looks identical to Andrews: Looks like the node went offline during the build. Check the slave log for the details.FATAL: null java.lang.NullPointerException at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.getOffset(TimestampAnnotatorFactory.java:65) at hudson.plugins.timestamper.annotator.TimestampAnnotatorFactory.newInstance(TimestampAnnotatorFactory.java:52) at hudson.console.ConsoleAnnotator._for(ConsoleAnnotator.java:143) at hudson.console.ConsoleAnnotator.initial(ConsoleAnnotator.java:133) at hudson.console.AnnotatedLargeText.createAnnotator(AnnotatedLargeText.java:140) at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:157) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:605) at hudson.model.Run.execute(Run.java:1568) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236)

            This appears to be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-16778

            v1.5.3 of Timestamper doesn't have this fix... like the changelog makes it look like. I guess it will be in 1.5.4.

            aje Andrew Erickson added a comment - This appears to be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-16778 v1.5.3 of Timestamper doesn't have this fix... like the changelog makes it look like. I guess it will be in 1.5.4.

            duplicate, moving activity to JENKINS-16778

            aje Andrew Erickson added a comment - duplicate, moving activity to JENKINS-16778

            People

              stevengbrown Steven G Brown
              marc_guenther Marc Günther
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: