Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63472

Logstash "Enable Globally" causes never ending stage log loops

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • logstash-plugin
    • None
    • version 2.4.0 of plugin

      As commented on https://github.com/jenkinsci/logstash-plugin/pull/93

       

      Since we picked up 2.4.0 last week we've had some odd symptoms such as console logs for stages looping indefinitely in the master, and then spotted this stack in the slave image logs;

       

      {{WARNING: null
      java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
      at jenkins.model.Jenkins.get(Jenkins.java:777)
      at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:75)
      at jenkins.plugins.logstash.LogstashConfiguration.getInstance(LogstashConfiguration.java:289)
      at jenkins.plugins.logstash.LogstashWriter.getIndexerDao(LogstashWriter.java:162)
      at jenkins.plugins.logstash.LogstashWriter.getDaoOrNull(LogstashWriter.java:201)
      at jenkins.plugins.logstash.LogstashWriter.<init>(LogstashWriter.java:82)
      at jenkins.plugins.logstash.pipeline.GlobalDecorator.decorate(GlobalDecorator.java:41)
      at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator$DecoratedTaskListener.getLogger(TaskListenerDecorator.java:237)
      at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator$CloseableTaskListener.getLogger(TaskListenerDecorator.java:279)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:740)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      at hudson.remoting.Request$2.run(Request.java:369)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      at java.lang.Thread.run(Thread.java:748)}}

      If we turn off "Enable Globally" all is well again (except no logstash output of course )

          [JENKINS-63472] Logstash "Enable Globally" causes never ending stage log loops

          Can you try and disable the "Enable Globally" option, add the logstash plugin to one of the builds directly and see if it also fails?

          mwinter69 can you maybe take a look if time allows?

          Jakub Bochenski added a comment - Can you try and disable the "Enable Globally" option, add the logstash plugin to one of the builds directly and see if it also fails? mwinter69 can you maybe take a look if time allows?

          Markus Winter added a comment -

          This requires a major refactoring as the configuration has to be made independent of the Jenkins instance (probably with a Supplier pattern) and get injected into each run.

          Markus Winter added a comment - This requires a major refactoring as the configuration has to be made independent of the Jenkins instance (probably with a Supplier pattern) and get injected into each run.

          Steve Boardwell added a comment - - edited

          jbochenski I was trying out the logstash plugin and can confirm this still exists.

          In my case:

          • enabling globally failed
          • (global setting disabled) using the logstash block failed  
          • (global setting disabled) using the logstashSend step worked
            • however, this method does not allow logs to be send one per line

          So the following would work... 

          pipeline {
              agent any
          
              stages {
                  stage('Hello') {
                      steps {
                        //logstash {
                          echo 'Hello Logstash'
                          echo "Current result: ${currentBuild.currentResult}" 
                          sh "echo Hi from the shell"
                          logstashSend failBuild: true, maxLines: 300
                          //script { currentBuild.currentResult = 'SUCCESS' }
                        //}
                      }
                  }
              }
          }
          

          ...but this doesn't.

          pipeline {
              agent any
          
              stages {
                  stage('Hello') {
                      steps {
                        logstash {
                          echo 'Hello Logstash'
                          echo "Current result: ${currentBuild.currentResult}" 
                          sh "echo Hi from the shell"
                          //logstashSend failBuild: true, maxLines: 300
                          script { currentBuild.currentResult = 'SUCCESS' }
                        }
                      }
                  }
              }
          }
          

          Steve Boardwell added a comment - - edited jbochenski I was trying out the logstash plugin and can confirm this still exists. In my case: enabling globally failed (global setting disabled) using the logstash block failed   (global setting disabled) using the logstashSend step worked however, this method does not allow logs to be send one per line So the following would work...  pipeline { agent any stages { stage( 'Hello' ) { steps { //logstash { echo 'Hello Logstash' echo "Current result: ${currentBuild.currentResult}" sh "echo Hi from the shell" logstashSend failBuild: true , maxLines: 300 //script { currentBuild.currentResult = 'SUCCESS' } //} } } } } ...but this doesn't. pipeline { agent any stages { stage( 'Hello' ) { steps { logstash { echo 'Hello Logstash' echo "Current result: ${currentBuild.currentResult}" sh "echo Hi from the shell" //logstashSend failBuild: true , maxLines: 300 script { currentBuild.currentResult = 'SUCCESS' } } } } } }

          It seems like the change to using GlobalConfiguration instead Tool configuration (https://github.com/jenkinsci/logstash-plugin/commit/c014184142bea52430281576b3a2fdaa26d3d1bb) may have introduced the problem for pipelines but pipeline support was only added later and e.g. sh steps are not tested (tests using echo are passing)

          Steve Boardwell added a comment - It seems like the change to using GlobalConfiguration instead Tool configuration ( https://github.com/jenkinsci/logstash-plugin/commit/c014184142bea52430281576b3a2fdaa26d3d1bb ) may have introduced the problem for pipelines but pipeline support was only added later and e.g. sh steps are not tested (tests using echo are passing)

          Unfortunately I'm not using this plugin with pipelines, and the pipeline related code wasn't written by me.

          PRs to fix this are welcome, but I won't be actively working on this issue

          Jakub Bochenski added a comment - Unfortunately I'm not using this plugin with pipelines, and the pipeline related code wasn't written by me. PRs to fix this are welcome, but I won't be actively working on this issue

            Unassigned Unassigned
            admoriarty Aidan Moriarty
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: