Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65221

Logstash-plugin errors even tho its disabled in Jenkins configuration

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • logstash-plugin
    • None

      Our jobs are getting several of the following errors even though the Logstash config is disabled in Jenkins master configuration.

      How can I force the jobs to stop attempting to log to the logstash server?

      [logstash-plugin]: Failed to send log data: 10.192.1.25:514.
      [logstash-plugin]: No Further logs will be sent to 10.192.1.25:514.
      java.net.ConnectException: Connection timed out: connect
       at java.net.DualStackPlainSocketImpl.connect0(Native Method)
       at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
       at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:607)
       at java.net.Socket.connect(Socket.java:556)
       at java.net.Socket.<init>(Socket.java:452)
       at java.net.Socket.<init>(Socket.java:229)
       at jenkins.plugins.logstash.persistence.LogstashDao.push(LogstashDao.java:17)
       at jenkins.plugins.logstash.LogstashWriter.write(LogstashWriter.java:184)
       at jenkins.plugins.logstash.LogstashWriter.write(LogstashWriter.java:118)
       at jenkins.plugins.logstash.LogstashOutputStream.eol(LogstashOutputStream.java:64)
       at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
       at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
       at java.io.PrintStream.write(PrintStream.java:449)
       at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
       at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$NewlineSafeTaskListener$1.write(DurableTaskStep.java:454)
       at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
       at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$NewlineSafeTaskListener$1.write(DurableTaskStep.java:458)
       at java.io.PrintStream.write(PrintStream.java:480)
       at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
       at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:73)
       at hudson.remoting.ProxyOutputStream$Chunk.lambda$execute$0(ProxyOutputStream.java:259)
       at hudson.remoting.PipeWriter$1.run(PipeWriter.java:159)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
       at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
       at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
       at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19)
       at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
       at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      + grep mail:
      Cannot contact wkr_gld1vm42: java.lang.InterruptedException

          [JENKINS-65221] Logstash-plugin errors even tho its disabled in Jenkins configuration

          Temporary workaround was to re-enable it, disable the "Enable Globally" option, then disable the "Enable sending logs to an Indexer" again.

          Tylee Gooderham added a comment - Temporary workaround was to re-enable it, disable the "Enable Globally" option, then disable the "Enable sending logs to an Indexer" again.

          I'm not sure I understand. If you have "Enable Globally" option turned on then it's expected that the job will try to send the logs, right?

          Jakub Bochenski added a comment - I'm not sure I understand. If you have "Enable Globally" option turned on then it's expected that the job will try to send the logs, right?

          Hi Jakub, yes, except the "Enable sending logs to an index" is turned off, so the entire mechanism should be disabled"

           

          Tylee Gooderham added a comment - Hi Jakub, yes, except the "Enable sending logs to an index" is turned off, so the entire mechanism should be disabled"  

          Additionally, I found out something interesting... This only affected pipeline jobs. And not ALL of them. I didn't spend more time on figuring out what was specific about those particular jobs that made them unique.

          Tylee Gooderham added a comment - Additionally, I found out something interesting... This only affected pipeline jobs. And not ALL of them. I didn't spend more time on figuring out what was specific about those particular jobs that made them unique.

          Without a way to reproduce it there's not much I can do, sorry

          Jakub Bochenski added a comment - Without a way to reproduce it there's not much I can do, sorry

          Tylee Gooderham added a comment - - edited

          Its pretty straightforward for me.... 

          • Enable the indexer (click the "Enable sending logs to an Indexer" option)
          • Enable the "Enable Globally" option, so everything logs for every line of output
          • Save configuration
          • Now disable indexer, and save.

          In my environment, I have about 1000 jobs running 24x7, of which around 60 are pipelines.
          It seems these (or some subset of these) are the ones left running, after the indexer has been turned off

          If there's some logging you'd like captured I'd be happy to try to get that for you

          Tylee Gooderham added a comment - - edited Its pretty straightforward for me....  Enable the indexer (click the " Enable sending logs to an Indexer " option) Enable the " Enable Globally " option, so everything logs for every line of output Save configuration Now disable indexer, and save. In my environment, I have about 1000 jobs running 24x7, of which around 60 are pipelines. It seems these (or some subset of these) are the ones left running, after the indexer has been turned off If there's some logging you'd like captured I'd be happy to try to get that for you

          jbochenski I need to ask a silly question...
          When one of the agents pushes the payload to the index server, is it routed first thru the Jenkins server? Or is it direct?

           

          Tylee Gooderham added a comment - jbochenski  I need to ask a silly question... When one of the agents pushes the payload to the index server, is it routed first thru the Jenkins server? Or is it direct?  

          jbochenski Can this be responded to, please?

          Is there any intention on addressing this issue?

           

          Tylee Gooderham added a comment - jbochenski  Can this be responded to, please? Is there any intention on addressing this issue?  

          Well I don't use pipelines myself much and the pipeline related code was contributed by other people.

          If you can provide a minimal reproducing example I can take a look. If not, the MR https://github.com/jenkinsci/logstash-plugin/pull/93 is quite recent, you could try reverting it and checking if a modified plugin fixes your issue (if you file a MR it will be deployed to incremental repo automatically)

          Jakub Bochenski added a comment - Well I don't use pipelines myself much and the pipeline related code was contributed by other people. If you can provide a minimal reproducing example I can take a look. If not, the MR https://github.com/jenkinsci/logstash-plugin/pull/93 is quite recent, you could try reverting it and checking if a modified plugin fixes your issue (if you file a MR it will be deployed to incremental repo automatically)

            jbochenski Jakub Bochenski
            tgooderham Tylee Gooderham
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: