Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54128

Deprecated calls to Run.getLogFile

    XMLWordPrintable

Details

    • workflow-job 2.29 groovy-postbuild-2.5

    Description

      Timestamper causes lots of WARNING messages in the log:

      Oct 17, 2018 8:19:32 AM org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFile
      WARNING: Avoid calling getLogFile on **** #3
      java.lang.UnsupportedOperationException
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1064)
      at hudson.plugins.timestamper.io.LogFileReader.nextLine(LogFileReader.java:157)
      at hudson.plugins.timestamper.action.TimestampsActionOutput$1.readNextLine(TimestampsActionOutput.java:173)
      at hudson.plugins.timestamper.action.TimestampsActionOutput$1.read(TimestampsActionOutput.java:132)
      at java.io.BufferedReader.fill(BufferedReader.java:161)
      at java.io.BufferedReader.readLine(BufferedReader.java:324)
      at java.io.BufferedReader.readLine(BufferedReader.java:389)
      at hudson.plugins.timestamper.action.TimestampsAction.doIndex(TimestampsAction.java:102)
      at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
      at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
      at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
      at org.kohsuke.stapler.IndexDispatcher.dispatch(IndexDispatcher.java:27)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:212)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:253)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:253)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:253)
      at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:124)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.Server.handle(Server.java:503)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
      at java.lang.Thread.run(Thread.java:748)
      zebra-cd-preprod-jenkins-587d4f6d59-k7cv4 zebra-cd-preprod-jenkins

      Installed plugins:

      scm-api:2.2.8
      docker-workflow:1.17
      bouncycastle-api:2.17
      parameter-separator:1.0
      script-security:1.46
      ssh-agent:1.17
      htmlpublisher:1.17
      kubernetes:1.12.7
      workflow-aggregator:2.6
      jquery:1.12.4-0
      plain-credentials:1.4
      jackson2-api:2.8.11.3
      greenballs:1.15
      pipeline-utility-steps:2.1.0
      pipeline-stage-tags-metadata:1.3.2
      jquery-detached:1.2.1
      timestamper:1.8.10
      kubernetes-credentials:0.4.0
      pipeline-milestone-step:1.3.1
      workflow-api:2.30
      ansicolor:0.5.2
      authentication-tokens:1.3
      pipeline-graph-analysis:1.7
      branch-api:2.0.20
      variant:1.1
      workflow-step-api:2.16
      matrix-auth:2.3
      mattermost:2.5.2
      workflow-cps-global-lib:2.12
      ldap:1.20
      pipeline-build-step:2.7
      apache-httpcomponents-client-4-api:4.5.5-3.0
      structs:1.17
      git:3.9.1
      pipeline-stage-step:2.3
      workflow-support:2.21
      antisamy-markup-formatter:1.5
      workflow-basic-steps:2.11
      workflow-cps:2.58
      pipeline-model-extensions:1.3.2
      pipeline-rest-api:2.10
      gitlab-plugin:1.5.10
      docker-commons:1.13
      workflow-multibranch:2.20
      uno-choice:2.1
      ssh-credentials:1.14
      pipeline-input-step:2.8
      credentials:2.1.18
      workflow-durable-task-step:2.22
      pipeline-model-declarative-agent:1.1.1
      token-macro:2.5
      momentjs:1.1.1
      pipeline-model-definition:1.3.2
      git-client:2.7.3
      git-server:1.7
      durable-task:1.26
      ace-editor:1.1
      ssh-slaves:1.28.1
      jsch:0.1.54.2
      performance:3.12
      pipeline-model-api:1.3.2
      workflow-job:2.26
      command-launcher:1.2
      cloudbees-folder:6.6
      display-url-api:2.2.0
      rebuild:1.29
      locale:1.3
      credentials-binding:1.16
      mailer:1.21
      matrix-project:1.13
      workflow-scm-step:2.7
      collapsing-console-sections:1.7.0
      job-dsl:1.70
      handlebars:1.1.1
      junit:1.26.1
      pipeline-stage-view:2.10
      configuration-as-code:1.1

      Jenkins version: 2.147

      Attachments

        Issue Links

          Activity

            FYI, I can't reproduce it at the moment...

            aroq Alexander Tolstikov added a comment - FYI, I can't reproduce it at the moment...
            medianick Nick Jones added a comment -

            I'm getting this – or something very similar to it – myself, running a pre-release build of Timestamper 1.18.11 (from JENKINS-54081):

            Avoid calling getLogFile on **** #31
            java.lang.UnsupportedOperationException
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1064)
            	at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:371)
            	at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:312)
            	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
            	at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
            	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
            	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
            	at hudson.security.ACL.impersonate(ACL.java:290)
            	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

            I don't think it's actually Timestamper, but rather the changes in Pipeline Job 2.26.

            medianick Nick Jones added a comment - I'm getting this – or something very similar to it – myself, running a pre-release build of Timestamper 1.18.11 (from JENKINS-54081 ): Avoid calling getLogFile on **** #31 java.lang.UnsupportedOperationException at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1064) at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:371) at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:312) at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50) at hudson.security.ACL.impersonate(ACL.java:290) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) I don't think it's actually Timestamper, but rather the changes in Pipeline Job 2.26.
            markewaite Mark Waite added a comment - - edited

            I'm seeing this as well in the Jenkins log. It seems to be an intentional message from workflow-job-plugin.

            Now I'll need to reconsider how I test specific bugs with Pipeline jobs. I've used the groovy post build plugin to read the logfile from the manager object, but that is apparently no longer a preferred action.

            Any hints of the preferred replacement for getLogFile()? That message was added to workflow-job-plugin 2 years ago. I'm a little ashamed I didn't detect the message sooner, though looking at the tags which include that commit, it seems to have first appeared in workflow-job-plugin 2.26, not in any of the preceding plugin releases. Maybe I'm not as inattentive as I thought...

            markewaite Mark Waite added a comment - - edited I'm seeing this as well in the Jenkins log. It seems to be an intentional message from workflow-job-plugin. Now I'll need to reconsider how I test specific bugs with Pipeline jobs. I've used the groovy post build plugin to read the logfile from the manager object, but that is apparently no longer a preferred action. Any hints of the preferred replacement for getLogFile()? That message was added to workflow-job-plugin 2 years ago . I'm a little ashamed I didn't detect the message sooner, though looking at the tags which include that commit, it seems to have first appeared in workflow-job-plugin 2.26, not in any of the preceding plugin releases. Maybe I'm not as inattentive as I thought...

            I have also recently started seeing this and am also using the manager object to read the log file.

            lostinberlin Steve Boardwell added a comment - I have also recently started seeing this and am also using the manager object to read the log file.
            dnusbaum Devin Nusbaum added a comment -

            That message was added to workflow-job-plugin 2 years ago .... it seems to have first appeared in workflow-job-plugin 2.26, not in any of the preceding plugin releases.

            markewaite The message was added in this PR sometime in its 2-year lifespan, but only recently merged and released in workflow-job 2.26. The log-handling rewrite for JEP-210 has changed the nature of various log-related methods. Previously, `getLogFile` always returned a reference to a file that already existed on disk, but now with external logging, that call has to read the entire log (which may involve network calls) and write it to a temporary file which is then returned. The best alternative that I know of would be to call WorkflowRun#getLogText and then handle it directly, but depending on your use case, maybe WorkflowRun#getLogReader or WorkflowRun#getLog(int) would be useful as well.

            dnusbaum Devin Nusbaum added a comment - That message was added to workflow-job-plugin 2 years ago .... it seems to have first appeared in workflow-job-plugin 2.26, not in any of the preceding plugin releases. markewaite The message was added in this PR sometime in its 2-year lifespan, but only recently merged and released in workflow-job 2.26. The log-handling rewrite for JEP-210 has changed the nature of various log-related methods. Previously, `getLogFile` always returned a reference to a file that already existed on disk, but now with external logging, that call has to read the entire log (which may involve network calls) and write it to a temporary file which is then returned. The best alternative that I know of would be to call WorkflowRun#getLogText and then handle it directly, but depending on your use case, maybe WorkflowRun#getLogReader or WorkflowRun#getLog(int) would be useful as well.
            jglick Jesse Glick added a comment -

            For the timestamper case, this warning is displayed only if you use the View as plain text link—which AFAIK never worked for Pipeline builds anyway. I checked whether it was straightforward to suppress this link in the patch for JENKINS-48344, but unfortunately it is not, due to the JavaScript tricks involved.

            The call from WarningsPublisher.parseConsoleLog is unrelated.

            jglick Jesse Glick added a comment - For the timestamper case, this warning is displayed only if you use the View as plain text link—which AFAIK never worked for Pipeline builds anyway. I checked whether it was straightforward to suppress this link in the patch for JENKINS-48344 , but unfortunately it is not, due to the JavaScript tricks involved. The call from WarningsPublisher.parseConsoleLog is unrelated.
            jglick Jesse Glick added a comment -

            As are calls from GroovyPostbuildRecorder.

            jglick Jesse Glick added a comment - As are calls from GroovyPostbuildRecorder .
            markewaite Mark Waite added a comment - - edited

            jglick I don't quite understand the comment where you said:

            The call from WarningsPublisher.parseConsoleLog is unrelated.

            As are calls from GroovyPostbuildRecorder.

            I believe I'm using the GroovyPostbuildRecorder when I use the manage object provided by the groovy post build plugin to call logContains(). Should I be using something else, or just accept the extra message in the Jenkins log and hope for a fix from the groovy post build plugin?

            markewaite Mark Waite added a comment - - edited jglick I don't quite understand the comment where you said: The call from WarningsPublisher.parseConsoleLog is unrelated. As are calls from GroovyPostbuildRecorder. I believe I'm using the GroovyPostbuildRecorder when I use the manage object provided by the groovy post build plugin to call logContains() . Should I be using something else, or just accept the extra message in the Jenkins log and hope for a fix from the groovy post build plugin?
            jglick Jesse Glick added a comment -

            just accept the extra message in the Jenkins log and hope for a fix from the groovy post build plugin

            This.

            jglick Jesse Glick added a comment - just accept the extra message in the Jenkins log and hope for a fix from the groovy post build plugin This.
            medianick Nick Jones added a comment -

            I've added the linenumbers-plugin here, as it too has this issue now that workflow-job 2.26 is released. In my case, it seems to generate hundreds to thousands of copies of the original log file, all named with "deprecated" (e.g., deprecated5973622952721928030.log). I haven't figured out what causes the particular quantity of these files (I've seen over 6000 created by a single build), but it seems perhaps related to how the build was executed – triggering a build directly generates 1, but triggering a build from another build (in my case, a Pipeline build that spawns a number of builds to do a regression test) generates hundreds to thousands of copies. These all coincide with log messages in the Jenkins log like:

            java.lang.UnsupportedOperationException
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1082)
            	at org.jenkinsci.plugins.linenumbers.LineNumbersAnnotator.annotate(LineNumbersAnnotator.java:54)
            	at hudson.console.ConsoleAnnotator$1Aggregator.annotate(ConsoleAnnotator.java:114)
            	at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:145)
            	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
            	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
            	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
            

            Given the impact to the master node to store this number of files (6000 files = 14GB for a single build, in my case), and unless I know it's something other than this bug that causes it, I'd think it makes sense to raise to Major (if not Blocker), if the affected plugins don't have their own issues tracking this problem.

            medianick Nick Jones added a comment - I've added the linenumbers-plugin here, as it too has this issue now that workflow-job 2.26 is released. In my case, it seems to generate hundreds to thousands of copies of the original log file, all named with "deprecated" (e.g., deprecated5973622952721928030.log). I haven't figured out what causes the particular quantity of these files (I've seen over 6000 created by a single build), but it seems perhaps related to how the build was executed – triggering a build directly generates 1, but triggering a build from another build (in my case, a Pipeline build that spawns a number of builds to do a regression test) generates hundreds to thousands of copies. These all coincide with log messages in the Jenkins log like: java.lang.UnsupportedOperationException at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1082) at org.jenkinsci.plugins.linenumbers.LineNumbersAnnotator.annotate(LineNumbersAnnotator.java:54) at hudson.console.ConsoleAnnotator$1Aggregator.annotate(ConsoleAnnotator.java:114) at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:145) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at java.io.FilterOutputStream.write(FilterOutputStream.java:77) Given the impact to the master node to store this number of files (6000 files = 14GB for a single build, in my case), and unless I know it's something other than this bug that causes it, I'd think it makes sense to raise to Major (if not Blocker), if the affected plugins don't have their own issues tracking this problem.
            medianick Nick Jones added a comment -

            Sure enough, uninstalling the linenumbers-plugin and rerunning my problematic build generated just 1 "deprecated*.log" file, and I have only one "Avoid calling getLogFile" error in the logs, which comes from hudson.plugins.warnings.WarningsPublisher.parseConsoleLog.

            medianick Nick Jones added a comment - Sure enough, uninstalling the linenumbers-plugin and rerunning my problematic build generated just 1 "deprecated*.log" file, and I have only one "Avoid calling getLogFile" error in the logs, which comes from hudson.plugins.warnings.WarningsPublisher.parseConsoleLog.
            jglick Jesse Glick added a comment -

            That one could be solved with a simple s/getLogFile/getLogText/ I think.

             I can look at having WorkflowRun avoid creating multiple temp files like this.

            jglick Jesse Glick added a comment - That one could be solved with a simple s/ getLogFile / getLogText / I think.  I can look at having WorkflowRun avoid creating multiple temp files like this.
            dnusbaum Devin Nusbaum added a comment - - edited

            Pipeline Job Plugin version 2.29 was just released with a fix that prevents a new temporary file from being created each time WorkflowRun#getLogFile is called. The warning in the logs will need to be fixed by the plugins that are calling Run#getLogFile themselves, likely by switching to using Run#getLogText instead..

            dnusbaum Devin Nusbaum added a comment - - edited Pipeline Job Plugin version 2.29 was just released with a fix that prevents a new temporary file from being created each time WorkflowRun#getLogFile is called. The warning in the logs will need to be fixed by the plugins that are calling Run#getLogFile themselves, likely by switching to using Run#getLogText instead..
            jglick Jesse Glick added a comment -

            Best to leave this open at least until distinct issues are filed for such plugins.

            jglick Jesse Glick added a comment - Best to leave this open at least until distinct issues are filed for such plugins.
            drulli Ulli Hafner added a comment - - edited

            Should Run#getLogText be used for pipelines and freestyle jobs? Or just for pipelines?

            Are there some recommendation on how to process the AnnotatedLargeText instance? In the warnings plugin I need to scan the log line by line with a parser. One way I see is to use writeLogTo and write the output stream to an intermediate file. Or is this not recommended?

            drulli Ulli Hafner added a comment - - edited Should Run#getLogText be used for pipelines and freestyle jobs? Or just for pipelines? Are there some recommendation on how to process the AnnotatedLargeText instance? In the warnings plugin I need to scan the log line by line with a parser. One way I see is to use writeLogTo and write the output stream to an intermediate file. Or is this not recommended?
            jglick Jesse Glick added a comment -

            You may use Run.getLogText, or really anything besides getLogFile, on any build, Pipeline or freestyle. In your case the simplest fix looks to be to just call Run.getLogReader rather than using ParserRegistry.createReader(File).

            jglick Jesse Glick added a comment - You may use Run.getLogText , or really anything besides getLogFile , on any build, Pipeline or freestyle. In your case the simplest fix looks to be to just call Run.getLogReader rather than using ParserRegistry.createReader(File) .
            drulli Ulli Hafner added a comment -

            I see, that looks quite easy. Is there a direct way to access the log on the agent? The I could run all warnings parsers on the agent.

            drulli Ulli Hafner added a comment - I see, that looks quite easy. Is there a direct way to access the log on the agent? The I could run all warnings parsers on the agent.
            jglick Jesse Glick added a comment -

            Is there a direct way to access the log on the agent?

            No. If you want to run warning parsers on the agent side, what you would rather want is a block-scoped step (perhaps via SimpleBuildWrapper) that uses a ConsoleLogFilter to track warnings as they are emitted, rather than parsing them later. (This would also be far more efficient in conjunction with cloud log storage.) But it is a significant rewrite to do that, and of course requires a different kind of project configuration as well.

            jglick Jesse Glick added a comment - Is there a direct way to access the log on the agent? No. If you want to run warning parsers on the agent side, what you would rather want is a block-scoped step (perhaps via SimpleBuildWrapper ) that uses a ConsoleLogFilter to track warnings as they are emitted, rather than parsing them later. (This would also be far more efficient in conjunction with cloud log storage.) But it is a significant rewrite to do that, and of course requires a different kind of project configuration as well.
            drulli Ulli Hafner added a comment -

            I removed the component 'warnings' since I fixed it in the beta of the warnings-ng plugin now.

            drulli Ulli Hafner added a comment - I removed the component 'warnings' since I fixed it in the beta of the warnings-ng plugin now.

            FTR we're having this issue on XWiki. Every day we get 1TB (yes, terabyte not gigabyte!) of jenkins.log files because of the following filling the logs:

            May 21, 2019 8:09:53 AM org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFile
            WARNING: Avoid calling getLogFile on XWiki/xwiki-platform/master #1822
            java.lang.UnsupportedOperationException
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1087)
            	at org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager.logContains(GroovyPostbuildRecorder.java:257)
            

            I'll try replace our call to logContains() and reimplement it using Run.getLogText or Run.getLogReader in the meantime.

            vmassol Vincent Massol added a comment - FTR we're having this issue on XWiki. Every day we get 1TB (yes, terabyte not gigabyte!) of jenkins.log files because of the following filling the logs: May 21, 2019 8:09:53 AM org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFile WARNING: Avoid calling getLogFile on XWiki/xwiki-platform/master #1822 java.lang.UnsupportedOperationException at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1087) at org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager.logContains(GroovyPostbuildRecorder.java:257) I'll try replace our call to logContains() and reimplement it using Run.getLogText or Run.getLogReader in the meantime.
            vmassol Vincent Massol added a comment - - edited

            Any idea why the following pipeline:

            node() {
                echo "test"
                def reader = currentBuild.rawBuild.getLogReader()
                reader.eachLine() { line ->
                    echo "Line: ${line}"
                    if (line.matches(".*test.*")) {
                        echo "got it!"
                    }
                }
            }
            

            Only prints the first log line:

            Started by user Vincent Massol
            Running in Durability level: MAX_SURVIVABILITY
            [Pipeline] Start of Pipeline
            [Pipeline] node
            Running on Jenkins in /var/jenkins_home/workspace/Test
            [Pipeline] {
            [Pipeline] echo
            test
            [Pipeline] echo
            Line: Started by user Vincent Massol
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            Finished: SUCCESS
            

            ?

            Thanks

            EDIT: see https://gitter.im/jenkinsci/jenkins?at=5ce3c760ad024978c6026d6f Seems it could be related to JENKINS-46988

            vmassol Vincent Massol added a comment - - edited Any idea why the following pipeline: node() { echo "test" def reader = currentBuild.rawBuild.getLogReader() reader.eachLine() { line -> echo "Line: ${line}" if (line.matches( ".*test.*" )) { echo "got it!" } } } Only prints the first log line: Started by user Vincent Massol Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/jenkins_home/workspace/Test [Pipeline] { [Pipeline] echo test [Pipeline] echo Line: Started by user Vincent Massol [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS ? Thanks EDIT: see https://gitter.im/jenkinsci/jenkins?at=5ce3c760ad024978c6026d6f Seems it could be related to JENKINS-46988

            ok works with:

                new BufferedReader(currentBuild.rawBuild.getLogReader()).with { br ->
                    def line = null
                    line = br.readLine()
                    while (line != null) {
                        ...
                        line = br.readLine()
                    }
                }
            
            vmassol Vincent Massol added a comment - ok works with: new BufferedReader(currentBuild.rawBuild.getLogReader()).with { br -> def line = null line = br.readLine() while (line != null ) { ... line = br.readLine() } }
            vmassol Vincent Massol added a comment - - edited

            jglick I'd love your help on this. I reimplemented BadgeManager#logContains() in our pipeline library in Groovy, since BadgeManager#logContains() calls Run#getLogFile() which emits the warnings in the jenkins logs. The only change I did was to use Run#getLogReader() as in:

            private def logContains(regexp)
            {
                return contains(currentBuild.rawBuild.getLogReader(), regexp)
            }
            
            private def contains(reader, regexp)
            {
                def matcher = getMatcher(reader, regexp)
                return (matcher != null) && matcher.matches()
            }
            
            private def getMatcher(reader, regexp)
            {
                def matcher = null
                new BufferedReader(reader).with { br ->
                    def pattern = Pattern.compile(regexp)
                    def line = null
                    line = br.readLine()
                    while (line != null) {
                        def m = pattern.matcher(line);
                        if (m.matches()) {
                            matcher = m
                            break
                        }
                        line = br.readLine()
                    }
                }
                return matcher
            }
            

            Source here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/63df1a4a1a4f75e47b97803da614f7fc9db88c39/vars/checkForFalsePositives.groov

            However, when we execute it, it's extra slow.... as the log grows in size, it takes substantially more and more time (we run 24 maven builds in a single pipeline job). On the 24th Maven build, it takes 24mn to execute (so 300mn = 5 hours in total) whereas it was taking 28s with BadgeManager#logContains(). I can't figure out why that speed change. Could it be simply because the code logic was moved from Java to Groovy? Or is there something that makes Run#getLogReader() slow?

            Thanks a lot for any help on this, I'm struggling to find a workaround (see my other message above, the warnings fills out logs at a rate of over 1TB of logs every day...).

            vmassol Vincent Massol added a comment - - edited jglick I'd love your help on this. I reimplemented BadgeManager#logContains() in our pipeline library in Groovy, since BadgeManager#logContains() calls Run#getLogFile() which emits the warnings in the jenkins logs. The only change I did was to use Run#getLogReader() as in: private def logContains(regexp) { return contains(currentBuild.rawBuild.getLogReader(), regexp) } private def contains(reader, regexp) { def matcher = getMatcher(reader, regexp) return (matcher != null ) && matcher.matches() } private def getMatcher(reader, regexp) { def matcher = null new BufferedReader(reader).with { br -> def pattern = Pattern.compile(regexp) def line = null line = br.readLine() while (line != null ) { def m = pattern.matcher(line); if (m.matches()) { matcher = m break } line = br.readLine() } } return matcher } Source here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/63df1a4a1a4f75e47b97803da614f7fc9db88c39/vars/checkForFalsePositives.groov However, when we execute it, it's extra slow.... as the log grows in size, it takes substantially more and more time (we run 24 maven builds in a single pipeline job). On the 24th Maven build, it takes 24mn to execute (so 300mn = 5 hours in total) whereas it was taking 28s with BadgeManager#logContains(). I can't figure out why that speed change. Could it be simply because the code logic was moved from Java to Groovy? Or is there something that makes Run#getLogReader() slow? Thanks a lot for any help on this, I'm struggling to find a workaround (see my other message above, the warnings fills out logs at a rate of over 1TB of logs every day...).
            jglick Jesse Glick added a comment -

            All of these methods ought to be marked @NonCPS.

            jglick Jesse Glick added a comment - All of these methods ought to be marked @NonCPS .

            All of these methods ought to be marked @NonCPS.

            jglick hmm, are you saying that the performance issues I see are actually caused by the fact that I didn't mark them @NonCPS? Indeed, didn't think of this. I'm surprised it would make such a difference but maybe that's the case.

            vmassol Vincent Massol added a comment - All of these methods ought to be marked @NonCPS. jglick hmm, are you saying that the performance issues I see are actually caused by the fact that I didn't mark them @NonCPS ? Indeed, didn't think of this. I'm surprised it would make such a difference but maybe that's the case.
            jglick Jesse Glick added a comment -

            I'm surprised it would make such a difference

            Orders of magnitude. CPS-transformed code is intended for “glue”: a handful of if-statements and the like. Not actual programming.

            jglick Jesse Glick added a comment - I'm surprised it would make such a difference Orders of magnitude. CPS-transformed code is intended for “glue”: a handful of if -statements and the like. Not actual programming.
            ikedam ikedam added a comment -

            The issue in groovy-postbuild is fixed in groovy-postbuild-2.5.
            (Removed groovy-postbuild from component/s)
            It will be available in the update center in a day.
            Please try that.

            ikedam ikedam added a comment - The issue in groovy-postbuild is fixed in groovy-postbuild-2.5. (Removed groovy-postbuild from component/s) It will be available in the update center in a day. Please try that.

            Awesome, thanks for both of you jglick and ikedam!

            vmassol Vincent Massol added a comment - Awesome, thanks for both of you jglick and ikedam !
            markewaite Mark Waite added a comment -

            Thanks very much ikedam and jglick. I'm now using the groovy-postbuild plugin 2.5 and am not seeing any log messages for my use of the groovy post build plugin to assert that specific strings are detected in the log file of the current build.

            markewaite Mark Waite added a comment - Thanks very much ikedam and jglick . I'm now using the groovy-postbuild plugin 2.5 and am not seeing any log messages for my use of the groovy post build plugin to assert that specific strings are detected in the log file of the current build.

            Getting similar exceptions in the log:

             

            Oct 24, 2019 8:32:17 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFileAvoid calling getLogFile on build-packer #8
            java.lang.UnsupportedOperationException
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1088)
            	at org.jenkinsci.plugins.compressbuildlog.BuildLogCompressor$CompressBuildlogRunListener.onFinalized(BuildLogCompressor.java:85)
            	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
            	at hudson.model.Run.onEndBuilding(Run.java:2000)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:612)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:133)
            	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:999)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
            	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
            	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
            	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
            	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            
            Oct 24, 2019 8:34:35 PM INFO hudson.model.AsyncPeriodicWork$1 run
            

             

             

            Jenkins version: 2.201

            I don't have groovy-postbuild plugin installed.

            Plugin versions.

             

            docker-workflow  1.21
            workflow-aggregator 2.6
            workflow-api 2.37
            workflow-basic-steps 2.18
            workflow-cps 2.74
            workflow-cps-global-lib 2.15
            workflow-durable-task-step 2.34
            workflow-job 2.35
            workflow-multibranch 2.21
            workflow-remote-loader 1.5
            workflow-scm-step 2.9
            workflow-step-api 2.20
            workflow-support 3.3
            

            Any help would much appreciated.

             

             

            cosbug Constantin Bugneac added a comment - Getting similar exceptions in the log:   Oct 24, 2019 8:32:17 PM WARNING org.jenkinsci.plugins.workflow.job.WorkflowRun getLogFileAvoid calling getLogFile on build-packer #8 java.lang.UnsupportedOperationException at org.jenkinsci.plugins.workflow.job.WorkflowRun.getLogFile(WorkflowRun.java:1088) at org.jenkinsci.plugins.compressbuildlog.BuildLogCompressor$CompressBuildlogRunListener.onFinalized(BuildLogCompressor.java:85) at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255) at hudson.model.Run.onEndBuilding(Run.java:2000) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:612) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:999) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Oct 24, 2019 8:34:35 PM INFO hudson.model.AsyncPeriodicWork$1 run     Jenkins version: 2.201 I don't have groovy-postbuild plugin installed. Plugin versions.   docker-workflow 1.21 workflow-aggregator 2.6 workflow-api 2.37 workflow-basic-steps 2.18 workflow-cps 2.74 workflow-cps-global-lib 2.15 workflow-durable-task-step 2.34 workflow-job 2.35 workflow-multibranch 2.21 workflow-remote-loader 1.5 workflow-scm-step 2.9 workflow-step-api 2.20 workflow-support 3.3 Any help would much appreciated.    
            markewaite Mark Waite added a comment -

            cosbug that seems to be a call reading the log file from org.jenkinsci.plugins.compressbuildlog, the compress build log plugin, probably at line 85. You're welcome to submit a pull request to that repository proposing to correct that plugin. Alternately, you could disable build log compression in your job definitions.

            markewaite Mark Waite added a comment - cosbug that seems to be a call reading the log file from org.jenkinsci.plugins.compressbuildlog, the compress build log plugin, probably at line 85 . You're welcome to submit a pull request to that repository proposing to correct that plugin. Alternately, you could disable build log compression in your job definitions.

            Thank you Mark.

            cosbug Constantin Bugneac added a comment - Thank you Mark.

            Is there a fix for timestamper-plugin? Since LogFileReader.nextLine() is called for every line when downloading logs from Jenkins, the log is flooded with bogus errors. I took a look at https://github.com/jenkinsci/timestamper-plugin/blob/master/src/main/java/hudson/plugins/timestamper/io/LogFileReader.java#L157 but am at a loss on how to keep the contract of that method while avoiding the call to getLogFile. (Would be nice if Run had a method hasLog()...)

            tgr Tobias Gruetzmacher added a comment - Is there a fix for timestamper-plugin ? Since LogFileReader.nextLine() is called for every line when downloading logs from Jenkins, the log is flooded with bogus errors. I took a look at https://github.com/jenkinsci/timestamper-plugin/blob/master/src/main/java/hudson/plugins/timestamper/io/LogFileReader.java#L157 but am at a loss on how to keep the contract of that method while avoiding the call to getLogFile . (Would be nice if Run had a method hasLog() ...)
            basil Basil Crow added a comment -

            tgr, I have a rough draft of a fix for Timestamper. I plan to post a PR soon and release the fix in the forthcoming Timestamper 1.11.1 release. To help me test this new functionality, would you mind posting the stack trace that you are seeing with the call to getLogFile?

            basil Basil Crow added a comment - tgr , I have a rough draft of a fix for Timestamper. I plan to post a PR soon and release the fix in the forthcoming Timestamper 1.11.1 release. To help me test this new functionality, would you mind posting the stack trace that you are seeing with the call to getLogFile ?
            basil Basil Crow added a comment -

            I posted jenkinsci/timestamper-plugin#52, which eliminates the last call to the last call to Run#getLogFile in Timestamper and replaces it with a call to Run#getLogReader.

            basil Basil Crow added a comment - I posted jenkinsci/timestamper-plugin#52 , which eliminates the last call to the last call to Run#getLogFile in Timestamper and replaces it with a call to Run#getLogReader .
            basil Basil Crow added a comment -

            I removed the timestamper-plugin component from this issue, since a fix has been released in Timestamper 1.11.1.

            FYI, the usage-in-plugins job shows that there are still 16 plugins with calls to the deprecated Run#getLogFile method. Most of these plugins haven't been updated in over a year, but I noticed that google-storage, sauce-ondemand, stoplightio-report, and token-macro have all had recent updates, so I added them to this issue's component list.

            basil Basil Crow added a comment - I removed the timestamper-plugin component from this issue, since a fix has been released in Timestamper 1.11.1 . FYI, the usage-in-plugins job shows that there are still 16 plugins with calls to the deprecated Run#getLogFile method. Most of these plugins haven't been updated in over a year, but I noticed that google-storage , sauce-ondemand , stoplightio-report , and token-macro have all had recent updates, so I added them to this issue's component list.
            slide_o_mix Alex Earl added a comment -

            I removed token macro since the PR was merged and released.

            slide_o_mix Alex Earl added a comment - I removed token macro since the PR was merged and released.
            markewaite Mark Waite added a comment -

            Fix merged in linenumbers plugin 1.3 released 11 months ago.

            markewaite Mark Waite added a comment - Fix merged in linenumbers plugin 1.3 released 11 months ago.

            People

              Unassigned Unassigned
              aroq Alexander Tolstikov
              Votes:
              5 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated: