Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55024

Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

      When upgrading to version 2.29 of the "Pipeline: Job" plugin, users had a completely empty console output in case of an error.

      Downgrading to version 2.28 fixed the issue.

      When investigating we found an error log that a log file could not be located:

      java.io.FileNotFoundException: /var/lib/jenkins/jobs/iText-merge-pipeline/builds/319/log (No such file or directory)

      See attachments for screenshots of the console output and the log.

      If I read the change log for version 2.29 

      • JENKINS-54128: Change the implementation of WorkflowRun#getLogFile to avoid creating a new temporary file each time the method is called.
      • Fix: Do not call WorkflowRun#getLogFile when an error occurs while opening the log file for a Pipeline to avoid logging an additional stack trace

      My guess would be that WorkflowRun#getLogFile is also not called when an error is thrown in a pipeline script.

          [JENKINS-55024] Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

          There were several questions for information. I'll try to bring some structure in them, and then answer them one by one.

          1. What does the build folder for the build look like?
          2. Does the log file exist?
          3. Are there any other errors in the Jenkins logs that seem relevant?
          4. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed?
          5. For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing?
          6. I am not quite sure what you mean by "full build step logs".
          7. Do we have a reproducible bug?
          8. marco_andries are you still seeing this issue?
          9. Can you answer the questions in my earlier comment?
          10. Can you attach the Pipeline script that is failing so we can see what it is doing?

          Question 1

          I am unsure what kind of answer you want:

          • a screenshot of the build folder?
          • a directory listing of the build folder?
          • a tarball of the build folder?

          I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs ls -ld $(find /var/lib/jenkins). Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information.

          Question 2

          How can I find out if the log file exists?

          Question 3

          Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look.

          Question 4

          This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce:

          1. Pipeline Job Plugin is at version 1.28.
          2. Start a failing job, job 1, and let it finish.
          3. Look at the console output of job 1 -> there is console output for job 1.
          4. Upgrade Pipeline Job Plugin to > 1.28.
          5. Restart Jenkins.
          6. Look at console output of job 1 -> there is console output for job 1.
          7. Start a failing job, job 2, and let it finish.
          8. Look at the console output of job 2 -> the console output for job 2 is empty.
          9. Downgrade Pipeline Job Plugin to 1.28.
          10. Restart Jenkins.
          11. Look at the console output of job 1 -> there is console output for job 1.
          12. Look at the console output of job 2 -> the console output for job 2 is empty.
          13. Start a failing job, job 3, and let it finish.
          14. Look at the console output of job 3 -> there is console output for job 3.

          Question 5

          For builds from before the update, there is indeed a console log, as described in my answer to question 4.

          I do not know about markup, is that important or relevant, and if so, how can I verify this?

          The step logs are expected to be broken.

          -> I don't understand what this means, could you please explain?

          Question 6

          Is this a question for marco_andries or for beyerj?

          Question 7

          Is this a question for marco_andries or for beyerj/dnusbaum?

          Question 8

          Yes, we are still seeing this issue.

          Question 9

          I am collaborating with marco_andries in providing you the answers.

          Question 10

          test-pipeline-Jenkinsfile

          Amedee Van Gasse added a comment - There were several questions for information. I'll try to bring some structure in them, and then answer them one by one. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs". Do we have a reproducible bug? marco_andries  are you still seeing this issue? Can you answer the questions in my earlier comment? Can you attach the Pipeline script that is failing so we can see what it is doing? Question 1 I am unsure what kind of answer you want: a screenshot of the build folder? a directory listing of the build folder? a tarball of the build folder? I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs  ls -ld $(find /var/lib/jenkins) . Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information. Question 2 How can I find out if the log file exists? Question 3 Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look. Question 4 This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce: Pipeline Job Plugin is at version 1.28. Start a failing job, job 1, and let it finish. Look at the console output of job 1 -> there is console output for job 1. Upgrade Pipeline Job Plugin to > 1.28. Restart Jenkins. Look at console output of job 1 -> there is console output for job 1. Start a failing job, job 2, and let it finish. Look at the console output of job 2 -> the console output for job 2 is empty. Downgrade Pipeline Job Plugin to 1.28. Restart Jenkins. Look at the console output of job 1 -> there is console output for job 1. Look at the console output of job 2 -> the console output for job 2 is empty. Start a failing job, job 3, and let it finish. Look at the console output of job 3 -> there is console output for job 3. Question 5 For builds from before the update, there is indeed a console log, as described in my answer to question 4. I do not know about markup, is that important or relevant, and if so, how can I verify this? The step logs are expected to be broken. -> I don't understand what this means, could you please explain? Question 6 Is this a question for marco_andries or for beyerj ? Question 7 Is this a question for  marco_andries or for beyerj / dnusbaum ? Question 8 Yes, we are still seeing this issue. Question 9 I am collaborating with marco_andries in providing you the answers. Question 10 test-pipeline-Jenkinsfile

          Amedee Van Gasse added a comment - - edited

          Meanwhile the job to make a directory listing, mentioned under question 1, had failed 

          line 2: /usr/bin/ls: Argument list too long

          I should have known... I'll do it again, this time with

          find /var/lib/jenkins -print0 | xargs -0 ls -ld

          Amedee Van Gasse added a comment - - edited Meanwhile the job to make a directory listing, mentioned under question 1, had failed  line 2: /usr/bin/ls: Argument list too long I should have known... I'll do it again, this time with find / var /lib/jenkins -print0 | xargs -0 ls -ld

          The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!

          Amedee Van Gasse added a comment - The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!

          I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB.
          After compression with xz, it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more.
          consoleText-iTextMergePipeline.txt.xz

          Amedee Van Gasse added a comment - I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB. After compression with xz , it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more. consoleText-iTextMergePipeline.txt.xz

          dnusbaum jglick beyerj I think that I answered all the questions you asked marco_andries, can you have another look at this issue? Thanks!

          Amedee Van Gasse added a comment - dnusbaum jglick beyerj I think that I answered all the questions you asked marco_andries , can you have another look at this issue? Thanks!

          Marco Andries added a comment -

          amedee, yes I believe you answered all questions.

          Marco Andries added a comment - amedee , yes I believe you answered all questions.

          Marco Andries added a comment -

          Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs.

          I think to have found the issue we're experiencing.....

          I saw the following when I look at the logs of a stage:

          When I check that location, I find the following:

          + ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3
          total 288
          drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive
          rw-rr- 1 jenkins jenkins 54829 Jun 5 14:43 build.xml
          rw-rr- 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml
          rw-rr- 1 jenkins jenkins 39174 Jun 5 14:43 log.gz
          rw-rr- 1 jenkins jenkins 329 Jun 5 14:43 log-index
          rw-rr- 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml
          rw-rr- 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml
          drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow
          drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files

          The log file is indeed not there but there's a log.gz file.

          I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz

          And now the console output is no longer empty and the stage log is also correct:

          So it seems there's an issue with packing and unpacking the log.

          Hope this helps....

          Marco Andries added a comment - Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs. I think to have found the issue we're experiencing..... I saw the following when I look at the logs of a stage: When I check that location, I find the following: + ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3 total 288 drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive rw-r r - 1 jenkins jenkins 54829 Jun 5 14:43 build.xml rw-r r - 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml rw-r r - 1 jenkins jenkins 39174 Jun 5 14:43 log.gz rw-r r - 1 jenkins jenkins 329 Jun 5 14:43 log-index rw-r r - 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml rw-r r - 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files The log file is indeed not there but there's a log.gz file. I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz And now the console output is no longer empty and the stage log is also correct: So it seems there's an issue with packing and unpacking the log. Hope this helps....

          Devin Nusbaum added a comment - - edited

          Ok, looks like a duplicate of JENKINS-54678. Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog, you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.

          Devin Nusbaum added a comment - - edited Ok, looks like a duplicate of  JENKINS-54678 . Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog , you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.

          Marco Andries added a comment -

          dnusbaum, thanks for the reply. I will have a look.

          Marco Andries added a comment - dnusbaum , thanks for the reply. I will have a look.

          Marco Andries added a comment -

          dnusbaum, it seems to solve the issue with the empty console outputs.

          No idea why we first only had it when an error was thrown.

          But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off.

          Thanks!

          Marco Andries added a comment - dnusbaum , it seems to solve the issue with the empty console outputs. No idea why we first only had it when an error was thrown. But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off. Thanks!

            Unassigned Unassigned
            marco_andries Marco Andries
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: