Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55024

Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      When upgrading to version 2.29 of the "Pipeline: Job" plugin, users had a completely empty console output in case of an error.

      Downgrading to version 2.28 fixed the issue.

      When investigating we found an error log that a log file could not be located:

      java.io.FileNotFoundException: /var/lib/jenkins/jobs/iText-merge-pipeline/builds/319/log (No such file or directory)

      See attachments for screenshots of the console output and the log.

      If I read the change log for version 2.29 

      • JENKINS-54128: Change the implementation of WorkflowRun#getLogFile to avoid creating a new temporary file each time the method is called.
      • Fix: Do not call WorkflowRun#getLogFile when an error occurs while opening the log file for a Pipeline to avoid logging an additional stack trace

      My guess would be that WorkflowRun#getLogFile is also not called when an error is thrown in a pipeline script.

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment -

            I don't think the changes in workflow-job 2.29 and workflow-api 2.32 related to WorkflowRun#getLogFile are related to an empty console view, since the WorkflowRun#getLogFile method is not actually used when displaying logs.

            The stack trace you posted shows that the call to new RandomAccessFile here threw a FileNotFoundException when trying to open the logs for a step because the log file did not exist. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? It seems like something else might have deleted that file, but I am not sure what could have caused it. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? CC Jesse Glick in case this scenario rings a bell to him.

            Show
            dnusbaum Devin Nusbaum added a comment - I don't think the changes in workflow-job 2.29 and workflow-api 2.32 related to WorkflowRun#getLogFile are related to an empty console view, since the WorkflowRun#getLogFile method is not actually used when displaying logs. The stack trace you posted shows that the call to new RandomAccessFile here threw a FileNotFoundException when trying to open the logs for a step because the log file did not exist. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? It seems like something else might have deleted that file, but I am not sure what could have caused it. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? CC Jesse Glick in case this scenario rings a bell to him.
            Hide
            beyerj Jens Beyer added a comment - - edited

            Regarding the exception:

            The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system.

            For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files.

            But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.

             

            Regarding the empty console: I opened JENKINS-55465 which may also be the reason for your issue.

            Show
            beyerj Jens Beyer added a comment - - edited Regarding the exception: The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system. For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files. But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.   Regarding the empty console: I opened JENKINS-55465 which may also be the reason for your issue.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.

            For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs".

            Show
            dnusbaum Devin Nusbaum added a comment - But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs. For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs".
            Hide
            beyerj Jens Beyer added a comment -

            Devin Nusbaum Sorry I just realized that my issue with the build step logs (the first one I mentioned) was probably also caused by the full console issue with compressedBuildLog (JENKINS-55465, the second one I mentioned). Please disregard.

            Show
            beyerj Jens Beyer added a comment - Devin Nusbaum Sorry I just realized that my issue with the build step logs (the first one I mentioned) was probably also caused by the full console issue with compressedBuildLog ( JENKINS-55465 , the second one I mentioned). Please disregard.
            Hide
            jglick Jesse Glick added a comment -

            After the back and forth here, I am unclear on what the status of this is. Do we have a reproducible bug?

            Show
            jglick Jesse Glick added a comment - After the back and forth here, I am unclear on what the status of this is. Do we have a reproducible bug?
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Jesse Glick Nothing reproducible that I am aware of other than the duplicate related to compressBuildLog. Marco Andries are you still seeing this issue? If so, can you answer the questions in my earlier comment and perhaps attach the Pipeline script that is failing so we can see what it is doing? I am going to close this as incomplete for now, but please feel free to reopen it if you are able to provide additional data.

            Show
            dnusbaum Devin Nusbaum added a comment - Jesse Glick Nothing reproducible that I am aware of other than the duplicate related to compressBuildLog . Marco Andries are you still seeing this issue? If so, can you answer the questions in my earlier comment and perhaps attach the Pipeline script that is failing so we can see what it is doing? I am going to close this as incomplete for now, but please feel free to reopen it if you are able to provide additional data.
            Hide
            amedee Amedee Van Gasse added a comment - - edited

            Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround.

            The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28.

            I will be taking over this issue and answering all your questions about additional information until we get to a solution.

            I previously wasn't involved with reporting this issue so it is possible that I will be asking things that have already been asked before, please bear with me.

            Show
            amedee Amedee Van Gasse added a comment - - edited Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround. The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28. I will be taking over this issue and answering all your questions about additional information until we get to a solution. I previously wasn't involved with reporting this issue so it is possible that I will be asking things that have already been asked before, please bear with me.
            Hide
            amedee Amedee Van Gasse added a comment -

            There were several questions for information. I'll try to bring some structure in them, and then answer them one by one.

            1. What does the build folder for the build look like?
            2. Does the log file exist?
            3. Are there any other errors in the Jenkins logs that seem relevant?
            4. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed?
            5. For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing?
            6. I am not quite sure what you mean by "full build step logs".
            7. Do we have a reproducible bug?
            8. Marco Andries are you still seeing this issue?
            9. Can you answer the questions in my earlier comment?
            10. Can you attach the Pipeline script that is failing so we can see what it is doing?

            Question 1

            I am unsure what kind of answer you want:

            • a screenshot of the build folder?
            • a directory listing of the build folder?
            • a tarball of the build folder?

            I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs ls -ld $(find /var/lib/jenkins). Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information.

            Question 2

            How can I find out if the log file exists?

            Question 3

            Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look.

            Question 4

            This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce:

            1. Pipeline Job Plugin is at version 1.28.
            2. Start a failing job, job 1, and let it finish.
            3. Look at the console output of job 1 -> there is console output for job 1.
            4. Upgrade Pipeline Job Plugin to > 1.28.
            5. Restart Jenkins.
            6. Look at console output of job 1 -> there is console output for job 1.
            7. Start a failing job, job 2, and let it finish.
            8. Look at the console output of job 2 -> the console output for job 2 is empty.
            9. Downgrade Pipeline Job Plugin to 1.28.
            10. Restart Jenkins.
            11. Look at the console output of job 1 -> there is console output for job 1.
            12. Look at the console output of job 2 -> the console output for job 2 is empty.
            13. Start a failing job, job 3, and let it finish.
            14. Look at the console output of job 3 -> there is console output for job 3.

            Question 5

            For builds from before the update, there is indeed a console log, as described in my answer to question 4.

            I do not know about markup, is that important or relevant, and if so, how can I verify this?

            The step logs are expected to be broken.

            -> I don't understand what this means, could you please explain?

            Question 6

            Is this a question for Marco Andries or for Jens Beyer?

            Question 7

            Is this a question for Marco Andries or for Jens Beyer/Devin Nusbaum?

            Question 8

            Yes, we are still seeing this issue.

            Question 9

            I am collaborating with Marco Andries in providing you the answers.

            Question 10

            test-pipeline-Jenkinsfile

            Show
            amedee Amedee Van Gasse added a comment - There were several questions for information. I'll try to bring some structure in them, and then answer them one by one. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs". Do we have a reproducible bug? Marco Andries  are you still seeing this issue? Can you answer the questions in my earlier comment? Can you attach the Pipeline script that is failing so we can see what it is doing? Question 1 I am unsure what kind of answer you want: a screenshot of the build folder? a directory listing of the build folder? a tarball of the build folder? I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs  ls -ld $(find /var/lib/jenkins) . Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information. Question 2 How can I find out if the log file exists? Question 3 Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look. Question 4 This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce: Pipeline Job Plugin is at version 1.28. Start a failing job, job 1, and let it finish. Look at the console output of job 1 -> there is console output for job 1. Upgrade Pipeline Job Plugin to > 1.28. Restart Jenkins. Look at console output of job 1 -> there is console output for job 1. Start a failing job, job 2, and let it finish. Look at the console output of job 2 -> the console output for job 2 is empty. Downgrade Pipeline Job Plugin to 1.28. Restart Jenkins. Look at the console output of job 1 -> there is console output for job 1. Look at the console output of job 2 -> the console output for job 2 is empty. Start a failing job, job 3, and let it finish. Look at the console output of job 3 -> there is console output for job 3. Question 5 For builds from before the update, there is indeed a console log, as described in my answer to question 4. I do not know about markup, is that important or relevant, and if so, how can I verify this? The step logs are expected to be broken. -> I don't understand what this means, could you please explain? Question 6 Is this a question for Marco Andries or for Jens Beyer ? Question 7 Is this a question for  Marco Andries or for Jens Beyer / Devin Nusbaum ? Question 8 Yes, we are still seeing this issue. Question 9 I am collaborating with Marco Andries in providing you the answers. Question 10 test-pipeline-Jenkinsfile
            Hide
            amedee Amedee Van Gasse added a comment - - edited

            Meanwhile the job to make a directory listing, mentioned under question 1, had failed 

            line 2: /usr/bin/ls: Argument list too long

            I should have known... I'll do it again, this time with

            find /var/lib/jenkins -print0 | xargs -0 ls -ld
            Show
            amedee Amedee Van Gasse added a comment - - edited Meanwhile the job to make a directory listing, mentioned under question 1, had failed  line 2: /usr/bin/ls: Argument list too long I should have known... I'll do it again, this time with find / var /lib/jenkins -print0 | xargs -0 ls -ld
            Hide
            amedee Amedee Van Gasse added a comment -

            The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!

            Show
            amedee Amedee Van Gasse added a comment - The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!
            Hide
            amedee Amedee Van Gasse added a comment -

            I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB.
            After compression with xz, it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more.
            consoleText-iTextMergePipeline.txt.xz

            Show
            amedee Amedee Van Gasse added a comment - I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB. After compression with xz , it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more. consoleText-iTextMergePipeline.txt.xz
            Hide
            amedee Amedee Van Gasse added a comment -

            Devin Nusbaum Jesse Glick Jens Beyer I think that I answered all the questions you asked Marco Andries, can you have another look at this issue? Thanks!

            Show
            amedee Amedee Van Gasse added a comment - Devin Nusbaum Jesse Glick Jens Beyer I think that I answered all the questions you asked Marco Andries , can you have another look at this issue? Thanks!
            Hide
            marco_andries Marco Andries added a comment -

            Amedee Van Gasse, yes I believe you answered all questions.

            Show
            marco_andries Marco Andries added a comment - Amedee Van Gasse , yes I believe you answered all questions.
            Hide
            marco_andries Marco Andries added a comment -

            Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs.

            I think to have found the issue we're experiencing.....

            I saw the following when I look at the logs of a stage:

            When I check that location, I find the following:

            + ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3
            total 288
            drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive
            rw-rr- 1 jenkins jenkins 54829 Jun 5 14:43 build.xml
            rw-rr- 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml
            rw-rr- 1 jenkins jenkins 39174 Jun 5 14:43 log.gz
            rw-rr- 1 jenkins jenkins 329 Jun 5 14:43 log-index
            rw-rr- 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml
            rw-rr- 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml
            drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow
            drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files

            The log file is indeed not there but there's a log.gz file.

            I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz

            And now the console output is no longer empty and the stage log is also correct:

            So it seems there's an issue with packing and unpacking the log.

            Hope this helps....

            Show
            marco_andries Marco Andries added a comment - Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs. I think to have found the issue we're experiencing..... I saw the following when I look at the logs of a stage: When I check that location, I find the following: + ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3 total 288 drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive rw-r r - 1 jenkins jenkins 54829 Jun 5 14:43 build.xml rw-r r - 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml rw-r r - 1 jenkins jenkins 39174 Jun 5 14:43 log.gz rw-r r - 1 jenkins jenkins 329 Jun 5 14:43 log-index rw-r r - 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml rw-r r - 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files The log file is indeed not there but there's a log.gz file. I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz And now the console output is no longer empty and the stage log is also correct: So it seems there's an issue with packing and unpacking the log. Hope this helps....
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Ok, looks like a duplicate of JENKINS-54678. Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog, you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Ok, looks like a duplicate of  JENKINS-54678 . Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog , you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.
            Hide
            marco_andries Marco Andries added a comment -

            Devin Nusbaum, thanks for the reply. I will have a look.

            Show
            marco_andries Marco Andries added a comment - Devin Nusbaum , thanks for the reply. I will have a look.
            Hide
            marco_andries Marco Andries added a comment -

            Devin Nusbaum, it seems to solve the issue with the empty console outputs.

            No idea why we first only had it when an error was thrown.

            But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off.

            Thanks!

            Show
            marco_andries Marco Andries added a comment - Devin Nusbaum , it seems to solve the issue with the empty console outputs. No idea why we first only had it when an error was thrown. But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off. Thanks!

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              marco_andries Marco Andries
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: