-
Bug
-
Resolution: Duplicate
-
Major
-
Jenkins version 2.138.3 running on Linux
When upgrading to version 2.29 of the "Pipeline: Job" plugin, users had a completely empty console output in case of an error.
Downgrading to version 2.28 fixed the issue.
When investigating we found an error log that a log file could not be located:
java.io.FileNotFoundException: /var/lib/jenkins/jobs/iText-merge-pipeline/builds/319/log (No such file or directory)
See attachments for screenshots of the console output and the log.
If I read the change log for version 2.29
- JENKINS-54128: Change the implementation of WorkflowRun#getLogFile to avoid creating a new temporary file each time the method is called.
- Fix: Do not call WorkflowRun#getLogFile when an error occurs while opening the log file for a Pipeline to avoid logging an additional stack trace
My guess would be that WorkflowRun#getLogFile is also not called when an error is thrown in a pipeline script.
- duplicates
-
-
- Open
-
[JENKINS-55024] Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs
Regarding the exception:
The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system.
For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files.
But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.
Regarding the empty console: I opened JENKINS-55465 which may also be the reason for your issue.
But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.
For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs".
dnusbaum Sorry I just realized that my issue with the build step logs (the first one I mentioned) was probably also caused by the full console issue with compressedBuildLog (JENKINS-55465, the second one I mentioned). Please disregard.
After the back and forth here, I am unclear on what the status of this is. Do we have a reproducible bug?
jglick Nothing reproducible that I am aware of other than the duplicate related to compressBuildLog. marco_andries are you still seeing this issue? If so, can you answer the questions in my earlier comment and perhaps attach the Pipeline script that is failing so we can see what it is doing? I am going to close this as incomplete for now, but please feel free to reopen it if you are able to provide additional data.
Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround.
The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28.
I will be taking over this issue and answering all your questions about additional information until we get to a solution.
I previously wasn't involved with reporting this issue so it is possible that I will be asking things that have already been asked before, please bear with me.
Amedee Van Gasse
added a comment - - edited Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround.
The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28.
I will be taking over this issue and answering all your questions about additional information until we get to a solution.
I previously wasn't involved with reporting this issue so it is possible that I will be asking things that have already been asked before, please bear with me. Amedee Van Gasse
added a comment - There were several questions for information. I'll try to bring some structure in them, and then answer them one by one.
- What does the build folder for the build look like?
- Does the log file exist?
- Are there any other errors in the Jenkins logs that seem relevant?
- Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed?
- For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing?
- I am not quite sure what you mean by "full build step logs".
- Do we have a reproducible bug?
- marco_andries are you still seeing this issue?
- Can you answer the questions in my earlier comment?
- Can you attach the Pipeline script that is failing so we can see what it is doing?
Question 1
I am unsure what kind of answer you want:
- a screenshot of the build folder?
- a directory listing of the build folder?
- a tarball of the build folder?
I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs ls -ld $(find /var/lib/jenkins). Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information.
Question 2
How can I find out if the log file exists?
Question 3
Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look.
Question 4
This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce:
- Pipeline Job Plugin is at version 1.28.
- Start a failing job, job 1, and let it finish.
- Look at the console output of job 1 -> there is console output for job 1.
- Upgrade Pipeline Job Plugin to > 1.28.
- Restart Jenkins.
- Look at console output of job 1 -> there is console output for job 1.
- Start a failing job, job 2, and let it finish.
- Look at the console output of job 2 -> the console output for job 2 is empty.
- Downgrade Pipeline Job Plugin to 1.28.
- Restart Jenkins.
- Look at the console output of job 1 -> there is console output for job 1.
- Look at the console output of job 2 -> the console output for job 2 is empty.
- Start a failing job, job 3, and let it finish.
- Look at the console output of job 3 -> there is console output for job 3.
Question 5
For builds from before the update, there is indeed a console log, as described in my answer to question 4.
I do not know about markup, is that important or relevant, and if so, how can I verify this?
The step logs are expected to be broken.
-> I don't understand what this means, could you please explain?
Question 6
Is this a question for marco_andries or for beyerj?
Question 7
Is this a question for marco_andries or for beyerj/dnusbaum?
Question 8
Yes, we are still seeing this issue.
Question 9
I am collaborating with marco_andries in providing you the answers.
Question 10
Amedee Van Gasse
added a comment - There were several questions for information. I'll try to bring some structure in them, and then answer them one by one.
What does the build folder for the build look like?
Does the log file exist?
Are there any other errors in the Jenkins logs that seem relevant?
Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed?
For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing?
I am not quite sure what you mean by "full build step logs".
Do we have a reproducible bug?
marco_andries are you still seeing this issue?
Can you answer the questions in my earlier comment?
Can you attach the Pipeline script that is failing so we can see what it is doing?
Question 1
I am unsure what kind of answer you want:
a screenshot of the build folder?
a directory listing of the build folder?
a tarball of the build folder?
I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs ls -ld $(find /var/lib/jenkins) . Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information.
Question 2
How can I find out if the log file exists?
Question 3
Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look.
Question 4
This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce:
Pipeline Job Plugin is at version 1.28.
Start a failing job, job 1, and let it finish.
Look at the console output of job 1 -> there is console output for job 1.
Upgrade Pipeline Job Plugin to > 1.28.
Restart Jenkins.
Look at console output of job 1 -> there is console output for job 1.
Start a failing job, job 2, and let it finish.
Look at the console output of job 2 -> the console output for job 2 is empty.
Downgrade Pipeline Job Plugin to 1.28.
Restart Jenkins.
Look at the console output of job 1 -> there is console output for job 1.
Look at the console output of job 2 -> the console output for job 2 is empty.
Start a failing job, job 3, and let it finish.
Look at the console output of job 3 -> there is console output for job 3.
Question 5
For builds from before the update, there is indeed a console log, as described in my answer to question 4.
I do not know about markup, is that important or relevant, and if so, how can I verify this?
The step logs are expected to be broken.
-> I don't understand what this means, could you please explain?
Question 6
Is this a question for marco_andries or for beyerj ?
Question 7
Is this a question for marco_andries or for beyerj / dnusbaum ?
Question 8
Yes, we are still seeing this issue.
Question 9
I am collaborating with marco_andries in providing you the answers.
Question 10
test-pipeline-Jenkinsfile Meanwhile the job to make a directory listing, mentioned under question 1, had failed
line 2: /usr/bin/ls: Argument list too long
I should have known... I'll do it again, this time with
find /var/lib/jenkins -print0 | xargs -0 ls -ld
Amedee Van Gasse
added a comment - - edited Meanwhile the job to make a directory listing, mentioned under question 1, had failed
line 2: /usr/bin/ls: Argument list too long
I should have known... I'll do it again, this time with
find / var /lib/jenkins -print0 | xargs -0 ls -ld
Amedee Van Gasse
added a comment - The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!
Amedee Van Gasse
added a comment - The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome! Amedee Van Gasse
added a comment - I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB.
After compression with xz, it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more.
consoleText-iTextMergePipeline.txt.xz
Amedee Van Gasse
added a comment - I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB.
After compression with xz , it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more.
consoleText-iTextMergePipeline.txt.xz Amedee Van Gasse
added a comment - dnusbaum jglick beyerj I think that I answered all the questions you asked marco_andries, can you have another look at this issue? Thanks!
Amedee Van Gasse
added a comment - dnusbaum jglick beyerj I think that I answered all the questions you asked marco_andries , can you have another look at this issue? Thanks! 
Marco Andries
added a comment - amedee , yes I believe you answered all questions. Marco Andries
added a comment - Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs.
I think to have found the issue we're experiencing.....
I saw the following when I look at the logs of a stage:
When I check that location, I find the following:
+ ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3
total 288
drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive
rw-rr- 1 jenkins jenkins 54829 Jun 5 14:43 build.xml
rw-rr- 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml
rw-rr- 1 jenkins jenkins 39174 Jun 5 14:43 log.gz
rw-rr- 1 jenkins jenkins 329 Jun 5 14:43 log-index
rw-rr- 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml
rw-rr- 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml
drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow
drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files
The log file is indeed not there but there's a log.gz file.
I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz
And now the console output is no longer empty and the stage log is also correct:
So it seems there's an issue with packing and unpacking the log.
Hope this helps....
Marco Andries
added a comment - Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs.
I think to have found the issue we're experiencing.....
I saw the following when I look at the logs of a stage:
When I check that location, I find the following:
+ ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3
total 288
drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive
rw-r r - 1 jenkins jenkins 54829 Jun 5 14:43 build.xml
rw-r r - 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml
rw-r r - 1 jenkins jenkins 39174 Jun 5 14:43 log.gz
rw-r r - 1 jenkins jenkins 329 Jun 5 14:43 log-index
rw-r r - 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml
rw-r r - 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml
drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow
drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files
The log file is indeed not there but there's a log.gz file.
I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz
And now the console output is no longer empty and the stage log is also correct:
So it seems there's an issue with packing and unpacking the log.
Hope this helps.... Ok, looks like a duplicate of JENKINS-54678. Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog, you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.
Marco Andries
added a comment - dnusbaum , thanks for the reply. I will have a look. Marco Andries
added a comment - dnusbaum, it seems to solve the issue with the empty console outputs.
No idea why we first only had it when an error was thrown.
But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off.
Thanks!
Marco Andries
added a comment - dnusbaum , it seems to solve the issue with the empty console outputs.
No idea why we first only had it when an error was thrown.
But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off.
Thanks! 
I don't think the changes in workflow-job 2.29 and workflow-api 2.32 related to WorkflowRun#getLogFile are related to an empty console view, since the WorkflowRun#getLogFile method is not actually used when displaying logs.
The stack trace you posted shows that the call to new RandomAccessFile here threw a FileNotFoundException when trying to open the logs for a step because the log file did not exist. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? It seems like something else might have deleted that file, but I am not sure what could have caused it. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? CC jglick in case this scenario rings a bell to him.