• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • core
    • Jenkins ver. 1.553, Chrome browser
      Jenkins ver. 2.150.1 from "Official" Docker Image running on Ubuntu 16.04 with jenkins_home mounted from the local file system on a RAID 0 setup.

      I have a bug which looks similar to JENKINS-14899:
      Sometimes, when a job runs for a long time without anything getting output, the jenkins page stops updating until the job finishes. Then it add the completion lines (SUCCESS or FAILURE) to the job.
      The difference is that the number of line does not matter (last bug occurence was on a job that output 264 lines (+ an unknown number of non-output lines)

      This results in a truncated /console, /consoleFull and /consoleText.

      All files do end with the two lines (if it succeeds):
      Notifying upstream projects of job completion
      Finished: SUCCESS
      but hundreds of lines didn't get stored by jenkins (over 10 minutes of logs for my last job!)

      I can often see this bug when I'm running a long impala query, where the output stops before the status line (something like "Returned 0 row(s) in 1987.42s") and the rest of the console logs never get output.

      This happens approximately one time out of 20 for the jobs I'm running myself on this Jenkins server.

          [JENKINS-29198] Console logs get truncated without reason

          Namikaze Minato added a comment - - edited

          Wow, I am just facing this issue at the very moment, in the middle of a map/reduce job. (I can see in the jobtracker this map/reduce job finished a long time ago and my (still running) jenkins job started the next map/reduce)
          I opened the console and all progressiveHtml calls were empty. http://prntscr.com/7o0vg9

          Things I tried:
          Refreshing - same state
          Opening another tab/window - same state
          Looking at /consoleText - same state
          Manually opening /consoleFull - same state
          Logging out and logging in in an incognito window - same state

          (So "runing for a long time without anything getting output" has nothing to do with this bug)
          Number of lines before stopping this time: 3201

          Namikaze Minato added a comment - - edited Wow, I am just facing this issue at the very moment, in the middle of a map/reduce job. (I can see in the jobtracker this map/reduce job finished a long time ago and my (still running) jenkins job started the next map/reduce) I opened the console and all progressiveHtml calls were empty. http://prntscr.com/7o0vg9 Things I tried: Refreshing - same state Opening another tab/window - same state Looking at /consoleText - same state Manually opening /consoleFull - same state Logging out and logging in in an incognito window - same state (So "runing for a long time without anything getting output" has nothing to do with this bug) Number of lines before stopping this time: 3201

          Daniel Beck added a comment -

          Please make sure any issues you experience happen on a reasonably recent version of Jenkins. Since it's been more than a year since 1.553 was released, this issue may well be obsolete.

          Daniel Beck added a comment - Please make sure any issues you experience happen on a reasonably recent version of Jenkins. Since it's been more than a year since 1.553 was released, this issue may well be obsolete.

          It does not happen with a recent version.
          Please close the issue.

          Namikaze Minato added a comment - It does not happen with a recent version. Please close the issue.

          Ben Eng added a comment - - edited

          I am seeing this exact problem today on version 2.121.3

          The console output displays to the end, showing Finished: SUCCESS. The consoleText output is truncated (at 4604 lines in the case I'm looking at this moment). The consoleFull output is truncated at exactly the same line as consoleText.

          Ben Eng added a comment - - edited I am seeing this exact problem today on version  2.121.3 The console output displays to the end, showing Finished: SUCCESS. The consoleText output is truncated (at 4604 lines in the case I'm looking at this moment). The consoleFull output is truncated at exactly the same line as consoleText.

          As of today am also seeing this on version 2.154 I don't see any success or failure messages. I also noticed that loading output for nodes in the blue ocean view seems to give a red console error. (I'm guessing these are somehow loaded from the log which doesn't have the info since it's truncated.)

          Ryan Fenton-Garcia added a comment - As of today am also seeing this on version 2.154 I don't see any success or failure messages. I also noticed that loading output for nodes in the blue ocean view seems to give a red console error. (I'm guessing these are somehow loaded from the log which doesn't have the info since it's truncated.)

          H L added a comment - - edited

          I am seeing this issue as well on Jenkins 2.150.1 that comes from the "official" Docker image provided from DockerHub. I'm trying to debug an issue with a high level of parallelism and the full logs are truncated. There are logs showing in the BlueOcean view for the individual nodes, but some of those appear truncated as well, but that is difficult to confirm since the full console, consoleText, and consoleFull are truncated.

          Additionally, I even got the infamous "Jenkins Devil" bug image directly in the console output a couple of times when trying to watch the logs live as they were output.

          Please help provide some insight into what could cause this behavior if it could be a bug in the Pipeline code that can be worked around. Even if a there is a bug in our Pipeline code causing this issue, Jenkins should still show the full console output.

          H L added a comment - - edited I am seeing this issue as well on Jenkins 2.150.1 that comes from the "official" Docker image provided from DockerHub. I'm trying to debug an issue with a high level of parallelism and the full logs are truncated. There are logs showing in the BlueOcean view for the individual nodes, but some of those appear truncated as well, but that is difficult to confirm since the full console, consoleText, and consoleFull are truncated. Additionally, I even got the infamous "Jenkins Devil" bug image directly in the console output a couple of times when trying to watch the logs live as they were output. Please help provide some insight into what could cause this behavior if it could be a bug in the Pipeline code that can be worked around. Even if a there is a bug in our Pipeline code causing this issue, Jenkins should still show the full console output.

          H L added a comment -

          Please increase the priority of this issue since this is impossible for a developer to troubleshoot without console output to indicate what the issue is.

          Issues causing outages in the console output seems as though they should be treated with the highest priority rather than Minor since there is currently no workaround.

          H L added a comment - Please increase the priority of this issue since this is impossible for a developer to troubleshoot without console output to indicate what the issue is. Issues causing outages in the console output seems as though they should be treated with the highest priority rather than Minor since there is currently no workaround.

          H L added a comment -

          jetpen rjfenton Please don't forget to vote for this issue if you are getting impacted. I seem to be the only one that has voted for it so far.

          H L added a comment - jetpen rjfenton Please don't forget to vote for this issue if you are getting impacted. I seem to be the only one that has voted for it so far.

          Please increase the priority.

           

          Sayed Rizwan Inamdar added a comment - Please increase the priority.  

          Andreas Sieferlinger added a comment - - edited

          We are seeing this issue aswell. Its seems like this happens mostly with pipelines that have multiple parallel steps.

          Running on 2.150.3

          We are using also the Timestamper plugin, just curious if anybody else facing the issue is using it aswell and if it might be related.

          Andreas Sieferlinger added a comment - - edited We are seeing this issue aswell. Its seems like this happens mostly with pipelines that have multiple parallel steps. Running on 2.150.3 We are using also the Timestamper plugin, just curious if anybody else facing the issue is using it aswell and if it might be related.

          Gabriel Rusu added a comment -

          Indeed. For us, this happens when we run like 6 builds in parallel. However, if we break them in two, having only 3 builds run at a time, same setup, just two sequence of 3 builds, we get full logs.

          Gabriel Rusu added a comment - Indeed. For us, this happens when we run like 6 builds in parallel. However, if we break them in two, having only 3 builds run at a time, same setup, just two sequence of 3 builds, we get full logs.

          Elliot Nelson added a comment -

          Recently seen this a couple times on Jenkins 2.193.

           

          In our case, it's a multijob that ends like this:

          ...
          Finished Build : #18084 of Job : fvt-worker-5 with status : SUCCESS
          Finished Build : #18084 of Job : fvt-worker-20 with status : SUCCESS
          Finished Build : #18085 of Job : fvt-worker-7 with status : SUCCESS
          Finished Build : #18085 of Job : fvt-worker-23 with status : SUCCESS
          Finished Build : #18084 of Job : fvt-worker-8 with status : SUCCESS
          Finished Build : Finished Build :  

           

          It's clear the job finished successfully (including another dozen subjobs that should be listed but aren't), and that it also completed all the publisher steps as expected, but there's about 120 lines of log at the end there that are missing (affects /console, /consoleFull, and /consoleText).

           

          Elliot Nelson added a comment - Recently seen this a couple times on Jenkins 2.193.   In our case, it's a multijob that ends like this: ... Finished Build : #18084 of Job : fvt-worker-5 with status : SUCCESS Finished Build : #18084 of Job : fvt-worker-20 with status : SUCCESS Finished Build : #18085 of Job : fvt-worker-7 with status : SUCCESS Finished Build : #18085 of Job : fvt-worker-23 with status : SUCCESS Finished Build : #18084 of Job : fvt-worker-8 with status : SUCCESS Finished Build : Finished Build :   It's clear the job finished successfully (including another dozen subjobs that should be listed but aren't), and that it also completed all the publisher steps as expected, but there's about 120 lines of log at the end there that are missing (affects /console, /consoleFull, and /consoleText).  

          ASHOK MOHANTY added a comment -

          In our case - "Console logs get truncated when job runs for a long time". Are there any log size limit !?

          ASHOK MOHANTY added a comment - In our case - "Console logs get truncated when job runs for a long time". Are there any log size limit !?

          ASHOK MOHANTY added a comment -

          We are using core 2.164.3 version. Ay idea, by upgrading to any specific version - will it help !! 

          And/or any workaround we can try !!

          ASHOK MOHANTY added a comment - We are using core 2.164.3 version. Ay idea, by upgrading to any specific version - will it help !!  And/or any workaround we can try !!

          Alok Singh added a comment -

          Did anyone get a workaround for this? 

          Alok Singh added a comment - Did anyone get a workaround for this? 

          Few Jenkins versions have this bug, just upgrade.

          Namikaze Minato added a comment - Few Jenkins versions have this bug, just upgrade.

          Alok Singh added a comment - - edited

          camusensei What version do you recommend? I have tried with 2.204.1 and 2.222.3 and I still run into the same issue.

          Alok Singh added a comment - - edited camusensei What version do you recommend? I have tried with 2.204.1 and 2.222.3 and I still run into the same issue.

          Namikaze Minato added a comment - - edited

          I have no problem on my jenkins servers:

          2.249.1
          2.138
          2.27
          2.26
          2.204.5
          2.235.5
          2.138.1

          You could try to test if the problem is reproducible, like a loop printing messages, e.g.:

          #!/bin/bash
          while sleep 1; do printf '%*s\n' 300 "$SECONDS"; done
          

          Try it and see if your problem always happens after the same time, or if such a simple script does not allow you to reproduce the problem?

           

          Namikaze Minato added a comment - - edited I have no problem on my jenkins servers: 2.249.1 2.138 2.27 2.26 2.204.5 2.235.5 2.138.1 You could try to test if the problem is reproducible, like a loop printing messages, e.g.: #!/bin/bash while sleep 1; do printf '%*s\n' 300 "$SECONDS"; done Try it and see if your problem always happens after the same time, or if such a simple script does not allow you to reproduce the problem?  

          Alok Singh added a comment -

          camusensei Nope it didn't work either. I think it also depends on the amount of data in one line we write to the BufferedInputStream which is causing this issue to pop up.

          Alok Singh added a comment - camusensei Nope it didn't work either. I think it also depends on the amount of data in one line we write to the BufferedInputStream which is causing this issue to pop up.

          I think you need to give more information on your problem which seems to be different from the one I used to have. Maybe open a new issue, and describe preciseley anything you can.

          Namikaze Minato added a comment - I think you need to give more information on your problem which seems to be different from the one I used to have. Maybe open a new issue, and describe preciseley anything you can.

          I am seeing this issue right now on Jenkins 2.335!

          There's something gone awry.

          We use kubernetes and had been having some connection issues we solved by stopping the use of websockets to agents, but the truncated logs keep happening, developers cannot debug builds, and I cannot tell them why it's failed or why there are no logs...

          Uzumaki Kushina added a comment - I am seeing this issue right now on Jenkins 2.335! There's something gone awry. We use kubernetes and had been having some connection issues we solved by stopping the use of websockets to agents, but the truncated logs keep happening, developers cannot debug builds, and I cannot tell them why it's failed or why there are no logs...

          The only thing interesting I see in system jenkins logs is:

          Mar 04, 2022 4:56:35 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942

          Uzumaki Kushina added a comment - The only thing interesting I see in system jenkins logs is: Mar 04, 2022 4:56:35 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRoot JENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942

          Nicolas added a comment - - edited

          Hello,

          I'm affected by this issue too, it's really annoying as it forces user to attempt a local rebuild to obtain logs, which even worse, doesn't work when the issue is specific to the server...

          I'm reproducing the issue with the following pipeline job:

          pipeline {
              agent any
          
              stages {
                  stage('Hello') {
                      matrix {
                          axes {
                              axis {
                                  name 'blarg'
                                  values 'foo', 'bar', 'baz', 'qux', 'wizz'
                              }
                          }
                          stages {
                              stage('Main') {
                                  steps {
                                      sh """
                                      for i in \$(seq 1 ${TURNS}); do
                                          echo hello world \${i}!
                                      done
                                      """
                                  }
                              }
                          }
                      }
                  }
              }
          }
          

          When TURNS is 100000, all logs look fine, when set to 200000, the logs are truncated and the job takes 10 to 40 times longer to complete.
          Note that when I redirect the logs to files by doing for example:

          (for i in \$(seq 1 ${TURNS}); do
              echo hello world \${i}!
          done) | tee ${blarg}.log
          

          Log files created are complete, so it doesn't look like a disk I/O bottleneck issue.

           

          Jenkins version: 2.375.1

          Nicolas added a comment - - edited Hello, I'm affected by this issue too, it's really annoying as it forces user to attempt a local rebuild to obtain logs, which even worse, doesn't work when the issue is specific to the server... I'm reproducing the issue with the following pipeline job: pipeline {     agent any     stages {         stage( 'Hello' ) {             matrix {                 axes {                     axis {                         name 'blarg'                         values 'foo' , 'bar' , 'baz' , 'qux' , 'wizz'                     }                 }                 stages {                     stage( 'Main' ) {                         steps {                             sh """                             for i in \$(seq 1 ${TURNS}); do                                 echo hello world \${i}!                             done                             """                         }                     }                 }             }         }     } } When TURNS is 100000, all logs look fine, when set to 200000, the logs are truncated and the job takes 10 to 40 times longer to complete. Note that when I redirect the logs to files by doing for example: ( for i in \$(seq 1 ${TURNS}); do echo hello world \${i}! done ) | tee ${blarg}.log Log files created are complete, so it doesn't look like a disk I/O bottleneck issue.   Jenkins version: 2.375.1

          Nicolas added a comment -

          In my situation, I think that this issue is related to https://issues.jenkins.io/browse/JENKINS-56446

          Nicolas added a comment - In my situation, I think that this issue is related to https://issues.jenkins.io/browse/JENKINS-56446

            Unassigned Unassigned
            camusensei Namikaze Minato
            Votes:
            20 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated: