Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29198

Console logs get truncated without reason

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Blocker
    • Resolution: Unresolved
    • core
    • Jenkins ver. 1.553, Chrome browser
      Jenkins ver. 2.150.1 from "Official" Docker Image running on Ubuntu 16.04 with jenkins_home mounted from the local file system on a RAID 0 setup.

    Description

      I have a bug which looks similar to JENKINS-14899:
      Sometimes, when a job runs for a long time without anything getting output, the jenkins page stops updating until the job finishes. Then it add the completion lines (SUCCESS or FAILURE) to the job.
      The difference is that the number of line does not matter (last bug occurence was on a job that output 264 lines (+ an unknown number of non-output lines)

      This results in a truncated /console, /consoleFull and /consoleText.

      All files do end with the two lines (if it succeeds):
      Notifying upstream projects of job completion
      Finished: SUCCESS
      but hundreds of lines didn't get stored by jenkins (over 10 minutes of logs for my last job!)

      I can often see this bug when I'm running a long impala query, where the output stops before the status line (something like "Returned 0 row(s) in 1987.42s") and the rest of the console logs never get output.

      This happens approximately one time out of 20 for the jobs I'm running myself on this Jenkins server.

      Attachments

        Activity

          camusensei Namikaze Minato added a comment - - edited

          I have no problem on my jenkins servers:

          2.249.1
          2.138
          2.27
          2.26
          2.204.5
          2.235.5
          2.138.1

          You could try to test if the problem is reproducible, like a loop printing messages, e.g.:

          #!/bin/bash
          while sleep 1; do printf '%*s\n' 300 "$SECONDS"; done
          

          Try it and see if your problem always happens after the same time, or if such a simple script does not allow you to reproduce the problem?

           

          camusensei Namikaze Minato added a comment - - edited I have no problem on my jenkins servers: 2.249.1 2.138 2.27 2.26 2.204.5 2.235.5 2.138.1 You could try to test if the problem is reproducible, like a loop printing messages, e.g.: #!/bin/bash while sleep 1; do printf '%*s\n' 300 "$SECONDS"; done Try it and see if your problem always happens after the same time, or if such a simple script does not allow you to reproduce the problem?  
          alok15ee Alok Singh added a comment -

          camusensei Nope it didn't work either. I think it also depends on the amount of data in one line we write to the BufferedInputStream which is causing this issue to pop up.

          alok15ee Alok Singh added a comment - camusensei Nope it didn't work either. I think it also depends on the amount of data in one line we write to the BufferedInputStream which is causing this issue to pop up.

          I think you need to give more information on your problem which seems to be different from the one I used to have. Maybe open a new issue, and describe preciseley anything you can.

          camusensei Namikaze Minato added a comment - I think you need to give more information on your problem which seems to be different from the one I used to have. Maybe open a new issue, and describe preciseley anything you can.

          I am seeing this issue right now on Jenkins 2.335!

          There's something gone awry.

          We use kubernetes and had been having some connection issues we solved by stopping the use of websockets to agents, but the truncated logs keep happening, developers cannot debug builds, and I cannot tell them why it's failed or why there are no logs...

          uzumaki_kushina Uzumaki Kushina added a comment - I am seeing this issue right now on Jenkins 2.335! There's something gone awry. We use kubernetes and had been having some connection issues we solved by stopping the use of websockets to agents, but the truncated logs keep happening, developers cannot debug builds, and I cannot tell them why it's failed or why there are no logs...

          The only thing interesting I see in system jenkins logs is:

          Mar 04, 2022 4:56:35 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRootJENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942

          uzumaki_kushina Uzumaki Kushina added a comment - The only thing interesting I see in system jenkins logs is: Mar 04, 2022 4:56:35 PM WARNING jenkins.branch.WorkspaceLocatorImpl getWorkspaceRoot JENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942

          People

            Unassigned Unassigned
            camusensei Namikaze Minato
            Votes:
            17 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated: