Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49153

Sh step repeatedly spews its last n output lines until a timeout is hit

XMLWordPrintable

      We find this happens often with steps like: 

      sh "docker-compose -f docker-compose-ci.yml logs containername"

      Or test steps that produce a lot of output.

      The outcome is often that a job uses ~5GB of space until it hits our global 1h timeout, even though that step does not have that much output. It's definitely Jenkins spinning on repeating the last few ~100 lines, although the process has already returned.

      As a workaround, if the above line is replaced with:

       

      sh "docker-compose logs containername | gzip > containername.log.gz"
archive("containername.log.gz")

      then we don't ever experience the hang.

      But for test output that isn't ideal.

      Note that this issue also happens if the command itself has a timeout:

      go test -timeout 10m

      It can still hang for an hour, spewing tons of logs that just repeat the last few pages.

       

          [JENKINS-49153] Sh step repeatedly spews its last n output lines until a timeout is hit

          Andrew Bayer added a comment -

          jglick svanoort - any thoughts on this? Do we have an effective maximum size for durable task logs?

          Andrew Bayer added a comment - jglick svanoort - any thoughts on this? Do we have an effective maximum size for durable task logs?

          Jesse Glick added a comment -

          Known bug in a code path slated to be rewritten for other reasons.

          Jesse Glick added a comment - Known bug in a code path slated to be rewritten for other reasons.

          Sam Van Oort added a comment -

          So much pain with classic logging.

          Sam Van Oort added a comment - So much pain with classic logging.

          John Zila added a comment -

          Thanks for the response! Should I be using a different type of logging?

          John Zila added a comment - Thanks for the response! Should I be using a different type of logging?

          Jesse Glick added a comment -

          No, there is known workaround in the current codebase.

          Jesse Glick added a comment - No, there is known workaround in the current codebase.

            Unassigned Unassigned
            jzila John Zila
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: