Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25026

Console output stops populating after several seconds

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • core
    • Jenkins 1.582 from Bitnami, running on AWS in m3.medium instance type
      Server OS: Ubuntu 14.04 64bit
      JDK: Oracle JDK 1.7.0_67
      Browser: Chrome 37 64bit

      Console output is populating log in browser window, but after several seconds stops. User has to refresh the page but it happens again. It looks like populating works only when log grown quickly and stops when job doesn't log for a while.

      It seems that problem is on the server side, browser repeats AJAX requests to progressiveHtml path and all of them have "X-More-Data: true" response header, but the last one hasn't (wrongly, because job logging on server side continues).

      Latest version I have seen this working: 1.466

          [JENKINS-25026] Console output stops populating after several seconds

          Daniel Beck added a comment -

          You marked this 'regression' but do not mention which version was known to work.

          Please provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Is this really about console-tail plugin, or did you just not read the note that says issues with Jenkins itself should use the 'core' component?

          Daniel Beck added a comment - You marked this 'regression' but do not mention which version was known to work. Please provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue Is this really about console-tail plugin, or did you just not read the note that says issues with Jenkins itself should use the 'core' component?

          Sorry for unclear information, updated components and added previous version where I have seen this working.

          Martin Schayna added a comment - Sorry for unclear information, updated components and added previous version where I have seen this working.

          Daniel Beck added a comment -

          Does the spinner (or "indeterminate progress indicator") at the bottom of the log disappear as well? Does the browser stop sending ajax update requests?

          IIRC X-More-Data only applies when there's more data, so if logging is quiet, it'll not have more data for a while. The client should still continue polling for changes.

          Daniel Beck added a comment - Does the spinner (or "indeterminate progress indicator") at the bottom of the log disappear as well? Does the browser stop sending ajax update requests? IIRC X-More-Data only applies when there's more data, so if logging is quiet, it'll not have more data for a while. The client should still continue polling for changes.

          The spinner disappears and the browser stops polling for changes.

          There is a javascript function fetchNext(...) in the console page which does polling. The code is pretty straightforward, it doesn't re-instantiate setTimeout when X-More-Data header is missing in the response and it stops polling (I think) forever:

              if (rsp.getResponseHeader("X-More-Data") == "true")
                  setTimeout(function() {
                      fetchNext(e, href);
                  }, 1000);
              else
                  $("spinner").style.display = "none";
          

          Martin Schayna added a comment - The spinner disappears and the browser stops polling for changes. There is a javascript function fetchNext(...) in the console page which does polling. The code is pretty straightforward, it doesn't re-instantiate setTimeout when X-More-Data header is missing in the response and it stops polling (I think) forever: if (rsp.getResponseHeader( "X-More-Data" ) == " true " ) setTimeout(function() { fetchNext(e, href); }, 1000); else $( "spinner" ).style.display = "none" ;

          Daniel Beck added a comment -

          Daniel Beck added a comment - Stapler sets X-More-Data if the file being read hasn't been completed yet though, so I was simply wrong about the behavior of this header: https://github.com/stapler/stapler/blob/master/core/src/main/java/org/kohsuke/stapler/framework/io/LargeText.java#L266 The Jenkins parts also look reasonable: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L1439 https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L501 https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L234 Does this occur for all jobs? Are you doing weird things in Groovy scripts?

          It happens for all job types, I have pure Java Maven jobs (e.g. "mvn clean compile test") and some Grails jobs ("grails prod war"). No weird things I hope, I'll try to attach screencast.

          Martin Schayna added a comment - It happens for all job types, I have pure Java Maven jobs (e.g. "mvn clean compile test") and some Grails jobs ("grails prod war"). No weird things I hope, I'll try to attach screencast.

          Screencast demonstrating bug, currently running job was Maven test goal.

          Martin Schayna added a comment - Screencast demonstrating bug, currently running job was Maven test goal.

          Daniel Beck added a comment -

          The screencast is very nice, thank you. And the behavior demonstrated is very weird.

          Does this happen for all jobs, all builds?

          Is Jenkins accessed through a reverse proxy?

          Daniel Beck added a comment - The screencast is very nice, thank you. And the behavior demonstrated is very weird. Does this happen for all jobs, all builds? Is Jenkins accessed through a reverse proxy?

          Jimmy Gleason added a comment -

          I'm having this issue as well. Has anyone made any progress toward finding a fix for this issue?

          Jimmy Gleason added a comment - I'm having this issue as well. Has anyone made any progress toward finding a fix for this issue?

          I have the same problem but it does not seem to be server related... in IE update is working as expected but not in FF. So it seems to be an issue on the client side.

          Christoph Vogtländer added a comment - I have the same problem but it does not seem to be server related... in IE update is working as expected but not in FF. So it seems to be an issue on the client side.

          In my case it doesn't work in all browsers (tested on Chrome 38, FF 33, IE 11).

          Polling stops when the job doesn't write to console for about a half seconds or so. When the job is very verbose, it seems to work.

          @Daniel:
          Yes, it happens for all jobs that are not so verbose.
          Yes, Jenkins is accessed through a reverse proxy, which is the Apache server in front of Tomcat on the same machine.

          Martin Schayna added a comment - In my case it doesn't work in all browsers (tested on Chrome 38, FF 33, IE 11). Polling stops when the job doesn't write to console for about a half seconds or so. When the job is very verbose, it seems to work. @Daniel: Yes, it happens for all jobs that are not so verbose. Yes, Jenkins is accessed through a reverse proxy, which is the Apache server in front of Tomcat on the same machine.

          Daniel Beck added a comment -

          Is the X-More-Data header set for all responses?

          Daniel Beck added a comment - Is the X-More-Data header set for all responses?

          When polling, typically for couple of seconds from refresh, there are X-More-Data and X-Text-Size headers present in all responses to 'progressiveHtml' requests. Responses look like this (only Content-Length, Date, X-ConsoleAnnotator and X-Text-Size headers differs between requests):

          Cache-Control:max-age=0, no-cache
          Connection:Keep-Alive
          Content-Encoding:gzip
          Content-Length:151
          Content-Type:text/html;charset=UTF-8
          Date:Thu, 30 Oct 2014 12:03:39 GMT
          Keep-Alive:timeout=5, max=91
          Server:Apache
          Vary:Accept-Encoding
          X-ConsoleAnnotator:7MYs6ttMMoModS4EVo7TUCG79S3ndnWRJpG1rPfajEK/KFxnFhMBC/D3zeM0xiuEYJDHZ4w/GiQAwBpuiCiRbTshPhZm+Jjau4J0XfzamZptCfN++vXbE0qLVLZbcpyl
          X-Frame-Options:SAMEORIGIN
          X-Mod-Pagespeed:1.7.30.4-
          X-More-Data:true
          X-Text-Size:2625
          

          The last response, after which the page stops polling, looks like this:

          Cache-Control:max-age=0, no-cache
          Connection:Keep-Alive
          Content-Length:0
          Content-Type:text/html;charset=UTF-8
          Date:Thu, 30 Oct 2014 12:03:41 GMT
          Keep-Alive:timeout=5, max=89
          Server:Apache
          X-Frame-Options:SAMEORIGIN
          

          Martin Schayna added a comment - When polling, typically for couple of seconds from refresh, there are X-More-Data and X-Text-Size headers present in all responses to 'progressiveHtml' requests. Responses look like this (only Content-Length, Date, X-ConsoleAnnotator and X-Text-Size headers differs between requests): Cache-Control:max-age=0, no-cache Connection:Keep-Alive Content-Encoding:gzip Content-Length:151 Content-Type:text/html;charset=UTF-8 Date:Thu, 30 Oct 2014 12:03:39 GMT Keep-Alive:timeout=5, max=91 Server:Apache Vary:Accept-Encoding X-ConsoleAnnotator:7MYs6ttMMoModS4EVo7TUCG79S3ndnWRJpG1rPfajEK/KFxnFhMBC/D3zeM0xiuEYJDHZ4w/GiQAwBpuiCiRbTshPhZm+Jjau4J0XfzamZptCfN++vXbE0qLVLZbcpyl X-Frame-Options:SAMEORIGIN X-Mod-Pagespeed:1.7.30.4- X-More-Data: true X-Text-Size:2625 The last response, after which the page stops polling, looks like this: Cache-Control:max-age=0, no-cache Connection:Keep-Alive Content-Length:0 Content-Type:text/html;charset=UTF-8 Date:Thu, 30 Oct 2014 12:03:41 GMT Keep-Alive:timeout=5, max=89 Server:Apache X-Frame-Options:SAMEORIGIN

          Chris Z added a comment -

          I must admin that this is frustrating for me and colleagues. Every time refreshing when some process is computing for while can get for example me to madness.

          Chris Z added a comment - I must admin that this is frustrating for me and colleagues. Every time refreshing when some process is computing for while can get for example me to madness.

          Daniel Beck added a comment -

          The last response, after which the page stops polling, looks like this:

          What's the HTTP response code for this response?

          Daniel Beck added a comment - The last response, after which the page stops polling, looks like this: What's the HTTP response code for this response?

          A colleague of mine installed a new server with Jenkins in the same version (1.582) from scratch and the console is polling OK now. Sorry, I cannot reproduce it, apparently it happens under certain conditions only.

          Martin Schayna added a comment - A colleague of mine installed a new server with Jenkins in the same version (1.582) from scratch and the console is polling OK now. Sorry, I cannot reproduce it, apparently it happens under certain conditions only.

          Daniel Beck added a comment -

          Martin: Clearly, otherwise I'd be able to reproduce the problem

          Would be interesting to know whether all affected instances run behind Apache.

          Daniel Beck added a comment - Martin: Clearly, otherwise I'd be able to reproduce the problem Would be interesting to know whether all affected instances run behind Apache.

          Thomas Muguet added a comment -

          HTTP requests and corresponding responses

          Thomas Muguet added a comment - HTTP requests and corresponding responses

          Thomas Muguet added a comment -

          Same here with fresh Bitnami stacks (1.575 and 1.585) and updated (currently 1.580.2), running behing Apache (2.4.10) + Tomcat (7.0.55) on Ubuntu 14.04.1 64bit.

          See attachement for the 3 last requests and corresponding responses.
          Let me know if you need more info.

          Thomas Muguet added a comment - Same here with fresh Bitnami stacks (1.575 and 1.585) and updated (currently 1.580.2), running behing Apache (2.4.10) + Tomcat (7.0.55) on Ubuntu 14.04.1 64bit. See attachement for the 3 last requests and corresponding responses. Let me know if you need more info.

          Daniel Beck added a comment -

          Interesting attachment.

          Here's the problem, it looks like it's not Jenkins who's responding, or it's an irregular response. There's no code path for the console log updates that does not set an X-Something header.

          https://github.com/stapler/stapler/blob/master/core/src/main/java/org/kohsuke/stapler/framework/io/LargeText.java#L264

          Check your Jenkins log, e.g. on the UI at the /log/all URL. Something may throw an exception to get out of the regular flow. Maybe this is JENKINS-23971?

          Daniel Beck added a comment - Interesting attachment. Here's the problem, it looks like it's not Jenkins who's responding, or it's an irregular response. There's no code path for the console log updates that does not set an X-Something header. https://github.com/stapler/stapler/blob/master/core/src/main/java/org/kohsuke/stapler/framework/io/LargeText.java#L264 Check your Jenkins log, e.g. on the UI at the /log/all URL. Something may throw an exception to get out of the regular flow. Maybe this is JENKINS-23971 ?

          Thomas Muguet added a comment -

          Nothing in the logs.
          Here are the logs that are enabled by the way:

          org.acegisecurity.userdetails ALL
          org.acegisecurity.ldap.LdapUtils ALL
          org.acegisecurity.ldap.search ALL
          org.acegisecurity.providers.ldap ALL
          org.acegisecurity.ldap ALL
          org.apache.sshd WARNING
          hudson.plugins.git ALL
          org.jenkinsci.plugins.gitclient ALL
          org.apache.catalina.core.ContainerBase.[Catalina].[localhost] INFO
          jenkins.plugins.git ALL
          org.acegisecurity.providers.ldap.authenticator ALL

          Thomas Muguet added a comment - Nothing in the logs. Here are the logs that are enabled by the way: org.acegisecurity.userdetails ALL org.acegisecurity.ldap.LdapUtils ALL org.acegisecurity.ldap.search ALL org.acegisecurity.providers.ldap ALL org.acegisecurity.ldap ALL org.apache.sshd WARNING hudson.plugins.git ALL org.jenkinsci.plugins.gitclient ALL org.apache.catalina.core.ContainerBase. [Catalina] . [localhost] INFO jenkins.plugins.git ALL org.acegisecurity.providers.ldap.authenticator ALL

          Daniel Beck added a comment -

          The interesting line is the one without logger name, just log level.

          Daniel Beck added a comment - The interesting line is the one without logger name, just log level.

          Thomas Muguet added a comment -

          Thanks for your quick answer.

          I guess I won't be really helpful because I did not have a default log level (it seems it is removed on startup), and setting it to ALL does not produce any more info in the logs.

          Thomas Muguet added a comment - Thanks for your quick answer. I guess I won't be really helpful because I did not have a default log level (it seems it is removed on startup), and setting it to ALL does not produce any more info in the logs.

          Daniel Beck added a comment -

          Is snyone experiencing this issue...

          • Without Apache in front of Jenkins?
          • Without Tomcat running Jenkins?
          • On a box that's not Bitnami?

          Daniel Beck added a comment - Is snyone experiencing this issue... Without Apache in front of Jenkins? Without Tomcat running Jenkins? On a box that's not Bitnami?

          Daniel Beck added a comment -

          Cannot reproduce the problem on the VMware VM bitnami-jenkins-1.596-0-ubuntu-14.04 I just downloaded. Tested with freestyle project executing the following build script:

          for I in $( seq 0 1000 ) ; do
            sleep 1
            echo JENKINS-25026 $I
          done

          The console output keeps updating.

          Neither Chrome 39 nor Firefox 34.0.5 on OS X show the reported issue when accessing the mentioned instance.

          Any console annotators enabled, like Timestamper, or Ant plugin or Gradle plugin adding a list of executed targets/tasks?

          Daniel Beck added a comment - Cannot reproduce the problem on the VMware VM bitnami-jenkins-1.596-0-ubuntu-14.04 I just downloaded. Tested with freestyle project executing the following build script: for I in $( seq 0 1000 ) ; do sleep 1 echo JENKINS-25026 $I done The console output keeps updating. Neither Chrome 39 nor Firefox 34.0.5 on OS X show the reported issue when accessing the mentioned instance. Any console annotators enabled, like Timestamper, or Ant plugin or Gradle plugin adding a list of executed targets/tasks?

          Thomas Muguet added a comment -

          When using a freestyle project executing the script you gave (without any console annotators), it seems to works. But putting the sleep value at 2 fails.
          I have a bunch of console annotators installed though, I will try to check that if you still can't reproduce this issue with a greater sleep value.

          Thomas Muguet added a comment - When using a freestyle project executing the script you gave (without any console annotators), it seems to works. But putting the sleep value at 2 fails. I have a bunch of console annotators installed though, I will try to check that if you still can't reproduce this issue with a greater sleep value.

          Daniel Beck added a comment -

          Reproduced with sleep 3. Investigating...

          Daniel Beck added a comment - Reproduced with sleep 3 . Investigating...

          Daniel Beck added a comment -

          And that's why you don't use some random vendor's preconfigured shit. It's broken.

          Disable mod_pagespeed (whatever the hell that is) in Apache and things start working.

          What a waste of time.

          Daniel Beck added a comment - And that's why you don't use some random vendor's preconfigured shit. It's broken. Disable mod_pagespeed (whatever the hell that is) in Apache and things start working. What a waste of time.

          Thomas Muguet added a comment -

          Thank you very much for your help. It's always good to know that this configuration is broken.

          Thomas Muguet added a comment - Thank you very much for your help. It's always good to know that this configuration is broken.

            Unassigned Unassigned
            mschayna Martin Schayna
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: