-
Bug
-
Resolution: Not A Defect
-
Major
-
Jenkins 1.582 from Bitnami, running on AWS in m3.medium instance type
Server OS: Ubuntu 14.04 64bit
JDK: Oracle JDK 1.7.0_67
Browser: Chrome 37 64bit
-
Powered by SuggestiMate
Console output is populating log in browser window, but after several seconds stops. User has to refresh the page but it happens again. It looks like populating works only when log grown quickly and stops when job doesn't log for a while.
It seems that problem is on the server side, browser repeats AJAX requests to progressiveHtml path and all of them have "X-More-Data: true" response header, but the last one hasn't (wrongly, because job logging on server side continues).
Latest version I have seen this working: 1.466
[JENKINS-25026] Console output stops populating after several seconds
Sorry for unclear information, updated components and added previous version where I have seen this working.
Does the spinner (or "indeterminate progress indicator") at the bottom of the log disappear as well? Does the browser stop sending ajax update requests?
IIRC X-More-Data only applies when there's more data, so if logging is quiet, it'll not have more data for a while. The client should still continue polling for changes.
The spinner disappears and the browser stops polling for changes.
There is a javascript function fetchNext(...) in the console page which does polling. The code is pretty straightforward, it doesn't re-instantiate setTimeout when X-More-Data header is missing in the response and it stops polling (I think) forever:
if (rsp.getResponseHeader("X-More-Data") == "true") setTimeout(function() { fetchNext(e, href); }, 1000); else $("spinner").style.display = "none";
Stapler sets X-More-Data if the file being read hasn't been completed yet though, so I was simply wrong about the behavior of this header:
https://github.com/stapler/stapler/blob/master/core/src/main/java/org/kohsuke/stapler/framework/io/LargeText.java#L266
The Jenkins parts also look reasonable:
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L1439
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L501
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Run.java#L234
Does this occur for all jobs? Are you doing weird things in Groovy scripts?
It happens for all job types, I have pure Java Maven jobs (e.g. "mvn clean compile test") and some Grails jobs ("grails prod war"). No weird things I hope, I'll try to attach screencast.
Screencast demonstrating bug, currently running job was Maven test goal.
The screencast is very nice, thank you. And the behavior demonstrated is very weird.
Does this happen for all jobs, all builds?
Is Jenkins accessed through a reverse proxy?
I'm having this issue as well. Has anyone made any progress toward finding a fix for this issue?
I have the same problem but it does not seem to be server related... in IE update is working as expected but not in FF. So it seems to be an issue on the client side.
In my case it doesn't work in all browsers (tested on Chrome 38, FF 33, IE 11).
Polling stops when the job doesn't write to console for about a half seconds or so. When the job is very verbose, it seems to work.
@Daniel:
Yes, it happens for all jobs that are not so verbose.
Yes, Jenkins is accessed through a reverse proxy, which is the Apache server in front of Tomcat on the same machine.
When polling, typically for couple of seconds from refresh, there are X-More-Data and X-Text-Size headers present in all responses to 'progressiveHtml' requests. Responses look like this (only Content-Length, Date, X-ConsoleAnnotator and X-Text-Size headers differs between requests):
Cache-Control:max-age=0, no-cache
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:151
Content-Type:text/html;charset=UTF-8
Date:Thu, 30 Oct 2014 12:03:39 GMT
Keep-Alive:timeout=5, max=91
Server:Apache
Vary:Accept-Encoding
X-ConsoleAnnotator:7MYs6ttMMoModS4EVo7TUCG79S3ndnWRJpG1rPfajEK/KFxnFhMBC/D3zeM0xiuEYJDHZ4w/GiQAwBpuiCiRbTshPhZm+Jjau4J0XfzamZptCfN++vXbE0qLVLZbcpyl
X-Frame-Options:SAMEORIGIN
X-Mod-Pagespeed:1.7.30.4-
X-More-Data:true
X-Text-Size:2625
The last response, after which the page stops polling, looks like this:
Cache-Control:max-age=0, no-cache Connection:Keep-Alive Content-Length:0 Content-Type:text/html;charset=UTF-8 Date:Thu, 30 Oct 2014 12:03:41 GMT Keep-Alive:timeout=5, max=89 Server:Apache X-Frame-Options:SAMEORIGIN
I must admin that this is frustrating for me and colleagues. Every time refreshing when some process is computing for while can get for example me to madness.
The last response, after which the page stops polling, looks like this:
What's the HTTP response code for this response?
A colleague of mine installed a new server with Jenkins in the same version (1.582) from scratch and the console is polling OK now. Sorry, I cannot reproduce it, apparently it happens under certain conditions only.
Martin: Clearly, otherwise I'd be able to reproduce the problem
Would be interesting to know whether all affected instances run behind Apache.
Same here with fresh Bitnami stacks (1.575 and 1.585) and updated (currently 1.580.2), running behing Apache (2.4.10) + Tomcat (7.0.55) on Ubuntu 14.04.1 64bit.
See attachement for the 3 last requests and corresponding responses.
Let me know if you need more info.
Interesting attachment.
Here's the problem, it looks like it's not Jenkins who's responding, or it's an irregular response. There's no code path for the console log updates that does not set an X-Something header.
Check your Jenkins log, e.g. on the UI at the /log/all URL. Something may throw an exception to get out of the regular flow. Maybe this is JENKINS-23971?
Nothing in the logs.
Here are the logs that are enabled by the way:
org.acegisecurity.userdetails | ALL |
org.acegisecurity.ldap.LdapUtils | ALL |
org.acegisecurity.ldap.search | ALL |
org.acegisecurity.providers.ldap | ALL |
org.acegisecurity.ldap | ALL |
org.apache.sshd | WARNING |
hudson.plugins.git | ALL |
org.jenkinsci.plugins.gitclient | ALL |
org.apache.catalina.core.ContainerBase.[Catalina].[localhost] | INFO |
jenkins.plugins.git | ALL |
org.acegisecurity.providers.ldap.authenticator | ALL |
Thanks for your quick answer.
I guess I won't be really helpful because I did not have a default log level (it seems it is removed on startup), and setting it to ALL does not produce any more info in the logs.
Is snyone experiencing this issue...
- Without Apache in front of Jenkins?
- Without Tomcat running Jenkins?
- On a box that's not Bitnami?
Cannot reproduce the problem on the VMware VM bitnami-jenkins-1.596-0-ubuntu-14.04 I just downloaded. Tested with freestyle project executing the following build script:
for I in $( seq 0 1000 ) ; do sleep 1 echo JENKINS-25026 $I done
The console output keeps updating.
Neither Chrome 39 nor Firefox 34.0.5 on OS X show the reported issue when accessing the mentioned instance.
Any console annotators enabled, like Timestamper, or Ant plugin or Gradle plugin adding a list of executed targets/tasks?
When using a freestyle project executing the script you gave (without any console annotators), it seems to works. But putting the sleep value at 2 fails.
I have a bunch of console annotators installed though, I will try to check that if you still can't reproduce this issue with a greater sleep value.
And that's why you don't use some random vendor's preconfigured shit. It's broken.
Disable mod_pagespeed (whatever the hell that is) in Apache and things start working.
What a waste of time.
Thank you very much for your help. It's always good to know that this configuration is broken.
You marked this 'regression' but do not mention which version was known to work.
Please provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue
Is this really about console-tail plugin, or did you just not read the note that says issues with Jenkins itself should use the 'core' component?