[JENKINS-37206] ATH occasionally hangs under load - looks like an SSE client connect issue (timeout?) - Jenkins Jira

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: blueocean-plugin
Labels:
- technical-debt

Similar Issues:
Powered by SuggestiMate

Show
Epic Link:
Acceptance Tests
Sprint:
1.0-beta-1, pacific, atlantic, 1.0-b05/b-06

Only seems to happen when the CI server is under a lot of load. Guess .... the SSE client connect fails (maybe due to a timeout) and doesn't handle it properly, resulting in the ATH test blocking waiting for an event that never arrives.

Here's the error log I've seen ...

2016-08-04 17:42:51,212 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - [0;35m========================================[0m
2016-08-04 17:42:51,216 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - SSE Gateway error to http://127.0.0.1:59010/sse-gateway/configure?batchId=4: 
2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - { UNSENT: 0,
2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   OPENED: 1,
2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   HEADERS_RECEIVED: 2,
2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   LOADING: 3,
2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   DONE: 4,
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   readyState: 4,
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   onreadystatechange: [Function],
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseText: '{"status":"error","message":"No \'subscribe\' or \'unsubscribe\' configurations provided in configuration request."}',
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseXML: '',
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   status: 400,
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   statusText: null,
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   withCredentials: false,
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   open: [Function],
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setDisableHeaderCheck: [Function],
2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setRequestHeader: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getResponseHeader: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getAllResponseHeaders: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getRequestHeader: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   send: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   handleError: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   abort: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   addEventListener: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   removeEventListener: [Function],
2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   dispatchEvent: [Function] }
master59010|Aug 04, 2016 5:42:52 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
2016-08-04 17:42:52,068 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
2016-08-04 17:42:52,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
master59010|Aug 04, 2016 5:43:23 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
2016-08-04 17:43:23,071 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
2016-08-04 17:43:23,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
master59010|Aug 04, 2016 5:43:54 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
2016-08-04 17:43:54,076 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.

From https://ci.blueocean.io/job/Acceptance-Tests/144/consoleFull

Although ... https://ci.blueocean.io/job/Acceptance-Tests/145/consoleText gets into what looks like the same blocked state, but without the previous SSE connect failure error. Hmmm ... I wonder is something causing Nightwatch to somehow not close out the previous test suite properly and then block not starting the next.

SCM/JIRA link daemon added a comment - 2016-10-13 08:55

Code changed in jenkins
User: Tom Fennelly
Path:
package.json
src/main/js/api/sse.js
src/main/js/globals.js
http://jenkins-ci.org/commit/blueocean-acceptance-test/125172d45887d4d8075864a0e265e2004d0de1c4
Log:
[FIX JENKINS-37206] Updated to use newer SSE client for stability (#61)

Updated to use newer SSE client

Updated to use newer SSE client

SCM/JIRA link daemon added a comment - 2016-10-13 08:55 Code changed in jenkins User: Tom Fennelly Path: package.json src/main/js/api/sse.js src/main/js/globals.js http://jenkins-ci.org/commit/blueocean-acceptance-test/125172d45887d4d8075864a0e265e2004d0de1c4 Log: [FIX JENKINS-37206] Updated to use newer SSE client for stability (#61) Updated to use newer SSE client Updated to use newer SSE client

Jenkins

Details

Description

Attachments

Activity

[JENKINS-37206] ATH occasionally hangs under load - looks like an SSE client connect issue (timeout?)

Collapse comment: SCM/JIRA link daemon added a comment - 2016-10-13 08:55

Expand comment: SCM/JIRA link daemon added a comment - 2016-10-13 08:55

People

Dates