Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37206

ATH occasionally hangs under load - looks like an SSE client connect issue (timeout?)

XMLWordPrintable

    • 1.0-beta-1, pacific, atlantic, 1.0-b05/b-06

      Only seems to happen when the CI server is under a lot of load. Guess .... the SSE client connect fails (maybe due to a timeout) and doesn't handle it properly, resulting in the ATH test blocking waiting for an event that never arrives.

      Here's the error log I've seen ...

      2016-08-04 17:42:51,212 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - ========================================
      2016-08-04 17:42:51,216 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - SSE Gateway error to http://127.0.0.1:59010/sse-gateway/configure?batchId=4: 
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - { UNSENT: 0,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   OPENED: 1,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   HEADERS_RECEIVED: 2,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   LOADING: 3,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   DONE: 4,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   readyState: 4,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   onreadystatechange: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseText: '{"status":"error","message":"No \'subscribe\' or \'unsubscribe\' configurations provided in configuration request."}',
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseXML: '',
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   status: 400,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   statusText: null,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   withCredentials: false,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   open: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setDisableHeaderCheck: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setRequestHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getResponseHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getAllResponseHeaders: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getRequestHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   send: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   handleError: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   abort: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   addEventListener: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   removeEventListener: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   dispatchEvent: [Function] }
      master59010|Aug 04, 2016 5:42:52 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:42:52,068 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      2016-08-04 17:42:52,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
      master59010|Aug 04, 2016 5:43:23 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:43:23,071 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      2016-08-04 17:43:23,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
      master59010|Aug 04, 2016 5:43:54 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:43:54,076 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      

      From https://ci.blueocean.io/job/Acceptance-Tests/144/consoleFull

      Although ... https://ci.blueocean.io/job/Acceptance-Tests/145/consoleText gets into what looks like the same blocked state, but without the previous SSE connect failure error. Hmmm ... I wonder is something causing Nightwatch to somehow not close out the previous test suite properly and then block not starting the next.

            tfennelly Tom FENNELLY
            tfennelly Tom FENNELLY
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: