Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31376

Sauce connect intermmentantly fails to launch

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: sauce-ondemand-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.635, Debian GNU/Linux 6.0.10, Sauce OnDemand plugin 1.140
    • Similar Issues:

      Description

      Since upgrading Jenkins and sauce-ondemand we are having mostly failures with sauce connect launching. Occasionally sauce does launch correctly. Jenkins is running on Ubuntu 12.04, but we're launching sauce on the slave which is Debian 6.0.10. We are not using sauce on demand 1.142 because that was working even worse for us.

      Starting Sauce Connect on slave node using tunnel identifier: test-pull-requests-jsunit-tests-1446583886414
      Launching Sauce Connect on localhost
      03 Nov 20:51:04 - Sauce Connect 4.3.11, build 1757 2b421bb
      03 Nov 20:51:04 - Using CA certificate bundle /etc/ssl/certs/ca-certificates.crt.
      03 Nov 20:51:04 - Using CA certificate verify path /etc/ssl/certs.
      03 Nov 20:51:04 - Starting up; pid 7789
      03 Nov 20:51:04 - Command line arguments: /home/autotest/sc-4.3.11-linux/bin/sc -u jenkinsbuild -k **** -P 4445
      03 Nov 20:51:04 - Using no proxy for connecting to Sauce Labs REST API.
      03 Nov 20:51:04 - Resolving saucelabs.com to 162.222.75.243 took 1 ms.
      03 Nov 20:51:04 - Started scproxy on port 35287.
      03 Nov 20:51:04 - Please wait for 'you may start your tests' to start your tests.
      03 Nov 20:51:04 - Starting secure remote tunnel VM...
      03 Nov 20:51:09 - Secure remote tunnel VM provisioned.
      03 Nov 20:51:09 - Tunnel ID: 98cdb43fe69b49c48ef2098d332ceded
      03 Nov 20:51:09 - Secure remote tunnel VM is now: booting
      03 Nov 20:52:16 - Error bringing up tunnel VM.
      03 Nov 20:52:16 - Sauce Connect could not establish a connection.
      03 Nov 20:52:16 - Please check your firewall and proxy settings.
      03 Nov 20:52:16 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
      03 Nov 20:52:16 - Cleaning up.
      03 Nov 20:52:16 - Checking domain overlap for my domain sauce-connect.proxy, other tunnel domain sauce-connect.proxy
      03 Nov 20:52:16 - Overlapping domain: sauce-connect.proxy, shutting down tunnel 98cdb43fe69b49c48ef2098d332ceded.
      03 Nov 20:52:19 - Goodbye.
      Error launching Sauce Connect
      Flushing Sauce Connect Input Stream
      Flushing Sauce Connect Error Stream
      Closing Sauce Connect process
      FATAL: Error launching Sauce Connect
      com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
      at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320)
      at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:841)
      at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:777)
      at hudson.remoting.UserRequest.perform(UserRequest.java:121)
      at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      at ......remote call to pullrequests.www.dash.d1.usw1.opendns.com(Native Method)
      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
      at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
      at hudson.remoting.Channel.call(Channel.java:778)
      at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:315)
      at hudson.model.Build$BuildExecution.doRun(Build.java:156)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
      at hudson.model.Run.execute(Run.java:1741)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:98)
      at hudson.model.Executor.run(Executor.java:410)
      Notifying upstream projects of job completion
      Finished: FAILURE

        Attachments

          Activity

          mmerritt Michael Merritt created issue -
          mmerritt Michael Merritt made changes -
          Field Original Value New Value
          Description Since upgrading Jenkins and sauce connect we are having mostly failures with sauce connect launching. Occasionally sauce does launch correctly. Jenkins is running on Ubuntu 12.04, but we're launching sauce on the slave which is Debian 6.0.10.

          Starting Sauce Connect on slave node using tunnel identifier: test-pull-requests-jsunit-tests-1446583886414
          Launching Sauce Connect on localhost
          03 Nov 20:51:04 - Sauce Connect 4.3.11, build 1757 2b421bb
          03 Nov 20:51:04 - Using CA certificate bundle /etc/ssl/certs/ca-certificates.crt.
          03 Nov 20:51:04 - Using CA certificate verify path /etc/ssl/certs.
          03 Nov 20:51:04 - Starting up; pid 7789
          03 Nov 20:51:04 - Command line arguments: /home/autotest/sc-4.3.11-linux/bin/sc -u jenkinsbuild -k **** -P 4445
          03 Nov 20:51:04 - Using no proxy for connecting to Sauce Labs REST API.
          03 Nov 20:51:04 - Resolving saucelabs.com to 162.222.75.243 took 1 ms.
          03 Nov 20:51:04 - Started scproxy on port 35287.
          03 Nov 20:51:04 - Please wait for 'you may start your tests' to start your tests.
          03 Nov 20:51:04 - Starting secure remote tunnel VM...
          03 Nov 20:51:09 - Secure remote tunnel VM provisioned.
          03 Nov 20:51:09 - Tunnel ID: 98cdb43fe69b49c48ef2098d332ceded
          03 Nov 20:51:09 - Secure remote tunnel VM is now: booting
          03 Nov 20:52:16 - Error bringing up tunnel VM.
          03 Nov 20:52:16 - Sauce Connect could not establish a connection.
          03 Nov 20:52:16 - Please check your firewall and proxy settings.
          03 Nov 20:52:16 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
          03 Nov 20:52:16 - Cleaning up.
          03 Nov 20:52:16 - Checking domain overlap for my domain sauce-connect.proxy, other tunnel domain sauce-connect.proxy
          03 Nov 20:52:16 - Overlapping domain: sauce-connect.proxy, shutting down tunnel 98cdb43fe69b49c48ef2098d332ceded.
          03 Nov 20:52:19 - Goodbye.
          Error launching Sauce Connect
          Flushing Sauce Connect Input Stream
          Flushing Sauce Connect Error Stream
          Closing Sauce Connect process
          FATAL: Error launching Sauce Connect
          com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
          at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:841)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:777)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to pullrequests.www.dash.d1.usw1.opendns.com(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          at hudson.remoting.Channel.call(Channel.java:778)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:315)
          at hudson.model.Build$BuildExecution.doRun(Build.java:156)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
          at hudson.model.Run.execute(Run.java:1741)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:410)
          Notifying upstream projects of job completion
          Finished: FAILURE
          Since upgrading Jenkins and sauce-ondemand we are having mostly failures with sauce connect launching. Occasionally sauce does launch correctly. Jenkins is running on Ubuntu 12.04, but we're launching sauce on the slave which is Debian 6.0.10. We are not using sauce on demand 1.142 because that was working even worse for us.

          Starting Sauce Connect on slave node using tunnel identifier: test-pull-requests-jsunit-tests-1446583886414
          Launching Sauce Connect on localhost
          03 Nov 20:51:04 - Sauce Connect 4.3.11, build 1757 2b421bb
          03 Nov 20:51:04 - Using CA certificate bundle /etc/ssl/certs/ca-certificates.crt.
          03 Nov 20:51:04 - Using CA certificate verify path /etc/ssl/certs.
          03 Nov 20:51:04 - Starting up; pid 7789
          03 Nov 20:51:04 - Command line arguments: /home/autotest/sc-4.3.11-linux/bin/sc -u jenkinsbuild -k **** -P 4445
          03 Nov 20:51:04 - Using no proxy for connecting to Sauce Labs REST API.
          03 Nov 20:51:04 - Resolving saucelabs.com to 162.222.75.243 took 1 ms.
          03 Nov 20:51:04 - Started scproxy on port 35287.
          03 Nov 20:51:04 - Please wait for 'you may start your tests' to start your tests.
          03 Nov 20:51:04 - Starting secure remote tunnel VM...
          03 Nov 20:51:09 - Secure remote tunnel VM provisioned.
          03 Nov 20:51:09 - Tunnel ID: 98cdb43fe69b49c48ef2098d332ceded
          03 Nov 20:51:09 - Secure remote tunnel VM is now: booting
          03 Nov 20:52:16 - Error bringing up tunnel VM.
          03 Nov 20:52:16 - Sauce Connect could not establish a connection.
          03 Nov 20:52:16 - Please check your firewall and proxy settings.
          03 Nov 20:52:16 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
          03 Nov 20:52:16 - Cleaning up.
          03 Nov 20:52:16 - Checking domain overlap for my domain sauce-connect.proxy, other tunnel domain sauce-connect.proxy
          03 Nov 20:52:16 - Overlapping domain: sauce-connect.proxy, shutting down tunnel 98cdb43fe69b49c48ef2098d332ceded.
          03 Nov 20:52:19 - Goodbye.
          Error launching Sauce Connect
          Flushing Sauce Connect Input Stream
          Flushing Sauce Connect Error Stream
          Closing Sauce Connect process
          FATAL: Error launching Sauce Connect
          com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
          at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:841)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:777)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to pullrequests.www.dash.d1.usw1.opendns.com(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          at hudson.remoting.Channel.call(Channel.java:778)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:315)
          at hudson.model.Build$BuildExecution.doRun(Build.java:156)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
          at hudson.model.Run.execute(Run.java:1741)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:410)
          Notifying upstream projects of job completion
          Finished: FAILURE
          Hide
          mmerritt Michael Merritt added a comment -

          This might be related to sauce labs outage. Waiting to see.

          Show
          mmerritt Michael Merritt added a comment - This might be related to sauce labs outage. Waiting to see.
          Hide
          thecampbellcash Andrew Campbell added a comment -

          Michael Merritt More than likely this failure was from the outage, the stack trace above does not tell us much unfortunately so it is hard to say for sure. If you can replicate please let me know but if not I would say this was an issue with the service and not the plugin.

          Show
          thecampbellcash Andrew Campbell added a comment - Michael Merritt More than likely this failure was from the outage, the stack trace above does not tell us much unfortunately so it is hard to say for sure. If you can replicate please let me know but if not I would say this was an issue with the service and not the plugin.
          Hide
          mmerritt Michael Merritt added a comment -

          Closing, this appeared to be related to one of Sauce's outages.

          Show
          mmerritt Michael Merritt added a comment - Closing, this appeared to be related to one of Sauce's outages.
          mmerritt Michael Merritt made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]
          Hide
          mmerritt Michael Merritt added a comment -

          Also, we do have other failures with the plug-in sometimes not coming up. How do we provide the most info for debugging these issues? Thanks.

          Show
          mmerritt Michael Merritt added a comment - Also, we do have other failures with the plug-in sometimes not coming up. How do we provide the most info for debugging these issues? Thanks.
          Hide
          thecampbellcash Andrew Campbell added a comment -

          Michael Merritt The most important things would be the jenkins standard output logs, the Sauce Connect logs (if you are using Sauce Connect) and the steps to reproduce the issue.

          Show
          thecampbellcash Andrew Campbell added a comment - Michael Merritt The most important things would be the jenkins standard output logs, the Sauce Connect logs (if you are using Sauce Connect) and the steps to reproduce the issue.
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 166715 ] JNJira + In-Review [ 209401 ]

            People

            Assignee:
            kohsuke Kohsuke Kawaguchi
            Reporter:
            mmerritt Michael Merritt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: