Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31615

Sometimes the plugin leaves behind a stale PID file or zombie Sauce Connect process, should the plugin handle this?

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • sauce-ondemand-plugin
    • None

      If Sauce Connect fails to close properly (for whatever reason), should the plugin force close the process and clean up it's pidfile? I am inclined to say that it should but am not aware of what limitations or issues making such a change would cause.

          [JENKINS-31615] Sometimes the plugin leaves behind a stale PID file or zombie Sauce Connect process, should the plugin handle this?

          Anytime this could get fixed pls ? Thanks

          Suprinder Pujji added a comment - Anytime this could get fixed pls ? Thanks

          Gavin Mogan added a comment -

          Hi suprinder

          This is a really old issue that probably should have been closed now.

          Are you still having this issue? what version of the plugin are you using? What version of sauce connect?

          I talked to the sauce connect team, the only reason stale pid files should exist is if the process is hard killed (kill -9 in linux) or power cycled the machine.

          Gavin Mogan added a comment - Hi suprinder This is a really old issue that probably should have been closed now. Are you still having this issue? what version of the plugin are you using? What version of sauce connect? I talked to the sauce connect team, the only reason stale pid files should exist is if the process is hard killed (kill -9 in linux) or power cycled the machine.

          Gavin

          I get the below error and I am using Sauce Connect 1.144

          Started by user Suprinder Pujji
          [EnvInject] - Loading node environment variables.
          Building on master in workspace /home/sfdc/jenkins/jobs/testCaPadSuite/workspace
          > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
          > git config remote.origin.url git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git # timeout=10
          Fetching upstream changes from git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git
          > git --version # timeout=10
          using GIT_SSH to set credentials
          > git fetch --tags --progress git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git +refs/heads/:refs/remotes/origin/
          > git rev-parse refs/remotes/origin/capad^

          {commit} # timeout=10
          > git rev-parse refs/remotes/origin/origin/capad^{commit}

          # timeout=10
          Checking out Revision 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d (refs/remotes/origin/capad)
          > git config core.sparsecheckout # timeout=10
          > git checkout -f 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d
          > git rev-list 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d # timeout=10
          Starting pre-build for Sauce Labs plugin
          Starting Sauce Connect on master node using identifier: default
          Launching Sauce Connect on sm1-jenkins3-1-sfm.eng.sfdc.net
          10 Sep 00:29:18 - Sauce Connect 4.3.12, build 1788 abd6986
          10 Sep 00:29:18 - Using CA certificate bundle /etc/ssl/certs/ca-bundle.crt.
          10 Sep 00:29:18 - Using CA certificate verify path /etc/ssl/certs.
          10 Sep 00:29:18 - Starting up; pid 30453
          10 Sep 00:29:18 - Command line arguments: /home/sfdc/sc-4.3.12-linux/bin/sc -u SFDC -k **** -P 4445 -vv -l -
          10 Sep 00:29:18 - Using no proxy for connecting to Sauce Labs REST API.
          10 Sep 00:29:18 - Resolving saucelabs.com to 162.222.75.243 took 1 ms.
          10 Sep 00:29:48 - Started scproxy on port 35944.
          10 Sep 00:29:48 - Please wait for 'you may start your tests' to start your tests.
          10 Sep 00:29:48 - Starting secure remote tunnel VM...
          10 Sep 00:30:18 - GET https://saucelabs.com/rest/v1/SFDC/tunnels?full=1: Timeout was reached.
          10 Sep 00:30:18 - Error shutting down overlapping tunnels.
          10 Sep 00:30:18 - Sauce Connect could not establish a connection.
          10 Sep 00:30:18 - Please check your firewall and proxy settings.
          10 Sep 00:30:18 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
          10 Sep 00:30:18 - Cleaning up.
          10 Sep 00:30:48 - GET https://saucelabs.com/rest/v1/SFDC/tunnels?full=1: Timeout was reached.
          10 Sep 00:30:48 - Goodbye.
          Error launching Sauce Connect
          Flushing Sauce Connect Input Stream
          Flushing Sauce Connect Error Stream
          Closing Sauce Connect process
          FATAL: Error launching Sauce Connect
          com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
          at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:913)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:348)
          at hudson.model.Build$BuildExecution.doRun(Build.java:156)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
          at hudson.model.Run.execute(Run.java:1744)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:374)

          Suprinder Pujji added a comment - Gavin I get the below error and I am using Sauce Connect 1.144 Started by user Suprinder Pujji [EnvInject] - Loading node environment variables. Building on master in workspace /home/sfdc/jenkins/jobs/testCaPadSuite/workspace > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git # timeout=10 Fetching upstream changes from git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git > git --version # timeout=10 using GIT_SSH to set credentials > git fetch --tags --progress git@git.soma.salesforce.com:smart/SM_Selenium_Tests-1.git +refs/heads/ :refs/remotes/origin/ > git rev-parse refs/remotes/origin/capad^ {commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/capad^{commit} # timeout=10 Checking out Revision 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d (refs/remotes/origin/capad) > git config core.sparsecheckout # timeout=10 > git checkout -f 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d > git rev-list 6fa00fd90f2c94e2b7dfb5dc8b7de1956ba8ae6d # timeout=10 Starting pre-build for Sauce Labs plugin Starting Sauce Connect on master node using identifier: default Launching Sauce Connect on sm1-jenkins3-1-sfm.eng.sfdc.net 10 Sep 00:29:18 - Sauce Connect 4.3.12, build 1788 abd6986 10 Sep 00:29:18 - Using CA certificate bundle /etc/ssl/certs/ca-bundle.crt. 10 Sep 00:29:18 - Using CA certificate verify path /etc/ssl/certs. 10 Sep 00:29:18 - Starting up; pid 30453 10 Sep 00:29:18 - Command line arguments: /home/sfdc/sc-4.3.12-linux/bin/sc -u SFDC -k **** -P 4445 -vv -l - 10 Sep 00:29:18 - Using no proxy for connecting to Sauce Labs REST API. 10 Sep 00:29:18 - Resolving saucelabs.com to 162.222.75.243 took 1 ms. 10 Sep 00:29:48 - Started scproxy on port 35944. 10 Sep 00:29:48 - Please wait for 'you may start your tests' to start your tests. 10 Sep 00:29:48 - Starting secure remote tunnel VM... 10 Sep 00:30:18 - GET https://saucelabs.com/rest/v1/SFDC/tunnels?full=1: Timeout was reached. 10 Sep 00:30:18 - Error shutting down overlapping tunnels. 10 Sep 00:30:18 - Sauce Connect could not establish a connection. 10 Sep 00:30:18 - Please check your firewall and proxy settings. 10 Sep 00:30:18 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode. 10 Sep 00:30:18 - Cleaning up. 10 Sep 00:30:48 - GET https://saucelabs.com/rest/v1/SFDC/tunnels?full=1: Timeout was reached. 10 Sep 00:30:48 - Goodbye. Error launching Sauce Connect Flushing Sauce Connect Input Stream Flushing Sauce Connect Error Stream Closing Sauce Connect process FATAL: Error launching Sauce Connect com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320) at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:913) at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:348) at hudson.model.Build$BuildExecution.doRun(Build.java:156) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537) at hudson.model.Run.execute(Run.java:1744) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:374)

          So it says
          shutting down overlapping tunnels.
          10 Sep 00:30:18 - Sauce Connect could not establish a connection.

          So I am guessing there is stale pid which is still alive somewhere because of which no new builds succeeds. But not sure if this is correct assumption

          Suprinder Pujji added a comment - So it says shutting down overlapping tunnels. 10 Sep 00:30:18 - Sauce Connect could not establish a connection. So I am guessing there is stale pid which is still alive somewhere because of which no new builds succeeds. But not sure if this is correct assumption

          Gavin Mogan added a comment -

          okay cool, we'll look into it.

          i suspect once you finish upgrading to the jenkins plugin this will be delt with.

          Gavin Mogan added a comment - okay cool, we'll look into it. i suspect once you finish upgrading to the jenkins plugin this will be delt with.

          I upgraded to 1.145 but I get below error. Not sure what to do here

          Starting pre-build for Sauce Labs plugin
          Starting Sauce Connect on master node using identifier: testCaPadSuite-1473696246755
          Launching Sauce Connect on sm1-jenkins3-1-sfm.eng.sfdc.net
          Error launching Sauce Connect
          Flushing Sauce Connect Input Stream
          Flushing Sauce Connect Error Stream
          Closing Sauce Connect process
          FATAL: Error launching Sauce Connect
          com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
          at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:916)
          at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:347)
          at hudson.model.Build$BuildExecution.doRun(Build.java:156)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
          at hudson.model.Run.execute(Run.java:1744)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:374)
          An attempt to send an e-mail to empty list of recipients, ignored.
          Finished: FAILURE

          Suprinder Pujji added a comment - I upgraded to 1.145 but I get below error. Not sure what to do here Starting pre-build for Sauce Labs plugin Starting Sauce Connect on master node using identifier: testCaPadSuite-1473696246755 Launching Sauce Connect on sm1-jenkins3-1-sfm.eng.sfdc.net Error launching Sauce Connect Flushing Sauce Connect Input Stream Flushing Sauce Connect Error Stream Closing Sauce Connect process FATAL: Error launching Sauce Connect com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:320) at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:916) at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:347) at hudson.model.Build$BuildExecution.doRun(Build.java:156) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537) at hudson.model.Run.execute(Run.java:1744) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:374) An attempt to send an e-mail to empty list of recipients, ignored. Finished: FAILURE

          Gavin Mogan added a comment -

          I'm mobile today and at a conference all week, so you'll have quicker support opening a saucelabs support ticket.

          I believe our latest version is 1.155

          In Addition, does your log say anything? All that stacktrace says is it had trouble starting up

          Gavin Mogan added a comment - I'm mobile today and at a conference all week, so you'll have quicker support opening a saucelabs support ticket. I believe our latest version is 1.155 In Addition, does your log say anything? All that stacktrace says is it had trouble starting up

          suprinder the error you are seeing is not related to pid file locking. If it were, you would see the following in the logs as the error code for a PID file collision is rather explicit.

          12 Sep 12:28:33 - Sauce Connect 4.3.16, build 2399 c7e5fec
          12 Sep 12:28:33 - can't lock pidfile currentA. Please check if Sauce Connect is already running.
          12 Sep 12:28:33 - Error creating pidfile currentA.
          

          So no, the bug here is unrelated to the issue you are facing and I would advice contact Sauce Labs support as halkeye suggested. From the logs you look to be having a connectivity issue of some kind.

          Andrew Campbell added a comment - suprinder the error you are seeing is not related to pid file locking. If it were, you would see the following in the logs as the error code for a PID file collision is rather explicit. 12 Sep 12:28:33 - Sauce Connect 4.3.16, build 2399 c7e5fec 12 Sep 12:28:33 - can't lock pidfile currentA. Please check if Sauce Connect is already running. 12 Sep 12:28:33 - Error creating pidfile currentA. So no, the bug here is unrelated to the issue you are facing and I would advice contact Sauce Labs support as halkeye suggested. From the logs you look to be having a connectivity issue of some kind.

          Sure Andrew/Gavin. Thanks for your prompt replies.

          Suprinder Pujji added a comment - Sure Andrew/Gavin. Thanks for your prompt replies.

            yfangsl Yeh Fang
            thecampbellcash Andrew Campbell
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: