Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30064

Optionally retry Sauce Connect failures when not related to credentials

XMLWordPrintable

      It would be optimal to have a retry checkbox in the sauce-ondemand-plugin with optionally a max retries count so when sauce connects fails it retries, I found that when running multiple tests in parallel cause the tunnel to fail from time to time, always using the same credentials and the same Jenkins slave. I can also recreate in a docker container and with a different Sauce Labs account.

      The reason to incorporate this retry mechanism in the plugin is because when the flaky tunnel issue happens the tests fails which is unfortunate.
      Without the retry in the plugin there is no option but to stop using it and open the tunnel via shell scripting and implement the retry in our own scripts however this could be useful for other users.

      Sample output failure

      Starting pre-build for Sauce Labs plugin
Starting Sauce Connect on master node using identifier: "sauceusername-19"
Launching Sauce Connect on ip-********.z.aws
20 Aug 10:57:12 - Sauce Connect 4.3.11, build 1757 2b421bb
20 Aug 10:57:12 - Using CA certificate bundle /etc/ssl/certs/ca-certificates.crt.
20 Aug 10:57:12 - Using CA certificate verify path /etc/ssl/certs.
20 Aug 10:57:12 - Starting up; pid 10545
20 Aug 10:57:12 - Command line arguments: /var/lib/jenkins/sc-4.3.11-linux/bin/sc -u sauceusername -k **** -P 4445 --tunnel-identifier "sauceusername-19"
20 Aug 10:57:12 - Using no proxy for connecting to Sauce Labs REST API.
20 Aug 10:57:12 - Resolving saucelabs.com to 162.222.73.28 took 1 ms.
20 Aug 10:57:12 - Started scproxy on port 40756.
20 Aug 10:57:12 - Please wait for 'you may start your tests' to start your tests.
20 Aug 10:57:12 - Starting secure remote tunnel VM...
20 Aug 10:57:15 - POST https://saucelabs.com/rest/v1/sauceusername/tunnels: Failure when receiving data from the peer.
20 Aug 10:57:15 - Error creating tunnel.
20 Aug 10:57:15 - Sauce Connect could not establish a connection.
20 Aug 10:57:15 - Please check your firewall and proxy settings.
20 Aug 10:57:15 - You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
20 Aug 10:57:15 - Cleaning up.
20 Aug 10:57:18 - Goodbye.

        1. jenkins_tunnel.log
          3 kB

          [JENKINS-30064] Optionally retry Sauce Connect failures when not related to credentials

          Val Redchenko added a comment -

          This would be a great improvement in my opinion.

          Val Redchenko added a comment - This would be a great improvement in my opinion.

          Leo Gallucci added a comment -

          For now I'm no longer using the plugin and connecting through shell scripting inside my docker container.

          How to install it:
          https://github.com/elgalu/docker-selenium/blob/95bd73db/Dockerfile#L498

          How to retry via `$SAUCE_TUNNEL_MAX_RETRY_ATTEMPTS`:
          https://github.com/elgalu/docker-selenium/blob/95bd73db/saucelabs/bin/start-saucelabs.sh#L41

          Leo Gallucci added a comment - For now I'm no longer using the plugin and connecting through shell scripting inside my docker container. How to install it: https://github.com/elgalu/docker-selenium/blob/95bd73db/Dockerfile#L498 How to retry via `$SAUCE_TUNNEL_MAX_RETRY_ATTEMPTS`: https://github.com/elgalu/docker-selenium/blob/95bd73db/saucelabs/bin/start-saucelabs.sh#L41

          Val Redchenko added a comment -

          Ditching the plugin in favor of shell scripting is our plan B, but it would still be nice to get the plugin retry connection out of the box. Looking through Sauce on Demand sources - if I can get it working I'll share it here.

          Val Redchenko added a comment - Ditching the plugin in favor of shell scripting is our plan B, but it would still be nice to get the plugin retry connection out of the box. Looking through Sauce on Demand sources - if I can get it working I'll share it here.

          Diego Molina added a comment -

          vredchenko, did you have the chance to check the plugin source code?

          I was thinking to make a change around here: https://github.com/jenkinsci/sauce-ondemand-plugin/blob/master/src/main/java/hudson/plugins/sauce_ondemand/SauceOnDemandBuildWrapper.java#L883, and I am also trying to find a way to have the amount of times configurable.

          What do you think?

          Diego Molina added a comment - vredchenko , did you have the chance to check the plugin source code? I was thinking to make a change around here: https://github.com/jenkinsci/sauce-ondemand-plugin/blob/master/src/main/java/hudson/plugins/sauce_ondemand/SauceOnDemandBuildWrapper.java#L883 , and I am also trying to find a way to have the amount of times configurable. What do you think?

          Diego Molina added a comment -

          Well, I have submitted a PR, lets see if they accept it:
          https://github.com/jenkinsci/sauce-ondemand-plugin/pull/18

          Diego Molina added a comment - Well, I have submitted a PR, lets see if they accept it: https://github.com/jenkinsci/sauce-ondemand-plugin/pull/18

            thecampbellcash Andrew Campbell
            elgalu Leo Gallucci
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: