Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41336

Cron based jobs are triggered at random times and also at the time Stash is unavailable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • None
    • Jenkins 2.7
      stash-pullrequest-builder-plugin 1.7

      The cron based jobs are working fine until stash-pullrequest-builder-plugin is installed and configured for one of our gating job. The frequency to run is every 2 min. Since then all the cron based jobs are not triggered at specified time, but rather at random time. They also get triggered every time Stash is unavailable, such as during its backup. Hence all cron based jobs failed.

          [JENKINS-41336] Cron based jobs are triggered at random times and also at the time Stash is unavailable

          Vivian zhang created issue -

          Description has nothing useful, during the problem catch /threadDump

          Kanstantsin Shautsou added a comment - Description has nothing useful, during the problem catch /threadDump

          integer the behavior is that all cron based jobs aren't executed when they are supposed to do it because Triggers are busy to interact with stash

          The thread is showing something like this (thanks danielbeck)

          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
          

          Trigger.checkTriggers should only call tasks which are immediate. These ones could start in another thread a longer process to really trigger the job or not.
          Here the StashBuildTrigger.run does too much stuffs and is quickly blocked or creates a lot of delay if the Stash isn't available or doesn't reply quickly (which is never the case).
          This bug makes this plugin not usable and the only solution is to deactivate/uninstall it for now.

          Arnaud Héritier added a comment - integer the behavior is that all cron based jobs aren't executed when they are supposed to do it because Triggers are busy to interact with stash The thread is showing something like this (thanks danielbeck ) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75) at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32) at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299) at hudson.triggers.Trigger.checkTriggers(Trigger.java:273) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222) Trigger.checkTriggers should only call tasks which are immediate. These ones could start in another thread a longer process to really trigger the job or not. Here the StashBuildTrigger.run does too much stuffs and is quickly blocked or creates a lot of delay if the Stash isn't available or doesn't reply quickly (which is never the case). This bug makes this plugin not usable and the only solution is to deactivate/uninstall it for now.

          Right or stop calling from cron if it has configured hooks, depends on plugin design.

          Kanstantsin Shautsou added a comment - Right or stop calling from cron if it has configured hooks, depends on plugin design.

          Vivian zhang added a comment -

          I'm surprised that the plugin requires cron to be set up for the job, which is very inefficient to go through all the open PRs every time. It should try to use the branch info provided by Stash post-commit hook instead.

          Vivian zhang added a comment - I'm surprised that the plugin requires cron to be set up for the job, which is very inefficient to go through all the open PRs every time. It should try to use the branch info provided by Stash post-commit hook instead.

          That's jenkins design - all Triggers inherit cron functionality, but it can be simply skipped.

          Kanstantsin Shautsou added a comment - That's jenkins design - all Triggers inherit cron functionality, but it can be simply skipped.

          Vivian zhang added a comment -

          What I meant was this plugin should rely on Stash's post-commit webhook plugin for sending change notification to Jenkins, and trigger the build with poll scm option checked (no cron needed here). Looks like this place is to report Jenkins core bug, maybe I should file an issue report at the plugin's github site?

          Vivian zhang added a comment - What I meant was this plugin should rely on Stash's post-commit webhook plugin for sending change notification to Jenkins, and trigger the build with poll scm option checked (no cron needed here). Looks like this place is to report Jenkins core bug, maybe I should file an issue report at the plugin's github site?

          And how it will work behind nat then?

          Kanstantsin Shautsou added a comment - And how it will work behind nat then?

          Vivian zhang added a comment -

          That's how it's been working without this plugin, although I still like some of the features provided by this plugin.

          Vivian zhang added a comment - That's how it's been working without this plugin, although I still like some of the features provided by this plugin.

          Larry Brigman added a comment -

          I'm seeing jobs queue when I get this error:

          Mar 23, 2017 4:43:00 PM stashpullrequestbuilder.stashpullrequestbuilder.StashRepository getTargetPullRequests
          INFO: Fetch PullRequests (ansible-checks).
          java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Read timed out
          at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.util.concurrent.FutureTask.get(FutureTask.java:202)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)

           

          Using Jenkins 2.44 on Ubuntu 14.04 LTS

          OpenJDK 64bit Java 1.7.0_79

          Stash Pull Request Builder 1.7

          Against Stash/Bitbucket v4.4.1

          The timeout is due to an issue on the bitbucket end but it shouldn't be causing a build to be queued.

          Larry Brigman added a comment - I'm seeing jobs queue when I get this error: Mar 23, 2017 4:43:00 PM stashpullrequestbuilder.stashpullrequestbuilder.StashRepository getTargetPullRequests INFO: Fetch PullRequests (ansible-checks). java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Read timed out at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)   Using Jenkins 2.44 on Ubuntu 14.04 LTS OpenJDK 64bit Java 1.7.0_79 Stash Pull Request Builder 1.7 Against Stash/Bitbucket v4.4.1 The timeout is due to an issue on the bitbucket end but it shouldn't be causing a build to be queued.

            Unassigned Unassigned
            vla Vivian zhang
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: