Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41336

Cron based jobs are triggered at random times and also at the time Stash is unavailable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The cron based jobs are working fine until stash-pullrequest-builder-plugin is installed and configured for one of our gating job. The frequency to run is every 2 min. Since then all the cron based jobs are not triggered at specified time, but rather at random time. They also get triggered every time Stash is unavailable, such as during its backup. Hence all cron based jobs failed.

        Attachments

          Activity

          vla Vivian zhang created issue -
          Hide
          integer Kanstantsin Shautsou added a comment -

          Description has nothing useful, during the problem catch /threadDump

          Show
          integer Kanstantsin Shautsou added a comment - Description has nothing useful, during the problem catch /threadDump
          Hide
          aheritier Arnaud Héritier added a comment -

          Kanstantsin Shautsou the behavior is that all cron based jobs aren't executed when they are supposed to do it because Triggers are busy to interact with stash

          The thread is showing something like this (thanks Daniel Beck)

          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
          at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
          

          Trigger.checkTriggers should only call tasks which are immediate. These ones could start in another thread a longer process to really trigger the job or not.
          Here the StashBuildTrigger.run does too much stuffs and is quickly blocked or creates a lot of delay if the Stash isn't available or doesn't reply quickly (which is never the case).
          This bug makes this plugin not usable and the only solution is to deactivate/uninstall it for now.

          Show
          aheritier Arnaud Héritier added a comment - Kanstantsin Shautsou the behavior is that all cron based jobs aren't executed when they are supposed to do it because Triggers are busy to interact with stash The thread is showing something like this (thanks Daniel Beck ) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261) at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75) at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32) at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299) at hudson.triggers.Trigger.checkTriggers(Trigger.java:273) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222) Trigger.checkTriggers should only call tasks which are immediate. These ones could start in another thread a longer process to really trigger the job or not. Here the StashBuildTrigger.run does too much stuffs and is quickly blocked or creates a lot of delay if the Stash isn't available or doesn't reply quickly (which is never the case). This bug makes this plugin not usable and the only solution is to deactivate/uninstall it for now.
          Hide
          integer Kanstantsin Shautsou added a comment -

          Right or stop calling from cron if it has configured hooks, depends on plugin design.

          Show
          integer Kanstantsin Shautsou added a comment - Right or stop calling from cron if it has configured hooks, depends on plugin design.
          Hide
          vla Vivian zhang added a comment -

          I'm surprised that the plugin requires cron to be set up for the job, which is very inefficient to go through all the open PRs every time. It should try to use the branch info provided by Stash post-commit hook instead.

          Show
          vla Vivian zhang added a comment - I'm surprised that the plugin requires cron to be set up for the job, which is very inefficient to go through all the open PRs every time. It should try to use the branch info provided by Stash post-commit hook instead.
          Hide
          integer Kanstantsin Shautsou added a comment -

          That's jenkins design - all Triggers inherit cron functionality, but it can be simply skipped.

          Show
          integer Kanstantsin Shautsou added a comment - That's jenkins design - all Triggers inherit cron functionality, but it can be simply skipped.
          Hide
          vla Vivian zhang added a comment -

          What I meant was this plugin should rely on Stash's post-commit webhook plugin for sending change notification to Jenkins, and trigger the build with poll scm option checked (no cron needed here). Looks like this place is to report Jenkins core bug, maybe I should file an issue report at the plugin's github site?

          Show
          vla Vivian zhang added a comment - What I meant was this plugin should rely on Stash's post-commit webhook plugin for sending change notification to Jenkins, and trigger the build with poll scm option checked (no cron needed here). Looks like this place is to report Jenkins core bug, maybe I should file an issue report at the plugin's github site?
          Hide
          integer Kanstantsin Shautsou added a comment -

          And how it will work behind nat then?

          Show
          integer Kanstantsin Shautsou added a comment - And how it will work behind nat then?
          Hide
          vla Vivian zhang added a comment -

          That's how it's been working without this plugin, although I still like some of the features provided by this plugin.

          Show
          vla Vivian zhang added a comment - That's how it's been working without this plugin, although I still like some of the features provided by this plugin.
          Hide
          lbrigman_arris Larry Brigman added a comment -

          I'm seeing jobs queue when I get this error:

          Mar 23, 2017 4:43:00 PM stashpullrequestbuilder.stashpullrequestbuilder.StashRepository getTargetPullRequests
          INFO: Fetch PullRequests (ansible-checks).
          java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Read timed out
          at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.util.concurrent.FutureTask.get(FutureTask.java:202)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
          at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)

           

          Using Jenkins 2.44 on Ubuntu 14.04 LTS

          OpenJDK 64bit Java 1.7.0_79

          Stash Pull Request Builder 1.7

          Against Stash/Bitbucket v4.4.1

          The timeout is due to an issue on the bitbucket end but it shouldn't be causing a build to be queued.

          Show
          lbrigman_arris Larry Brigman added a comment - I'm seeing jobs queue when I get this error: Mar 23, 2017 4:43:00 PM stashpullrequestbuilder.stashpullrequestbuilder.StashRepository getTargetPullRequests INFO: Fetch PullRequests (ansible-checks). java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Read timed out at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254) at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)   Using Jenkins 2.44 on Ubuntu 14.04 LTS OpenJDK 64bit Java 1.7.0_79 Stash Pull Request Builder 1.7 Against Stash/Bitbucket v4.4.1 The timeout is due to an issue on the bitbucket end but it shouldn't be causing a build to be queued.
          cloudbees CloudBees Inc. made changes -
          Field Original Value New Value
          Remote Link This issue links to "CloudBees Internal OSS-1984 (Web Link)" [ 18479 ]
          Show
          jonilover ivan palanevich added a comment - You can try to use that fix - https://github.com/kmoco2am/stash-pullrequest-builder-plugin/commit/da4ed53fb8f38f10df956175e7feda76f64f766e
          jonilover ivan palanevich made changes -
          Remote Link This issue links to "Custom fix (Web Link)" [ 21271 ]
          jbochenski Jakub Bochenski made changes -
          Assignee nathan m [ nemccarthy ]
          Hide
          jbochenski Jakub Bochenski added a comment -

          So the issue here is that StashBuildTrigger.run runs for too long?

          Show
          jbochenski Jakub Bochenski added a comment - So the issue here is that StashBuildTrigger.run runs for too long?
          Hide
          jbochenski Jakub Bochenski added a comment -

          This issue has been closed because of inactivity. Please reopen it if you think it's still valid

          Show
          jbochenski Jakub Bochenski added a comment - This issue has been closed because of inactivity. Please reopen it if you think it's still valid
          jbochenski Jakub Bochenski made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]
          Hide
          lehlen Luis Ehlen added a comment -

          I just updated to the latest Jenkins LTS 2.204.2 and the latest version of the stash pull request builder plugin 1,17 (from 1.8) and the mentioned behavior with the cron based jobs appeared.
          I can see this on my logs:

           

          Feb 19, 2020 12:25:29 PM WARNING hudson.triggers.Trigger checkTriggersTrigger

          stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run() triggered by hudson.maven.MavenModuleSet@45177041[JOB_NAME] spent too much time (54 sec) in its execution, other timers can be affected

          Show
          lehlen Luis Ehlen added a comment - I just updated to the latest Jenkins LTS 2.204.2 and the latest version of the stash pull request builder plugin 1,17 (from 1.8) and the mentioned behavior with the cron based jobs appeared. I can see this on my logs:   Feb 19, 2020 12:25:29 PM WARNING hudson.triggers.Trigger checkTriggers Trigger stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run() triggered by hudson.maven.MavenModuleSet@45177041 [JOB_NAME] spent too much time (54 sec) in its execution, other timers can be affected
          lehlen Luis Ehlen made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            vla Vivian zhang
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: