Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51673

PollSCM runs again when job on commit takes longer then the trigger

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: git-plugin, pipeline
    • Labels:
      None
    • Environment:
      Jenkins 2.125, Ubuntu 14
    • Similar Issues:

      Description

      When the pollSCM in a declarative pipeline is executed when the jobs on the same commit is queued and did not do a checkout scm the jobs is queued again on the same commit.

      A test Jekninsfile:

      pipeline {
          agent {
              node {
                  label 'local'
              }
          }
          triggers {
              pollSCM('H/2 * * * *')
          }
          stages {
              stage('Run') {
                  steps {
                      checkout scm
                      sh 'sleep 1000'
                  }
              }
          }
      }
      

      If a new commit is pushed within in 1000 seconds and the executors on jenkins are busy jenkins will fire a new job at each pollSCM till 1 job on this commit did do a checkout. In this example this will be 5-6 jobs on the same commit.

      When doing this on multiple repositories on a limited number of executors this will explode in 10000+ jobs if your unlucky. (happened to me)

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          PIpeline intentionally allows multiple concurrent builds of the same commit. Freestyle jobs block the polling while a job is running but that also prevents them from being used to test different commits concurrently.

          In this case, the preferred solution is to notify Jenkins from the git repository through either the notifyCommit API or with a webhook (as in GitHub, Bitbucket, Gitea, etc.). Refer to the "polling must die" blog post for the description why polling is less efficient and less effective than having the repository notify Jenkins on each change.

          I think this should be closed as "Won't Fix", though there might be some way to extend the Pipeline syntax to specifically disallow concurrent builds of the same git commit. I won't be creating that type of Pipeline syntax extension.

          Show
          markewaite Mark Waite added a comment - PIpeline intentionally allows multiple concurrent builds of the same commit. Freestyle jobs block the polling while a job is running but that also prevents them from being used to test different commits concurrently. In this case, the preferred solution is to notify Jenkins from the git repository through either the notifyCommit API or with a webhook (as in GitHub, Bitbucket, Gitea, etc.). Refer to the " polling must die " blog post for the description why polling is less efficient and less effective than having the repository notify Jenkins on each change. I think this should be closed as "Won't Fix", though there might be some way to extend the Pipeline syntax to specifically disallow concurrent builds of the same git commit. I won't be creating that type of Pipeline syntax extension.
          Hide
          pjvanthof Peter Hof added a comment -

          I do agree that polling is not the best option. But in this case it's the only way because of the firewall policy. Reverse proxy is also not allowed. Meaning the only way doing this by polling. I wish I could change the policy but this will not happen in a hospital.

          Why is multiple builds on the same commit is useful? Testing 1 commit twice will result in the same results or not?

          Show
          pjvanthof Peter Hof added a comment - I do agree that polling is not the best option. But in this case it's the only way because of the firewall policy. Reverse proxy is also not allowed. Meaning the only way doing this by polling. I wish I could change the policy but this will not happen in a hospital. Why is multiple builds on the same commit is useful? Testing 1 commit twice will result in the same results or not?
          Hide
          markewaite Mark Waite added a comment -

          I've used a cached repository inside the firewall which mirrors the external repository and then defined the git repository hooks on that cached repository to call the git/notifyCommit URL. That avoids polling to an SCM outside the firewall and still responds quickly to changes. That has required me to push my changes to both the official repository and the cache repository, but it has worked quite well for me.

          The decision on multiple builds being allowed on the same commit is a Pipeline decision. It is not something I control, nor is it something I want to see changed. Multiple builds of the same commit are a key part of the "Replay" function available in Pipeline. Multiple builds of the same commit may be done with different parameters, or for variations of the build, or for tests of repeatability of the build.

          Show
          markewaite Mark Waite added a comment - I've used a cached repository inside the firewall which mirrors the external repository and then defined the git repository hooks on that cached repository to call the git/notifyCommit URL. That avoids polling to an SCM outside the firewall and still responds quickly to changes. That has required me to push my changes to both the official repository and the cache repository, but it has worked quite well for me. The decision on multiple builds being allowed on the same commit is a Pipeline decision. It is not something I control, nor is it something I want to see changed. Multiple builds of the same commit are a key part of the "Replay" function available in Pipeline. Multiple builds of the same commit may be done with different parameters, or for variations of the build, or for tests of repeatability of the build.
          Hide
          pjvanthof Peter Hof added a comment -

          Does this cached repository then not must sync/poll github still? Committing to 2 repositories does not seems like a solution, just a workaround, this solution also will not work well with the github folder plugin I think.

          Show
          pjvanthof Peter Hof added a comment - Does this cached repository then not must sync/poll github still? Committing to 2 repositories does not seems like a solution, just a workaround, this solution also will not work well with the github folder plugin I think.
          Hide
          ceddaerrix Cedric H added a comment -

          I am meeting the same behavior on Jenkins 2.254 (on RHEL 7.7) with Git.

          Has any Jenkins workaround be found yet?

           

          Show
          ceddaerrix Cedric H added a comment - I am meeting the same behavior on Jenkins 2.254 (on RHEL 7.7) with Git. Has any Jenkins workaround be found yet?  

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            pjvanthof Peter Hof
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: