Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42254

multibranch pipeline job notified by github, but repo scan finds "No changes detected"

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • Jenkins ver. 2.32.2
      please see the attached file installed_plugins.xml for details about all the plugins

      Hi, i have a case where a jenkins multibranch build job was notified about a change, but did not detect the change and therefore did not build.
      The attached file github_webhook_payload.json contains the full payload that was send to jenkins.

      imho the most interesting part is this:

        "ref": "refs/heads/master",
        "before": "aac98c4d843ba4cf53b7fc375b7eb301337d65ec",
        "after": "9e25b9d14d2405e5b8c77d27d15e3ea7b19fa251",
      

      From this screenshot one can see that the request to jenkins was made at 20:33.

      The problem was that i assumed that the issue fix was build and pushed the issue forward.
      Then today, the dev got his issue back with "IT failed" and we found out, that it was only build this forenoon.

      When I opened https://jenkins.metasfresh.com/job/metasfresh-webui/indexing/events today, i found the following in the log:

      [Tue Feb 21 20:34:05 CET 2017] Received org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl UPDATED event from 192.30.252.42 ⇒ http://jenkins.metasfresh.com:47832/github-webhook/ with timestamp Tue Feb 21 20:34:00 CET 2017
      Connecting to https://api.github.com using metas-dev/******
      Looking up metasfresh/metasfresh-webui
      
        Getting remote branches...
      
          Checking branch master
            ‘Jenkinsfile’ found
          Met criteria
      No changes detected: master (still at aac98c4d843ba4cf53b7fc375b7eb301337d65ec)
      
      Done examining metasfresh/metasfresh-webui
      
      [Tue Feb 21 20:34:07 CET 2017] org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber$SCMHeadEventImpl UPDATED event from 192.30.252.42 ⇒ http://jenkins.metasfresh.com:47832/github-webhook/ with timestamp Tue Feb 21 20:34:00 CET 2017 processed in 2,5 sec
      

      I'm also attaching the job's config.xml file.

      Please let me know if i can provide further into.

      PS: I did not modify the jobs config, so i don't see a releation to issue 14855.

          [JENKINS-42254] multibranch pipeline job notified by github, but repo scan finds "No changes detected"

          We also have somehow a similar problem and see a lot of events - every 10 to 30 minutes - like the following below in our multibranch builds - where job configurations are modified from time to time:

          ...
          [Wed Feb 22 22:38:25 CET 2017] Received jenkins.plugins.git.GitSCMSource$ListenerImpl$1 UPDATED event from <xxx.xx.xxx.xx> ⇒ http://<dns>:41138/git/notifyCommit with timestamp Wed Feb 22 22:37:10 CET 2017
           > git rev-parse --is-inside-work-tree # timeout=10
          ...
          Checking branch master
                ‘Jenkinsfile’ found
          Met criteria
          Takeover for Synthetic » Builds » common » master by source #1 from source that no longer exists
          Branch reopened: master (fee002ae94cd1d25ee9a542f5abc049150d109fa)
          Scheduled build for branch: master
          [Wed Feb 22 22:38:26 CET 2017] jenkins.plugins.git.GitSCMSource$ListenerImpl$1 UPDATED event from <xxx.xx.xxx.xx> ⇒ http://<dns>:41138/git/notifyCommit with timestamp Wed 
          ...
          

          The following log output is extremely confusing from our point of view:
          Takeover for Synthetic » Builds » common » master by source #1 from source that no longer exists
          Branch reopened: master (fee002ae94cd1d25ee9a542f5abc049150d109fa)

          Bahram Parsapour added a comment - We also have somehow a similar problem and see a lot of events - every 10 to 30 minutes - like the following below in our multibranch builds - where job configurations are modified from time to time: ... [Wed Feb 22 22:38:25 CET 2017] Received jenkins.plugins.git.GitSCMSource$ListenerImpl$1 UPDATED event from <xxx.xx.xxx.xx> ⇒ http: //<dns>:41138/git/notifyCommit with timestamp Wed Feb 22 22:37:10 CET 2017 > git rev-parse --is-inside-work-tree # timeout=10 ... Checking branch master ‘Jenkinsfile’ found Met criteria Takeover for Synthetic » Builds » common » master by source #1 from source that no longer exists Branch reopened: master (fee002ae94cd1d25ee9a542f5abc049150d109fa) Scheduled build for branch: master [Wed Feb 22 22:38:26 CET 2017] jenkins.plugins.git.GitSCMSource$ListenerImpl$1 UPDATED event from <xxx.xx.xxx.xx> ⇒ http: //<dns>:41138/git/notifyCommit with timestamp Wed ... The following log output is extremely confusing from our point of view: Takeover for Synthetic » Builds » common » master by source #1 from source that no longer exists Branch reopened : master (fee002ae94cd1d25ee9a542f5abc049150d109fa)

          We currently have a hard-coded 5 second delay on github events because our experiments show that it typically takes 1-2 seconds for all the caching layers of github to sync up with the events.

          Should we consider making this delay configurable?

          Stephen Connolly added a comment - We currently have a hard-coded 5 second delay on github events because our experiments show that it typically takes 1-2 seconds for all the caching layers of github to sync up with the events. Should we consider making this delay configurable?

          Tobias Schoeneberg added a comment - - edited

          Hi stephenconnolly,
          thanks for your reply&suggestion

          My naive idea from a user's perspective would be to go with the information provided by github
          ..i.e. just check out the "after": "9e25b9d14d2405e5b8c77d27d15e3ea7b19fa251" revision from my screenshot-example and maybe even retry a few times,
          if github is not yet up2date with its own announcement
          ...but i guess it that was so easy, you wouldn't need me to come up with the idea

          So yes, making the hard-coded value of 5sec configurable seems to me like a good step..we can set it to say 20secs, to catch (just making up a number here) 96.27% ot the cases.

          PS: after reading more from the plugin's documentation, i'm also using the "Periodically if not otherwise run" option now. I think that will solve the remaining 3.74%, so we end up with a 100.1% sucess rate, which is just fine (scnr, it's carnival around here)

          Tobias Schoeneberg added a comment - - edited Hi stephenconnolly , thanks for your reply&suggestion My naive idea from a user's perspective would be to go with the information provided by github ..i.e. just check out the "after": "9e25b9d14d2405e5b8c77d27d15e3ea7b19fa251" revision from my screenshot-example and maybe even retry a few times, if github is not yet up2date with its own announcement ...but i guess it that was so easy, you wouldn't need me to come up with the idea So yes, making the hard-coded value of 5sec configurable seems to me like a good step..we can set it to say 20secs, to catch (just making up a number here) 96.27% ot the cases. PS: after reading more from the plugin's documentation, i'm also using the "Periodically if not otherwise run" option now. I think that will solve the remaining 3.74%, so we end up with a 100.1% sucess rate, which is just fine (scnr, it's carnival around here)

          My naive idea from a user's perspective would be to go with the information provided by github

          The issue here is that until we have a secure event from GitHub, we cannot trust the information provided by github. That would require that github assign sequence identifiers to all events (so we can avoid replay attacks) as well as signing information (since we cannot prevent users that need to terminate TLS before Jenkins and thus deny Jenkins the ability to verify a client certificate)... nevermind that an admin can redeliver older messages anyway!

          Given the above, we instead just go to the source of truth and ask again. The issue - that you have found - is that there can be delays between the different caches in GitHub.

          Stephen Connolly added a comment - My naive idea from a user's perspective would be to go with the information provided by github The issue here is that until we have a secure event from GitHub, we cannot trust the information provided by github. That would require that github assign sequence identifiers to all events (so we can avoid replay attacks) as well as signing information (since we cannot prevent users that need to terminate TLS before Jenkins and thus deny Jenkins the ability to verify a client certificate)... nevermind that an admin can redeliver older messages anyway! Given the above, we instead just go to the source of truth and ask again. The issue - that you have found - is that there can be delays between the different caches in GitHub.

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java
          src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java
          src/main/java/org/jenkinsci/plugins/github_branch_source/PushGHEventSubscriber.java
          http://jenkins-ci.org/commit/github-branch-source-plugin/92e38ad8e6ded68a6200639936e45534a4fe7076
          Log:
          JENKINS-42254 Make the github sync delay configurable

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java src/main/java/org/jenkinsci/plugins/github_branch_source/PushGHEventSubscriber.java http://jenkins-ci.org/commit/github-branch-source-plugin/92e38ad8e6ded68a6200639936e45534a4fe7076 Log: JENKINS-42254 Make the github sync delay configurable

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/test/java/com/cloudbees/hudson/plugins/folder/computed/ComputedFolderTest.java
          http://jenkins-ci.org/commit/cloudbees-folder-plugin/9f90a8fcf864d07922e3c42c55c796abd8e6303f
          Log:
          JENKINS-42254 Add a test case for the issue

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/test/java/com/cloudbees/hudson/plugins/folder/computed/ComputedFolderTest.java http://jenkins-ci.org/commit/cloudbees-folder-plugin/9f90a8fcf864d07922e3c42c55c796abd8e6303f Log: JENKINS-42254 Add a test case for the issue

          Stephen Connolly added a comment - Argh! no ignore  http://jenkins-ci.org/commit/cloudbees-folder-plugin/9f90a8fcf864d07922e3c42c55c796abd8e6303f

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java
          src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java
          src/main/java/org/jenkinsci/plugins/github_branch_source/PushGHEventSubscriber.java
          http://jenkins-ci.org/commit/github-branch-source-plugin/5273e4f18a1ebd4a29ec80ec79f8467faeb41195
          Log:
          Merge pull request #126 from jenkinsci/jenkins-42254

          JENKINS-42254 Make the github sync delay configurable

          Compare: https://github.com/jenkinsci/github-branch-source-plugin/compare/f7546e3755a5...5273e4f18a1e

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java src/main/java/org/jenkinsci/plugins/github_branch_source/PushGHEventSubscriber.java http://jenkins-ci.org/commit/github-branch-source-plugin/5273e4f18a1ebd4a29ec80ec79f8467faeb41195 Log: Merge pull request #126 from jenkinsci/jenkins-42254 JENKINS-42254 Make the github sync delay configurable Compare: https://github.com/jenkinsci/github-branch-source-plugin/compare/f7546e3755a5...5273e4f18a1e

          Will be in the 2.0.4 release of Github Branch Source

          Basically you can set the delay (anywhere between 0 and 300 seconds) using the system property -Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.eventDelaySeconds=20 

          If you don't want to restart you can also change the value using the system groovy console, e.g. 

          org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.eventDelaySeconds=20

          Stephen Connolly added a comment - Will be in the 2.0.4 release of Github Branch Source Basically you can set the delay (anywhere between 0 and 300 seconds) using the system property -Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.eventDelaySeconds=20   If you don't want to restart you can also change the value using the system groovy console, e.g.  org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.eventDelaySeconds=20

          great, thanks 

          Tobias Schoeneberg added a comment - great, thanks 

            stephenconnolly Stephen Connolly
            tobi42 Tobias Schoeneberg
            Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: