Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63555

GitHub Org being reset randomly

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We are seeing a weird issue with our Jenkins / GitHub Org integration. It seems that randomly, all the build history for pipelines inside a GitHub Org folder disappear. They also trigger builds on all the pipelines inside the org folder. I have looked through the Jenkins log and haven't found anything that is obviously triggering this. 

      Edit: It also is triggering all PRs in all repos in the Org to rebuild.

        Attachments

          Activity

          ccovarru Cesar Covarrubias created issue -
          Hide
          ccovarru Cesar Covarrubias added a comment -

          Following up on this issue, it looks like whatever triggers this is causing all the open PRs to build, causing hundreds of build jobs to build at the same time, spiking CPU on the head node.

          Show
          ccovarru Cesar Covarrubias added a comment - Following up on this issue, it looks like whatever triggers this is causing all the open PRs to build, causing hundreds of build jobs to build at the same time, spiking CPU on the head node.
          ccovarru Cesar Covarrubias made changes -
          Field Original Value New Value
          Summary Build History Missing in GitHub Org Folder GitHub Org being reset randomly
          ccovarru Cesar Covarrubias made changes -
          Description We are seeing a weird issue with our Jenkins / GitHub Org integration. It seems that randomly, all the build history for pipelines inside a GitHub Org folder disappear. They also trigger builds on all the pipelines inside the org folder. I have looked through the Jenkins log and haven't found anything that is obviously triggering this.  We are seeing a weird issue with our Jenkins / GitHub Org integration. It seems that randomly, all the build history for pipelines inside a GitHub Org folder disappear. They also trigger builds on all the pipelines inside the org folder. I have looked through the Jenkins log and haven't found anything that is obviously triggering this. 

          Edit: It also is triggering all PRs in all repos in the Org to rebuild.
          Hide
          bitwiseman Liam Newman added a comment -

          Cesar Covarrubias
          What is likely happening is an edge case where the scan fails but doesn't know that it failed and marks all children for deletion. The next scan succeeds and believe it has found all new repos (with new branches and PRs).

          If you set your "Orphaned Item Strategy" to retain old items for 1 day and leave the number of items blank (unlimited), that should at least keep your repos, branches and PRs from being deleted. They may still all run with then are found again, but maybe not.

          Some additional questions:
          Are you using github app credentials?
          Are you using github.com or GitHub Enterprise?
          Do your Jenkins System logs have warnings or alerts related to repo scanning?

          Show
          bitwiseman Liam Newman added a comment - Cesar Covarrubias What is likely happening is an edge case where the scan fails but doesn't know that it failed and marks all children for deletion. The next scan succeeds and believe it has found all new repos (with new branches and PRs). If you set your "Orphaned Item Strategy" to retain old items for 1 day and leave the number of items blank (unlimited), that should at least keep your repos, branches and PRs from being deleted. They may still all run with then are found again, but maybe not. Some additional questions: Are you using github app credentials? Are you using github.com or GitHub Enterprise? Do your Jenkins System logs have warnings or alerts related to repo scanning?
          Hide
          ccovarru Cesar Covarrubias added a comment -

          We did have Orphaned Strategy set to retain items for 10 days and 10 items. I just set the number of items for unlimited. I anticipate this happening again  since it's been happening every day or 2. 

          Yes we are using GitHub App creds.
          Our org is on github.com, but with a business account. 

          The main log we see is "GitHub webhooks activated for job OrgName/repo" for all our repos that match.

          Show
          ccovarru Cesar Covarrubias added a comment - We did have Orphaned Strategy set to retain items for 10 days and 10 items. I just set the number of items for unlimited. I anticipate this happening again  since it's been happening every day or 2.  Yes we are using GitHub App creds. Our org is on github.com, but with a business account.  The main log we see is "GitHub webhooks activated for job OrgName/repo" for all our repos that match.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            ccovarru Cesar Covarrubias
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: