Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57235

Seed job (job-dsl) runs trigger a rebuild of all multibranch pipelines branches

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Duplicate
    • job-dsl-plugin, multibranch-build-strategy-extension-plugin
    • None

    Description

      I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|https://github.com/helm/charts/tree/master/stable/jenkins] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

      My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and builtt.

      The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

      From what I understood, the seed job recreates all its managed jobs every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

      I can debug this further but I will need a bit of help (debugging tips).

      To me the solutions are:

      • Fix either the seed or the way the multibranch pipelines handles recreation.
      • Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

      I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

       

      Thanks for your help!

       

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-43693 multibranchPipelineJob overrides old branch indexing sources

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-57663 Do not trigger all branch builds when regenerating a multibranchPipelineJob

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            jpigree Jonathan Pigrée created issue -
            jpigree Jonathan Pigrée made changes -
            Field Original Value New Value
            Description I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://www.google.com/search?q=helm+chart+jenkins&oq=helm+chart+jenkins&aqs=chrome..69i57j69i61j69i60j69i61j0l2.2264j0j7&sourceid=chrome&ie=UTF-8]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple.

             

            Thanks for your help!

             

                          		I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple.

             

            Thanks for your help!

             

             
            jpigree Jonathan Pigrée made changes -
            Description I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple.

             

            Thanks for your help!

             

                          		I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple.

             

            Thanks for your help!

             

             
            jpigree Jonathan Pigrée made changes -
            Description I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple.

             

            Thanks for your help!

             

                          		I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

             

            Thanks for your help!

             

             
            jpigree Jonathan Pigrée made changes -
            Description I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build before.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all jobs it manages every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

             

            Thanks for your help!

             

                          		I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all its managed jobs every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

             

            Thanks for your help!

             

             
            jpigree Jonathan Pigrée made changes -
            Summary Seed job runs trigger a rebuild of all multibranch pipelines branches Seed job (job-dsl) runs trigger a rebuild of all multibranch pipelines branches
            jpigree Jonathan Pigrée made changes -
            Description I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and build.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all its managed jobs every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

             

            Thanks for your help!

             

                          		I have a fully configured as code jenkins deployed in GKE using the official Helm [chart|[https://github.com/helm/charts/tree/master/stable/jenkins]] (version 0.33.1). I also use a seed job (pre-installed by the chart) which creates all the other jobs.

            My problem is that each time someone triggers the seed job, it triggers a branch indexing in every existing multibranch pipelines. Those branch indexing trigger a build for every branches discovered even when they were already previously discovered and builtt.

            The consequence is that we get huge build spikes each time someone runs the jenkins seed. I searched the web for months for a fix but all I could find was workarounds. The latest I put in place is to install the "Basic Branch Strategy plugin" and enforce the newly added "skip initial build on first branch indexing" strategy on all my multibranch pipelines. It fixed my problem but the downside is that, my newly discovered branches stopped building automatically so the workaround isn't satisfying.

            From what I understood, the seed job recreates all its managed jobs every time it is run, so I guess the problem must come from there but I don't know how to confirm it.

            I can debug this further but I will need a bit of help (debugging tips).

            To me the solutions are:
             * Fix either the seed or the way the multibranch pipelines handles recreation.
             * Create a basic branch strategy which filters an initial trigger when the branch was already built before and the current commit is the same than the previous build's.

            I can provide further information if needed. I can even submit a fix if it is trivial/simple (I am not a Java developer).

             

            Thanks for your help!

             

             
            daspilker Daniel Spilker made changes -
            Link This issue is duplicated by JENKINS-57663 [ JENKINS-57663 ]
            daspilker Daniel Spilker made changes -
            Link This issue duplicates JENKINS-43693 [ JENKINS-43693 ]
            daspilker Daniel Spilker made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            jpigree Jonathan Pigrée added a comment -

            Thanks. I found this solution myself a while ago and forgot to update the ticket.

             

            jpigree Jonathan Pigrée added a comment - Thanks. I found this solution myself a while ago and forgot to update the ticket.  
            halkeye Gavin Mogan added a comment -

            jpigree -

            https://xkcd.com/979/

            What did you find out? How did you fix it?

            halkeye Gavin Mogan added a comment - jpigree - https://xkcd.com/979/ What did you find out? How did you fix it?
            jpigree Jonathan Pigrée added a comment - - edited

            Hi halkeye. Sorry, Actually the solution is in the Jira ticket duplicated (JENKINS-43693).

            => https://issues.jenkins-ci.org/browse/JENKINS-43693?focusedCommentId=296900&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-296900

            In short, you have to set fixed ids in the branch sources.

            The bug comes from the job-dsl plugin which autogenerates branch sources ids on every run (when they are not set), thus recreating branch sources which have to "takeover" and schedule a new build doing so.

            What is truly missleading is that this behavior is not mentioned in the seed documentation AND the examples do not set ids.

            jpigree Jonathan Pigrée added a comment - - edited Hi halkeye . Sorry, Actually the solution is in the Jira ticket duplicated ( JENKINS-43693 ). => https://issues.jenkins-ci.org/browse/JENKINS-43693?focusedCommentId=296900&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-296900 In short, you have to set fixed ids in the branch sources. The bug comes from the job-dsl plugin which autogenerates branch sources ids on every run (when they are not set), thus recreating branch sources which have to "takeover" and schedule a new build doing so. What is truly missleading is that this behavior is not mentioned in the seed documentation AND the examples do not set ids.
            halkeye Gavin Mogan added a comment -

            Ah, interesting, I actually found a different problem, I did also figure it out eventually from reading the other bugs.

             

            https://github.com/halkeye/jenkins-jobs/blob/master/jobs.groovy#L59-L84

             

            Buildstrategies, by default, is an OR, so it would discover branches, or skip indexing, but not both. So I had to convert it to an AND

            halkeye Gavin Mogan added a comment - Ah, interesting, I actually found a different problem, I did also figure it out eventually from reading the other bugs.   https://github.com/halkeye/jenkins-jobs/blob/master/jobs.groovy#L59-L84   Buildstrategies, by default, is an OR, so it would discover branches, or skip indexing, but not both. So I had to convert it to an AND
            jpigree Jonathan Pigrée added a comment -

            Oh, So you had another issue then. Glad that you resolved it.

            jpigree Jonathan Pigrée added a comment - Oh, So you had another issue then. Glad that you resolved it.
            daspilker Daniel Spilker made changes -
            Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]

            People

              daspilker Daniel Spilker
              jpigree Jonathan Pigrée
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: