Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43693

multibranchPipelineJob overrides old branch indexing sources

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • job-dsl-plugin
    • None
    • Debian 8
      Jenkins 2.46.1 (as service)

      Plugins (all newest stable):
      Job DSL 1.61
      Pipeline: Multibranch 2.14
    • job-dsl 1.75

      When running the multibranchPipelineJob step it seems like it overrides the sources of a Multibranch Pipeline Job instead of just updating it. After running the step, even though nothing has changed the output of the branch indexing will become:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
      Branch reopened: master (b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      Scheduled build for branch: master
      

       

      To test this, I have installed a brand new Jenkins with the Pipeline plugins and Job DSL plugin (all as the stable version) and made a freestyle job, with a Job DSL step that has the following script:

      multibranchPipelineJob('job-dsl-plugin_multibranch') {
        branchSources {
          git { 
            remote('https://github.com/jenkinsci/job-dsl-plugin.git')
          }
        }
      }
      

       

      After running the Job DSL step and afterwards running branch indexing a "master" job is created with the following log message from branch indexing:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Scheduled build for branch: master
      

       

      When running branch indexing again, following log message are shown as expected:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: master (still at b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      

       

      However, if I run the Job DSL again and run branch indexing, the first log message (with takeover and so on) are shown. This is not what I would have expected, I would have expected something like a "no changes detected" message.

       

          [JENKINS-43693] multibranchPipelineJob overrides old branch indexing sources

          Daniel Spilker added a comment - - edited

          The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job.

          You can use the Automatically Generated DSL to set the ID:

          multibranchPipelineJob('job-dsl-plugin_multibranch') {
            branchSources {
              branchSource {
                source {
                  git {
                    id('must-be-unique')
                    remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                    credentialsId(null)
                    remoteName('origin')
                    rawRefSpecs('+refs/heads/*:refs/remotes/origin/*')
                    includes('*')
                    excludes('')
                    ignoreOnPushNotifications(false)
                  }
                }
              }
            }
          }
          

          Daniel Spilker added a comment - - edited The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job. You can use the Automatically Generated DSL to set the ID: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { branchSource { source { git { id( 'must-be-unique' ) remote( 'https: //github.com/jenkinsci/job-dsl-plugin.git' ) credentialsId( null ) remoteName( 'origin' ) rawRefSpecs( '+refs/heads/*:refs/remotes/origin/*' ) includes( '*' ) excludes('') ignoreOnPushNotifications( false ) } } } } }

          kyle rockman added a comment -

          HUZZAH!  I've been fighting with this bug for a month or two now at work. daspilker Thank you that fixed the issue!

          kyle rockman added a comment - HUZZAH!  I've been fighting with this bug for a month or two now at work. daspilker Thank you that fixed the issue!

          Code changed in jenkins
          User: Daniel Spilker
          Path:
          docs/Home.md
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
          job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
          http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca
          Log:
          enhanced support for the Pipeline Multibranch Plugin

          [FIXES JENKINS-43693]

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca Log: enhanced support for the Pipeline Multibranch Plugin [FIXES JENKINS-43693]

          Code changed in jenkins
          User: Daniel Spilker
          Path:
          docs/Home.md
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
          job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
          job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
          http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130
          Log:
          Merge branch 'JENKINS-43693'

          Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130 Log: Merge branch ' JENKINS-43693 ' Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c

          I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?

          Chance Zibolski added a comment - I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?

          Mike Dziedziela added a comment - - edited

          I face the same issue.

          Jenkins - v2.60.1

          Job DSL - v1.64

          Pipeline Multibranch - v2.16

          I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it.

          Checking branch develop
                ‘Jenkinsfile’ found
              Met criteria
          Takeover for example_project » develop by source #1 from source that no longer exists
          Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471)
          Scheduled build for branch: develop
          
          Checking branch test
                ‘Jenkinsfile’ found
              Met criteria
          Takeover for example_project » test by source #1 from source that no longer exists
          Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e)
          Scheduled build for branch: test
          
          Checking branch master
                ‘Jenkinsfile’ found
              Met criteria
          Takeover for example_project » master by source #1 from source that no longer exists
          Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5)
          Scheduled build for branch: master

          let me know if you need more information.

          EDIT:

          Jenkins is integrated with GitLab v. 10.0.3 CE

          Mike Dziedziela added a comment - - edited I face the same issue. Jenkins - v2.60.1 Job DSL - v1.64 Pipeline Multibranch - v2.16 I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it. Checking branch develop ‘Jenkinsfile’ found Met criteria Takeover for example_project » develop by source #1 from source that no longer exists Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471) Scheduled build for branch: develop Checking branch test ‘Jenkinsfile’ found Met criteria Takeover for example_project » test by source #1 from source that no longer exists Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e) Scheduled build for branch: test Checking branch master ‘Jenkinsfile’ found Met criteria Takeover for example_project » master by source #1 from source that no longer exists Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5) Scheduled build for branch: master let me know if you need more information. EDIT: Jenkins is integrated with GitLab v. 10.0.3 CE

          Kat added a comment -

          So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6

          (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)

          Kat added a comment - So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6 (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)

          Denys Digtiar added a comment -

          I have created a fresh environment from 2.121 and installing:

          • Pipeline: Multibranch 2.18
          • Github Branch Source Plugin 2.3.4
          • Job DSL 1.69

          I was able to reproduce the issue using the steps in the description. I can still see

          Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
          Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020)
          Scheduled build for branch: master
          

          after I re-run the Seed Job.

          Denys Digtiar added a comment - I have created a fresh environment from 2.121 and installing: Pipeline: Multibranch 2.18 Github Branch Source Plugin 2.3.4 Job DSL 1.69 I was able to reproduce the issue using the steps in the description. I can still see Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020) Scheduled build for branch: master after I re-run the Seed Job.

          Denys Digtiar added a comment - - edited

          Sorry, daspilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this:

          multibranchPipelineJob('job-dsl-plugin_multibranch') {
            branchSources {
              def repo = 'https://github.com/jenkinsci/job-dsl-plugin.git'
              git {
                id = UUID.nameUUIDFromBytes(repo.getBytes())
                remote(repo)
              }
            }
          }

          Denys Digtiar added a comment - - edited Sorry, daspilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { def repo = 'https: //github.com/jenkinsci/job-dsl-plugin.git' git { id = UUID.nameUUIDFromBytes(repo.getBytes()) remote(repo) } } }

          duemir yes, it worked for me and I use repository name for id.

          Mike Dziedziela added a comment - duemir yes, it worked for me and I use repository name for id.

          This is how to do it for bitbucket-branch-source:

            branchSources {
            def repo = "https://bitbucket.com/..."
              branchSource {
                source { 
                  bitbucket {
                    id(UUID.nameUUIDFromBytes(repo.getBytes()).toString())
                    serverUrl("...")
                    repository("...")
                    repoOwner("...")
                   ...
                  }
                }
              }
            }

          Vyacheslav Spiridonov added a comment - This is how to do it for bitbucket-branch-source: branchSources { def repo = "https: //bitbucket.com/..." branchSource { source { bitbucket { id(UUID.nameUUIDFromBytes(repo.getBytes()).toString()) serverUrl( "..." ) repository( "..." ) repoOwner( "..." ) ... } } } }

          Ray Kivisto added a comment -

          Should this documentation be updated to include setting the id?

          https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob

          Ray Kivisto added a comment - Should this documentation be updated to include setting the id ? https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob

          J Knurek added a comment -

          I'm with rkivisto on this.

          I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected. 

          The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 

          J Knurek added a comment - I'm with rkivisto on this. I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected.  The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 

          Ray Kivisto added a comment -

          Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.

          Ray Kivisto added a comment - Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.

          PR for improving the documentation and removing support for randomly generated identifiers:

          https://github.com/jenkinsci/job-dsl-plugin/pull/1187

          To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.

          Daniel Spilker added a comment - PR for improving the documentation and removing support for randomly generated identifiers: https://github.com/jenkinsci/job-dsl-plugin/pull/1187 To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.

          René Scheibe added a comment - - edited

          daspilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?

          René Scheibe added a comment - - edited daspilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?

          renescheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems.

          I'll close this issue since there no more known problems.

          Daniel Spilker added a comment - renescheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems. I'll close this issue since there no more known problems.

          René Scheibe added a comment -

          Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced.

          That's for example all the sources under branchSources - branchSource which are for example:

          • bitbucket
          • git
          • github
          • mercurialSCMSource
          • subversionSCMSource

          Do you think these can also be covered somehow?

          René Scheibe added a comment - Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced. That's for example all the sources under branchSources - branchSource which are for example: bitbucket git github mercurialSCMSource subversionSCMSource Do you think these can also be covered somehow?

          Eugene G added a comment -

          I faced the same issue even though I don't use the Job DSL plugin to configure a GitHub Organization pipeline. It was created in UI. Any workarounds?

          Eugene G added a comment - I faced the same issue even though I don't use the Job DSL plugin to configure a GitHub Organization pipeline. It was created in UI. Any workarounds?

          Ryan Fox added a comment -

          Ditto here!  Would love to see resolution using the Job DSL plugin to configure a Gitlab Group pipeline

          Ryan Fox added a comment - Ditto here!  Would love to see resolution using the Job DSL plugin to configure a Gitlab Group pipeline

            daspilker Daniel Spilker
            anigif Anders Kielsholm
            Votes:
            1 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: