Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43693

multibranchPipelineJob overrides old branch indexing sources

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • job-dsl-plugin
    • None
    • Debian 8
      Jenkins 2.46.1 (as service)

      Plugins (all newest stable):
      Job DSL 1.61
      Pipeline: Multibranch 2.14
    • job-dsl 1.75

    Description

      When running the multibranchPipelineJob step it seems like it overrides the sources of a Multibranch Pipeline Job instead of just updating it. After running the step, even though nothing has changed the output of the branch indexing will become:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
      Branch reopened: master (b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      Scheduled build for branch: master
      

       

      To test this, I have installed a brand new Jenkins with the Pipeline plugins and Job DSL plugin (all as the stable version) and made a freestyle job, with a Job DSL step that has the following script:

      multibranchPipelineJob('job-dsl-plugin_multibranch') {
        branchSources {
          git { 
            remote('https://github.com/jenkinsci/job-dsl-plugin.git')
          }
        }
      }
      

       

      After running the Job DSL step and afterwards running branch indexing a "master" job is created with the following log message from branch indexing:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      Scheduled build for branch: master
      

       

      When running branch indexing again, following log message are shown as expected:

      Checking branch master
            ‘Jenkinsfile’ found
      Met criteria
      No changes detected: master (still at b409f3996299dd12244d6cf7f8b6d8df3aa674ca)
      

       

      However, if I run the Job DSL again and run branch indexing, the first log message (with takeover and so on) are shown. This is not what I would have expected, I would have expected something like a "no changes detected" message.

       

      Attachments

        Issue Links

          Activity

            daspilker Daniel Spilker added a comment - - edited

            The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job.

            You can use the Automatically Generated DSL to set the ID:

            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                branchSource {
                  source {
                    git {
                      id('must-be-unique')
                      remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                      credentialsId(null)
                      remoteName('origin')
                      rawRefSpecs('+refs/heads/*:refs/remotes/origin/*')
                      includes('*')
                      excludes('')
                      ignoreOnPushNotifications(false)
                    }
                  }
                }
              }
            }
            
            daspilker Daniel Spilker added a comment - - edited The generated config XML for the branch source contains an <id> element. Job DSL generates a random ID each time the seed job runs. I think the Multibranch plugin uses this ID to manage the repository state. So it's not OK to generate a random ID with each run of the seed job. You can use the Automatically Generated DSL to set the ID: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { branchSource { source { git { id( 'must-be-unique' ) remote( 'https: //github.com/jenkinsci/job-dsl-plugin.git' ) credentialsId( null ) remoteName( 'origin' ) rawRefSpecs( '+refs/heads/*:refs/remotes/origin/*' ) includes( '*' ) excludes('') ignoreOnPushNotifications( false ) } } } } }
            rocktavious kyle rockman added a comment -

            HUZZAH!  I've been fighting with this bug for a month or two now at work. daspilker Thank you that fixed the issue!

            rocktavious kyle rockman added a comment - HUZZAH!  I've been fighting with this bug for a month or two now at work. daspilker Thank you that fixed the issue!

            Code changed in jenkins
            User: Daniel Spilker
            Path:
            docs/Home.md
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
            job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
            http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca
            Log:
            enhanced support for the Pipeline Multibranch Plugin

            [FIXES JENKINS-43693]

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/1bd2d95eaae1356de6b3f92613766527280f41ca Log: enhanced support for the Pipeline Multibranch Plugin [FIXES JENKINS-43693]

            Code changed in jenkins
            User: Daniel Spilker
            Path:
            docs/Home.md
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy
            job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy
            job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy
            http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130
            Log:
            Merge branch 'JENKINS-43693'

            Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: docs/Home.md job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitBranchSourceContext.groovy job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/workflow/GitHubBranchSourceContext.groovy job-dsl-core/src/test/groovy/javaposse/jobdsl/dsl/helpers/workflow/BranchSourcesContextsSpec.groovy http://jenkins-ci.org/commit/job-dsl-plugin/f8b51912215c545d702b6e8c8fbe6fc129fb3130 Log: Merge branch ' JENKINS-43693 ' Compare: https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c

            I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?

            chancez Chance Zibolski added a comment - I believe I am still hitting this issue. After updating to jobDSL 1.66, after running my jobDSL job, if I try to build one of the updated multibranch projects it fails with `ERROR: Could not determine exact tip revision of master`. Is it possible this is still unfixed?
            widgetpl Mike Dziedziela added a comment - - edited

            I face the same issue.

            Jenkins - v2.60.1

            Job DSL - v1.64

            Pipeline Multibranch - v2.16

            I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it.

            Checking branch develop
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » develop by source #1 from source that no longer exists
            Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471)
            Scheduled build for branch: develop
            
            Checking branch test
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » test by source #1 from source that no longer exists
            Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e)
            Scheduled build for branch: test
            
            Checking branch master
                  ‘Jenkinsfile’ found
                Met criteria
            Takeover for example_project » master by source #1 from source that no longer exists
            Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5)
            Scheduled build for branch: master

            let me know if you need more information.

            EDIT:

            Jenkins is integrated with GitLab v. 10.0.3 CE

            widgetpl Mike Dziedziela added a comment - - edited I face the same issue. Jenkins - v2.60.1 Job DSL - v1.64 Pipeline Multibranch - v2.16 I have just commited to test branch but scanning has triggered build for all branches which had Jenkinsfile in it. Checking branch develop ‘Jenkinsfile’ found Met criteria Takeover for example_project » develop by source #1 from source that no longer exists Branch reopened: develop (11617f79c6408e1d5fd54ea60a64d31b787a9471) Scheduled build for branch: develop Checking branch test ‘Jenkinsfile’ found Met criteria Takeover for example_project » test by source #1 from source that no longer exists Branch reopened: test (f84bcda107da005f9363759a68af2008a135a84e) Scheduled build for branch: test Checking branch master ‘Jenkinsfile’ found Met criteria Takeover for example_project » master by source #1 from source that no longer exists Branch reopened: master (2a2521fe4ea2ab71edb51b3279b5e49c4c8e77d5) Scheduled build for branch: master let me know if you need more information. EDIT: Jenkins is integrated with GitLab v. 10.0.3 CE
            kizzie Kat added a comment -

            So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6

            (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)

            kizzie Kat added a comment - So the changes made to fix this one seem to still use a random UUID - I'm not sure whether or not generating it based off the name or the repo being cloned (or why not both) would fix this - If I'm reading the problem correctly you need a static UUID being generated for the job for it to be able to cope  - https://github.com/jenkinsci/job-dsl-plugin/compare/c722b8bb538c...f8b51912215c#diff-f468c06e75a4e86856c95107995aaa0fR6 (and yes I'm also still getting this issue, and have been wondering what is causing it for a while - going to try setting the id manually for a bit instead)
            duemir Denys Digtiar added a comment -

            I have created a fresh environment from 2.121 and installing:

            • Pipeline: Multibranch 2.18
            • Github Branch Source Plugin 2.3.4
            • Job DSL 1.69

            I was able to reproduce the issue using the steps in the description. I can still see

            Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists
            Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020)
            Scheduled build for branch: master
            

            after I re-run the Seed Job.

            duemir Denys Digtiar added a comment - I have created a fresh environment from 2.121 and installing: Pipeline: Multibranch 2.18 Github Branch Source Plugin 2.3.4 Job DSL 1.69 I was able to reproduce the issue using the steps in the description. I can still see Takeover for job-dsl-plugin_multibranch » master by source #1 from source that no longer exists Branch reopened: master (5045094d82daa475fe79e7f5e52eba58c6df9020) Scheduled build for branch: master after I re-run the Seed Job.
            duemir Denys Digtiar added a comment - - edited

            Sorry, daspilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this:

            multibranchPipelineJob('job-dsl-plugin_multibranch') {
              branchSources {
                def repo = 'https://github.com/jenkinsci/job-dsl-plugin.git'
                git {
                  id = UUID.nameUUIDFromBytes(repo.getBytes())
                  remote(repo)
                }
              }
            }
            duemir Denys Digtiar added a comment - - edited Sorry, daspilker . It looks like I misunderstood the resolution. Is the resolution to manually add unique Ids to the source definitions? Like this: multibranchPipelineJob( 'job-dsl-plugin_multibranch' ) { branchSources { def repo = 'https: //github.com/jenkinsci/job-dsl-plugin.git' git { id = UUID.nameUUIDFromBytes(repo.getBytes()) remote(repo) } } }

            duemir yes, it worked for me and I use repository name for id.

            widgetpl Mike Dziedziela added a comment - duemir yes, it worked for me and I use repository name for id.

            This is how to do it for bitbucket-branch-source:

              branchSources {
              def repo = "https://bitbucket.com/..."
                branchSource {
                  source { 
                    bitbucket {
                      id(UUID.nameUUIDFromBytes(repo.getBytes()).toString())
                      serverUrl("...")
                      repository("...")
                      repoOwner("...")
                     ...
                    }
                  }
                }
              }
            namelessspear Vyacheslav Spiridonov added a comment - This is how to do it for bitbucket-branch-source: branchSources { def repo = "https: //bitbucket.com/..." branchSource { source { bitbucket { id(UUID.nameUUIDFromBytes(repo.getBytes()).toString()) serverUrl( "..." ) repository( "..." ) repoOwner( "..." ) ... } } } }
            rkivisto Ray Kivisto added a comment -

            Should this documentation be updated to include setting the id?

            https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob

            rkivisto Ray Kivisto added a comment - Should this documentation be updated to include setting the id ? https://jenkinsci.github.io/job-dsl-plugin/#path/multibranchPipelineJob
            jknurek J Knurek added a comment -

            I'm with rkivisto on this.

            I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected. 

            The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 

            jknurek J Knurek added a comment - I'm with rkivisto on this. I don't think this issue should be "Closed/Fix" if the default functionality still doesn't work as expected.  The ability to set the `id` does solve the problem, but in my opinion this is a workaround. And if a workaround is required, this should be clearly documented. Or even better, the default functionality should be the expected functionality. 
            rkivisto Ray Kivisto added a comment -

            Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.

            rkivisto Ray Kivisto added a comment - Re-opening so we can track the discussion about if this is truly fixed or not, or if we should just update the plugin documentation with the `id` setting.

            PR for improving the documentation and removing support for randomly generated identifiers:

            https://github.com/jenkinsci/job-dsl-plugin/pull/1187

            To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.

            daspilker Daniel Spilker added a comment - PR for improving the documentation and removing support for randomly generated identifiers: https://github.com/jenkinsci/job-dsl-plugin/pull/1187 To sum things up, you need to set a constant and unique identifier for each branch source to make things work. That's a delicacy of the SCM API plugin. There must be unique identifier for each branch source, no two branch sources either in the same or different jobs may share the same ID. It's your responsibility to keep the ID unique when using Job DSL. The ID must not change over time, e.g. on the next run of the seed job.
            renescheibe René Scheibe added a comment - - edited

            daspilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?

            renescheibe René Scheibe added a comment - - edited daspilker is there a way in Job DSL to resemble the automatic id creation as done when the job is created via the UI? Or can the id automatically be set only on job creation but not changed when updating the job?

            renescheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems.

            I'll close this issue since there no more known problems.

            daspilker Daniel Spilker added a comment - renescheibe Job DSL should not replicate any plugin logic. The behavior will change over time, will most likely fail silently and cause more problems. I'll close this issue since there no more known problems.
            renescheibe René Scheibe added a comment -

            Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced.

            That's for example all the sources under branchSources - branchSource which are for example:

            • bitbucket
            • git
            • github
            • mercurialSCMSource
            • subversionSCMSource

            Do you think these can also be covered somehow?

            renescheibe René Scheibe added a comment - Still an issue: Enforcing to manually set the id is only possible for branch sources explicitly implemented in Job DSL. For branch sources that are provided via "dynamic" DSL that's not enforced. That's for example all the sources under branchSources - branchSource which are for example: bitbucket git github mercurialSCMSource subversionSCMSource Do you think these can also be covered somehow?
            kivagant Eugene G added a comment -

            I faced the same issue even though I don't use the Job DSL plugin to configure a GitHub Organization pipeline. It was created in UI. Any workarounds?

            kivagant Eugene G added a comment - I faced the same issue even though I don't use the Job DSL plugin to configure a GitHub Organization pipeline. It was created in UI. Any workarounds?
            psufoxman Ryan Fox added a comment -

            Ditto here!  Would love to see resolution using the Job DSL plugin to configure a Gitlab Group pipeline

            psufoxman Ryan Fox added a comment - Ditto here!  Would love to see resolution using the Job DSL plugin to configure a Gitlab Group pipeline

            People

              daspilker Daniel Spilker
              anigif Anders Kielsholm
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: