Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46202

Cannot discover all GitHub repos and their branches with Job DSL anymore

      Using the job DSL plugin v. 1.64, I can't automatically discover all my repositories and all it's branches within my GitHub organization folder.

      I used the job DSL plugin v. 1.61 before.

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          github {
            apiUri("")
            repoOwner("myOrg")
            scanCredentialsId("github")
            checkoutCredentialsId("github")
          }
        }
      }

       

      This is the configuration view after executing the above script


      scanCredentialsId("") and checkoutCredentialsId("") are now deprecated and replaced by credentialsId("github"). Now, the credentials work. But discovering all repositories and their branches doesn't seem to be possible anymore because the "includes", "excludes" methods also have been deprecated.
      Now, with the following, fixed script, I don't get all the branches anymore:

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          github {
            repoOwner("myOrg")
            credentialsId("github")
          }
        }
      }

       

      This is the configuration page after executing the new script:

      I now manually need to add the checkout behaviour "Discover branches -> All branches"

      How can I automatically discover all repos and their branches using the DSL script?

      I also tried the "fromSource" syntax as this one seems to have more options.

       

      organizationFolder('myOrg') {
        triggers {
          periodic(10)
        }
        organizations {
          fromSource {
            name("myOrg")
            sources {
              github {
                credentialsId("github")
                repoOwner("myOrg")
                repository("")
              }
            }
          }
        }
      }
      

       

      As you can see, I have tried to leave the string in the repository("") method empty, hoping that would cause it to discover all repos within my organization. I also tried asterisk: repository("*") which also led to an error.

          [JENKINS-46202] Cannot discover all GitHub repos and their branches with Job DSL anymore

          Workaround:

          configure {
            def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
            traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' {
              strategyId(1)
            }
            traits << 'org.jenkinsci.plugins.github__branch__source.OriginPullRequestDiscoveryTrait' {
             strategyId(1)
           }
          }

          Steffen Gebert added a comment - Workaround: configure { def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' { strategyId(1) } traits << 'org.jenkinsci.plugins.github__branch__source.OriginPullRequestDiscoveryTrait' { strategyId(1) } }

          Steffen Gebert added a comment - - edited

          Auto-generated DSL only shows multibranchPipelineJob -> branchSources -> branchSource -> source -> github -> traits -> originPullRequestDiscoveryTrait, not the other traits.

          Steffen Gebert added a comment - - edited Auto-generated DSL only shows multibranchPipelineJob -> branchSources -> branchSource -> source -> github -> traits -> originPullRequestDiscoveryTrait , not the other traits.

          The Automatically Generated DSL does not show traits because the structs plugin does not handle wildcards well in generic type declarations. See JENKINS-26535.

          To reproduce this, run the following script in Script Console:

          import org.jenkinsci.plugins.structs.describable.DescribableModel
          import org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator
          
          new DescribableModel(GitHubSCMNavigator)
          

          The output is:

          Result: GitHubSCMNavigator(repoOwner: String, apiUri?: String, buildForkPRHead?(deprecated): boolean, buildForkPRMerge?(deprecated): boolean, buildOriginBranch?(deprecated): boolean, buildOriginBranchWithPR?(deprecated): boolean, buildOriginPRHead?(deprecated): boolean, buildOriginPRMerge?(deprecated): boolean, credentialsId?: String, excludes?(deprecated): String, includes?(deprecated): String, pattern?(deprecated): String, scanCredentialsId?(deprecated): String, traits?: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMTrait<? extends jenkins.scm.api.trait.SCMTrait<?>>[])
          

          Daniel Spilker added a comment - The Automatically Generated DSL does not show traits because the structs plugin does not handle wildcards well in generic type declarations. See JENKINS-26535 . To reproduce this, run the following script in Script Console: import org.jenkinsci.plugins.structs.describable.DescribableModel import org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator new DescribableModel(GitHubSCMNavigator) The output is: Result: GitHubSCMNavigator(repoOwner: String , apiUri?: String , buildForkPRHead?(deprecated): boolean , buildForkPRMerge?(deprecated): boolean , buildOriginBranch?(deprecated): boolean , buildOriginBranchWithPR?(deprecated): boolean , buildOriginPRHead?(deprecated): boolean , buildOriginPRMerge?(deprecated): boolean , credentialsId?: String , excludes?(deprecated): String , includes?(deprecated): String , pattern?(deprecated): String , scanCredentialsId?(deprecated): String , traits?: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMTrait<? extends jenkins.scm.api.trait.SCMTrait<?>>[])

          Steffen Gebert added a comment - - edited

          Thanks Daniel for your quick feedback.

          Actually, it is showing the traits section to me, but lacking all but the originPullRequestDiscoveryTrait.

          Maybe it is working and the issue is only my naive try to specify it:

                      source {
                          github {
                              id('my-repo')
                              repoOwner('my-org')
                              repository('my-repo')
                              credentialsId('github-token')
                              apiUri('https://api.github.com')
                          }
                          traits {
                              originPullRequestDiscoveryTrait {
                                  strategyId(1)
                              }
                          }
                      }
          

          This results in

          ERROR: (script, line 24) No signature of method: javaposse.jobdsl.plugin.structs.DescribableListContext.traits() is applicable for argument types: (script$_run_closure1$_closure2$_closure6$_closure7$_closure10) values: [script$_run_closure1$_closure2$_closure6$_closure7$_closure10@d804df7]
          Possible solutions: wait(), wait(long), wait(long, int), split(groovy.lang.Closure), withTraits([Ljava.lang.Class;), is(java.lang.Object)
          

          It's listing a withTraits() method. Can we call this somehow?

          I've attached a config.xml of a manually created multi-branch job (including the <traits> section). multibranch-config-manual.xml

          Steffen Gebert added a comment - - edited Thanks Daniel for your quick feedback. Actually, it is showing the traits section to me, but lacking all but the originPullRequestDiscoveryTrait . Maybe it is working and the issue is only my naive try to specify it: source { github { id( 'my-repo' ) repoOwner( 'my-org' ) repository( 'my-repo' ) credentialsId( 'github-token' ) apiUri( 'https: //api.github.com' ) } traits { originPullRequestDiscoveryTrait { strategyId(1) } } } This results in ERROR: (script, line 24) No signature of method: javaposse.jobdsl.plugin.structs.DescribableListContext.traits() is applicable for argument types: (script$_run_closure1$_closure2$_closure6$_closure7$_closure10) values: [script$_run_closure1$_closure2$_closure6$_closure7$_closure10@d804df7] Possible solutions: wait(), wait(long), wait(long, int), split(groovy.lang.Closure), withTraits([Ljava.lang.Class;), is(java.lang.Object) It's listing a withTraits() method. Can we call this somehow? I've attached a config.xml of a manually created multi-branch job (including the <traits>  section).  multibranch-config-manual.xml

          This bug report was originally about the github-organization-folder-plugin but is actually a problem with the branch sources (see below), so it applies to workflow-multibranch as well.

          stephenking I see a lot of traits in my embedded API viewer (Jenkins 2.19.1, latest version of all plugins), but not the org.jenkinsci.plugins.github_branchsource.BranchDiscoveryTrait and org.jenkinsci.plugins.githubbranch_source.ForkPullRequestDiscoveryTrait traits from your config XML.

          To debug the problem, I ran the following script in Script Console:

          import org.jenkinsci.plugins.structs.describable.DescribableModel
          import org.jenkinsci.plugins.github_branch_source.GitHubSCMSource
          
          new DescribableModel(GitHubSCMSource)
          

          The output is (formatted for readability):

          GitHubSCMSource(
            repoOwner: String, 
            repository: String, 
            apiUri?: String, 
            buildForkPRHead?(deprecated): boolean, 
            buildForkPRMerge?(deprecated): boolean, 
            buildOriginBranch?(deprecated): boolean, 
            buildOriginBranchWithPR?(deprecated): boolean, 
            buildOriginPRHead?(deprecated): boolean, 
            buildOriginPRMerge?(deprecated): boolean, 
            credentialsId?: String, 
            excludes?(deprecated): String, 
            id?: String, 
            includes?(deprecated): String, 
            traits?: SCMSourceTrait{AuthorInChangelogTrait() | 
                     CheckoutOptionTrait(extension: CheckoutOption(timeout: int)) | 
          		   CleanAfterCheckoutTrait() | 
          		   CleanBeforeCheckoutTrait() | 
          		   CloneOptionTrait(extension: CloneOption(shallow: boolean, noTags: boolean, reference: String, timeout: int, depth?: int, honorRefspec?: boolean)) |
          		   ForkPullRequestDiscoveryTrait(strategyId: int, trust: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMHeadAuthority<? super org.jenkinsci.plugins.github_branch_source.GitHubSCMSourceRequest, ? extends jenkins.scm.api.mixin.ChangeRequestSCMHead2, ? extends jenkins.scm.api.SCMRevision>) | 
          		   GitBrowserSCMSourceTrait(browser: GitRepositoryBrowser{AssemblaWeb(repoUrl: String) | BitbucketWeb(repoUrl: String) | CGit(repoUrl: String) | FisheyeGitRepositoryBrowser(repoUrl: String) | GitBlitRepositoryBrowser(repoUrl: String, projectName: String) | GitLab(repoUrl: String, version: String) | GitList(repoUrl: String) | GitWeb(repoUrl: String) | GithubWeb(repoUrl: String) | Gitiles(repoUrl: String) | GitoriousWeb(repoUrl: String) | GogsGit(repoUrl: String) | KilnGit(repoUrl: String) | Phabricator(repoUrl: String, repo: String) | RedmineWeb(repoUrl: String) | RhodeCode(repoUrl: String) | Stash(repoUrl: String) | TFS2013GitRepositoryBrowser(repoUrl: String) | ViewGitWeb(repoUrl: String, projectName: String)}) |
          		   GitLFSPullTrait() | 
          		   GitToolSCMSourceTrait(gitTool: String) |
          		   IgnoreOnPushNotificationTrait() | 
          		   LocalBranchTrait() |
          		   OriginPullRequestDiscoveryTrait(strategyId: int) | 
          		   PruneStaleBranchTrait() | 
          		   RefSpecsSCMSourceTrait(templates: RefSpecTemplate(value: String)[]) | 
          		   RegexSCMHeadFilterTrait(regex: String) |
          		   RemoteNameSCMSourceTrait(remoteName: String) |
          		   SSHCheckoutTrait(credentialsId: String) | 
          		   SubmoduleOptionTrait(extension: SubmoduleOption(disableSubmodules: boolean, recursiveSubmodules: boolean, trackingSubmodules: boolean, reference: String, timeout: int, parentCredentials: boolean)) | 
          		   UserIdentityTrait(extension: UserIdentity(name: String, email: String)) | 
          		   WildcardSCMHeadFilterTrait(includes: String, excludes: String) | 
          		   WipeWorkspaceTrait() |
          		   jenkins.plugins.git.traits.BranchDiscoveryTrait~BranchDiscoveryTrait() | 
          		   org.jenkinsci.plugins.github_branch_source.BranchDiscoveryTrait~BranchDiscoveryTrait(strategyId: int)}[])
          

          As one can see, the problem with ForkPullRequestDiscoveryTrait is that the trust property is not supported by structs-plugin, probably because of JENKINS-26535.

          The BranchDiscoveryTrait is not available because there are two classes named BranchDiscoveryTrait in different packages (jenkins.plugins.git.traits and org.jenkinsci.plugins.github_branch_source). Job DSL only uses the class name to determine the class and for BranchDiscoveryTrait that's ambigious, so it's not available. github-branch-source-plugin and/or git-plugin can use a @Symbol annotation to set an unambigious alias.

          IMHO this needs to fixed in github-branch-source-plugin, git-plugin and/or structs-plugin. Pinging stephenconnolly, markewaite and jglick.

          Daniel Spilker added a comment - This bug report was originally about the github-organization-folder-plugin but is actually a problem with the branch sources (see below), so it applies to workflow-multibranch as well. stephenking I see a lot of traits in my embedded API viewer (Jenkins 2.19.1, latest version of all plugins), but not the org.jenkinsci.plugins.github_ branch source.BranchDiscoveryTrait and org.jenkinsci.plugins.github branch _source.ForkPullRequestDiscoveryTrait traits from your config XML. To debug the problem, I ran the following script in Script Console: import org.jenkinsci.plugins.structs.describable.DescribableModel import org.jenkinsci.plugins.github_branch_source.GitHubSCMSource new DescribableModel(GitHubSCMSource) The output is (formatted for readability): GitHubSCMSource( repoOwner: String , repository: String , apiUri?: String , buildForkPRHead?(deprecated): boolean , buildForkPRMerge?(deprecated): boolean , buildOriginBranch?(deprecated): boolean , buildOriginBranchWithPR?(deprecated): boolean , buildOriginPRHead?(deprecated): boolean , buildOriginPRMerge?(deprecated): boolean , credentialsId?: String , excludes?(deprecated): String , id?: String , includes?(deprecated): String , traits?: SCMSourceTrait{AuthorInChangelogTrait() | CheckoutOptionTrait(extension: CheckoutOption(timeout: int )) | CleanAfterCheckoutTrait() | CleanBeforeCheckoutTrait() | CloneOptionTrait(extension: CloneOption(shallow: boolean , noTags: boolean , reference: String , timeout: int , depth?: int , honorRefspec?: boolean )) | ForkPullRequestDiscoveryTrait(strategyId: int , trust: java.lang.UnsupportedOperationException: do not know how to categorize attributes of type jenkins.scm.api.trait.SCMHeadAuthority<? super org.jenkinsci.plugins.github_branch_source.GitHubSCMSourceRequest, ? extends jenkins.scm.api.mixin.ChangeRequestSCMHead2, ? extends jenkins.scm.api.SCMRevision>) | GitBrowserSCMSourceTrait(browser: GitRepositoryBrowser{AssemblaWeb(repoUrl: String ) | BitbucketWeb(repoUrl: String ) | CGit(repoUrl: String ) | FisheyeGitRepositoryBrowser(repoUrl: String ) | GitBlitRepositoryBrowser(repoUrl: String , projectName: String ) | GitLab(repoUrl: String , version: String ) | GitList(repoUrl: String ) | GitWeb(repoUrl: String ) | GithubWeb(repoUrl: String ) | Gitiles(repoUrl: String ) | GitoriousWeb(repoUrl: String ) | GogsGit(repoUrl: String ) | KilnGit(repoUrl: String ) | Phabricator(repoUrl: String , repo: String ) | RedmineWeb(repoUrl: String ) | RhodeCode(repoUrl: String ) | Stash(repoUrl: String ) | TFS2013GitRepositoryBrowser(repoUrl: String ) | ViewGitWeb(repoUrl: String , projectName: String )}) | GitLFSPullTrait() | GitToolSCMSourceTrait(gitTool: String ) | IgnoreOnPushNotificationTrait() | LocalBranchTrait() | OriginPullRequestDiscoveryTrait(strategyId: int ) | PruneStaleBranchTrait() | RefSpecsSCMSourceTrait(templates: RefSpecTemplate(value: String )[]) | RegexSCMHeadFilterTrait(regex: String ) | RemoteNameSCMSourceTrait(remoteName: String ) | SSHCheckoutTrait(credentialsId: String ) | SubmoduleOptionTrait(extension: SubmoduleOption(disableSubmodules: boolean , recursiveSubmodules: boolean , trackingSubmodules: boolean , reference: String , timeout: int , parentCredentials: boolean )) | UserIdentityTrait(extension: UserIdentity(name: String , email: String )) | WildcardSCMHeadFilterTrait(includes: String , excludes: String ) | WipeWorkspaceTrait() | jenkins.plugins.git.traits.BranchDiscoveryTrait~BranchDiscoveryTrait() | org.jenkinsci.plugins.github_branch_source.BranchDiscoveryTrait~BranchDiscoveryTrait(strategyId: int )}[]) As one can see, the problem with ForkPullRequestDiscoveryTrait is that the trust property is not supported by structs-plugin, probably because of JENKINS-26535 . The BranchDiscoveryTrait is not available because there are two classes named BranchDiscoveryTrait in different packages ( jenkins.plugins.git.traits and org.jenkinsci.plugins.github_branch_source ). Job DSL only uses the class name to determine the class and for BranchDiscoveryTrait that's ambigious, so it's not available. github-branch-source-plugin and/or git-plugin can use a @Symbol annotation to set an unambigious alias. IMHO this needs to fixed in github-branch-source-plugin, git-plugin and/or structs-plugin. Pinging stephenconnolly , markewaite and jglick .

          I ran into this issue today, have there been any changes to make this work?

          Or do I still need to use the workaround of manually adding to the XML in the configure block?

          Alex Richardson added a comment - I ran into this issue today, have there been any changes to make this work? Or do I still need to use the workaround of manually adding to the XML in the configure block?

          John Lonergan added a comment -

          +1 it would be helpful to include a link to the resolution?

          John Lonergan added a comment - +1 it would be helpful to include a link to the resolution?

            Unassigned Unassigned
            spiderfox Christian H
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: