Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38111

github organization should allow 'clean before checkout' behavior

      we are using the Github Organization plugin, and using a Jenkinsfile checked into each git repo in the organization.

      i want to be able to use the 'clean before checkout' option which is available when we configure pipeline builds outside of the github organization. it seems like this should be an option exposed in the organization configuration.

          [JENKINS-38111] github organization should allow 'clean before checkout' behavior

          In addition, I would prefer if those Additional Behaviors (scm.extensions) were available to the Jenkinsfile since some are not useful for every repository in a GitHub Org.

          Christian Höltje added a comment - In addition, I would prefer if those Additional Behaviors ( scm.extensions ) were available to the Jenkinsfile since some are not useful for every repository in a GitHub Org.

          Alex Dean added a comment - - edited

          re: making behaviors available via Jenkinsfile... it's unclear to me if projects which are being build via the GH Organization plugin need a 'checkout scm' step or not.

          I thought I would need this step to update to the latest code before building, but my build logs now contain redundant-seeming git operations. (Operations like "Fetching upstream changes from ..." are logged twice.) I suspect this means I might not need (or want) a 'checkout scm' step in my Jenkinsfile at all, but I can't find documentation one way or the other on this point.

          And tbh it makes sense that the plugin would handle the git operations for me. Otherwise there's a chicken/egg problem - the organization plugin has to check out the code in order to get the Jenkinsfile to start the build in the first place, at which point there doesn't seem to be much point to a second 'checkout scm' step.

          Alex Dean added a comment - - edited re: making behaviors available via Jenkinsfile... it's unclear to me if projects which are being build via the GH Organization plugin need a 'checkout scm' step or not. I thought I would need this step to update to the latest code before building, but my build logs now contain redundant-seeming git operations. (Operations like "Fetching upstream changes from ..." are logged twice.) I suspect this means I might not need (or want) a 'checkout scm' step in my Jenkinsfile at all, but I can't find documentation one way or the other on this point. And tbh it makes sense that the plugin would handle the git operations for me. Otherwise there's a chicken/egg problem - the organization plugin has to check out the code in order to get the Jenkinsfile to start the build in the first place, at which point there doesn't seem to be much point to a second 'checkout scm' step.

          The way GitHub Org Folder works is the first git checkout you see is the Jenkins Server (in a "sub job" or "mini job" or whatever it is called) checking out the source on the Jenkins Server (In a directory like JENKINS_HOME/workspace/<github-org>/<github-repo>/<branch>@script}}) to get the Jenkinsfile.

          It then uses that Jenkinsfile to actually do the build. The Jenkinsfile then allocates Workspaces via node() {} commands which may-or-may-not be empty.

          Christian Höltje added a comment - The way GitHub Org Folder works is the first git checkout you see is the Jenkins Server (in a "sub job" or "mini job" or whatever it is called) checking out the source on the Jenkins Server (In a directory like JENKINS_HOME /workspace/<github-org>/<github-repo>/<branch>@script}}) to get the Jenkinsfile . It then uses that Jenkinsfile to actually do the build. The Jenkinsfile then allocates Workspaces via node() { } commands which may-or-may-not be empty.

          Mark Waite added a comment -

          As far as I know, those behaviors are available from jobs generated by the GitHub Organizations Folder plugin, so long as the behavior is added in the Jenkinsfile. Refer to [the JENKINS-22547 branch}https://github.com/MarkEWaite/jenkins-bugs/blob/JENKINS-22547/Jenkinsfile] of my jenkins-bugs repository for an example of a Jenkinsfile which includes several additional behaviors added to the checkout() step, including CleanBeforeCheckout.

          Could you try that technique and see if it works as expected in your GitHub Organization Folders?

          Mark Waite added a comment - As far as I know, those behaviors are available from jobs generated by the GitHub Organizations Folder plugin, so long as the behavior is added in the Jenkinsfile. Refer to [the JENKINS-22547 branch}https://github.com/MarkEWaite/jenkins-bugs/blob/JENKINS-22547/Jenkinsfile] of my jenkins-bugs repository for an example of a Jenkinsfile which includes several additional behaviors added to the checkout() step, including CleanBeforeCheckout. Could you try that technique and see if it works as expected in your GitHub Organization Folders?

          Alex Dean added a comment -

          Mark: Thank you for the idea. While I think that would work for now, I'm concerned about needing to statically specify so much of what is already provided by the built-in `scm` global variable. (As Christian also raises in JENKINS-37817.)

          Christian: Thank you for that explanation. Makes sense. Based on your explanation, it sounds like being able to specify 'clean before checkout' in the org plugin wouldn't accomplish what I want anyway. (Since I think i'd be cleaning the `<branch>@script` working copy rather than the actual workspace where the build it going to happen? If I've got that right, then I think this issue is probably invalid.

          Alex Dean added a comment - Mark: Thank you for the idea. While I think that would work for now, I'm concerned about needing to statically specify so much of what is already provided by the built-in `scm` global variable. (As Christian also raises in JENKINS-37817 .) Christian: Thank you for that explanation. Makes sense. Based on your explanation, it sounds like being able to specify 'clean before checkout' in the org plugin wouldn't accomplish what I want anyway. (Since I think i'd be cleaning the `<branch>@script` working copy rather than the actual workspace where the build it going to happen? If I've got that right, then I think this issue is probably invalid.

          Right. There are two issues, really...

          1. The <branch>@script checkout on the master should probably always use shallow to prevent huge checkout times. If there was another way to just get the correct Jenkinsfile (e.g. https://raw.<github-url>/<org>/<repo>/<branch-or-ref>/Jenkinsfile?token=<token>) that'd be even better.
          2. The checkout scm needs to be configurable so changes specific to a repository can be setup, such as setting "clean before checkout" or "shallow clone". Otherwise this cannot be setup due to the way the GitHub Org Folder works.

          The second item is what I'm concerned about here. I should file another issue on the first item.

          Christian Höltje added a comment - Right. There are two issues, really... The <branch>@script checkout on the master should probably always use shallow to prevent huge checkout times. If there was another way to just get the correct Jenkinsfile (e.g. https://raw .<github-url>/<org>/<repo>/<branch-or-ref>/Jenkinsfile?token=<token> ) that'd be even better. The checkout scm needs to be configurable so changes specific to a repository can be setup, such as setting "clean before checkout" or "shallow clone". Otherwise this cannot be setup due to the way the GitHub Org Folder works. The second item is what I'm concerned about here. I should file another issue on the first item.

          I created JENKINS-38212 to address the "getting the Jenkinsfile" story.

          Christian Höltje added a comment - I created JENKINS-38212 to address the "getting the Jenkinsfile" story.

          Alex Dean added a comment -

          closing because i believe that allowing additional git operations via Github Organization configuration will not have the effect I want to achieve. i want to remove old artifacts from a workspace prior to a build, but i believe that running `git clean` & etc during the initial checkout would actually clean the `project@script` working copy instead.

          if a maintainer thinks i've summarized this incorrectly, please re-open.

          Alex Dean added a comment - closing because i believe that allowing additional git operations via Github Organization configuration will not have the effect I want to achieve. i want to remove old artifacts from a workspace prior to a build, but i believe that running `git clean` & etc during the initial checkout would actually clean the `project@script` working copy instead. if a maintainer thinks i've summarized this incorrectly, please re-open.

          markewaite I tried your Jenkinsfile against our Git Hub Enterprise and it failed because it doesn't have the correct credentials. I could shoe-horn them in there someplace (not sure where, since this interface is confusing).

          Either way, having to write that massive amount of boiler plate when all the values are in the scm object right now is really bad UX design.

          Christian Höltje added a comment - markewaite I tried your Jenkinsfile against our Git Hub Enterprise and it failed because it doesn't have the correct credentials. I could shoe-horn them in there someplace (not sure where, since this interface is confusing). Either way, having to write that massive amount of boiler plate when all the values are in the scm object right now is really bad UX design.

          alexdean It isn't clear what added the web-ui options would do. It would probably do both the <branch>@script and the checkout scm. Which would be better than nothing, but not allow the per-repository control that people really need.

          Christian Höltje added a comment - alexdean It isn't clear what added the web-ui options would do. It would probably do both the <branch>@script and the checkout scm . Which would be better than nothing, but not allow the per-repository control that people really need.

          Mark Waite added a comment - - edited

          docwhat when you say that people need "per-repository control", do you mean that they want a setting which controls all repositories within the entire organization (in my case, that would be "git-plugin", "git-client-plugin", and "jenkins-bugs"), or would it be a setting per repository (one each for "git-plugin", "git-client-plugin", and "jenkins-bugs"), or would it be a setting per branch ("git-plugin:master", "git-plugin:3.0.0-beta", etc.)?

          I assumed from the phrase "per-repository" that you meant one for each repository in the organization. If that is the case, then doesn't that mean that the definition needs to be associated to each repository? I'm not sure how we would associate the definition to a repository, since currently the Jenkinsfile is per branch. I guess we could declare that the "per-repository" control is the Jenkinsfile which comes from the master branch, though the git purists will likely complain that master is just a convention, not a mandate.

          Mark Waite added a comment - - edited docwhat when you say that people need "per-repository control", do you mean that they want a setting which controls all repositories within the entire organization (in my case, that would be "git-plugin", "git-client-plugin", and "jenkins-bugs"), or would it be a setting per repository (one each for "git-plugin", "git-client-plugin", and "jenkins-bugs"), or would it be a setting per branch ("git-plugin:master", "git-plugin:3.0.0-beta", etc.)? I assumed from the phrase "per-repository" that you meant one for each repository in the organization. If that is the case, then doesn't that mean that the definition needs to be associated to each repository? I'm not sure how we would associate the definition to a repository, since currently the Jenkinsfile is per branch. I guess we could declare that the "per-repository" control is the Jenkinsfile which comes from the master branch, though the git purists will likely complain that master is just a convention, not a mandate.

          By "per repository" I guess I really meant "per branch". Specifically, control it via a Jenkinsfile. Ideally by modifying the scm object or creating a copy in a simple way (not by copying every element one by one).

          Some examples that I have been recently asked about:

          • Shallow clones
          • Altering timeouts
          • Clean before/after checkouts
          • Calculate Changelog against specific branch
          • etc.

          Christian Höltje added a comment - By "per repository" I guess I really meant "per branch". Specifically, control it via a Jenkinsfile . Ideally by modifying the scm object or creating a copy in a simple way (not by copying every element one by one). Some examples that I have been recently asked about: Shallow clones Altering timeouts Clean before/after checkouts Calculate Changelog against specific branch etc.

          Mark Waite added a comment - - edited

          Those can all be set through the checkout() step. They may also be accessible from the scm object. I've not experimented with trying to use them from the scm object. If I find a way to read the existing value from the scm object so that it can be reused in the checkout() step, I'll certainly post it here.

          Mark Waite added a comment - - edited Those can all be set through the checkout() step. They may also be accessible from the scm object. I've not experimented with trying to use them from the scm object. If I find a way to read the existing value from the scm object so that it can be reused in the checkout() step, I'll certainly post it here.

          Christian Höltje added a comment - - edited

          They cannot be read without triggering the need for a lot of accept approvals. See the linked issue: JENKINS-37817 for the list the last time I tried. That list grew as of yesterday's versions of things.

          Christian Höltje added a comment - - edited They cannot be read without triggering the need for a lot of accept approvals. See the linked issue: JENKINS-37817 for the list the last time I tried. That list grew as of yesterday's versions of things.

            kohsuke Kohsuke Kawaguchi
            alexdean Alex Dean
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: