Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48510

Pipeline Milestone step: Adding parameter for milestone classification

      Adding a parameter for milestone classification, so an older build will not override a newer build if only they belong to the same group.

      The parameter would group different milestones. The ordinal parameter is used to order the milestones in the group. The abort behavior in a group is the same as now.
      extended simplified script from above:
      sleep PARAMETER
      milestone label: 'mileStone', ordinalGroup: projectName
      sleep 5

      Example1:
      project1: started with PARAMETER = 20 (sleeps 20 sec), ordinalGroup: 'p1'
      project2: started after project1 with PARAMETER = 5, ordinalGroup: 'p2'
      --> project1 and project2 run in the same job

      Example2:
      project1: started with PARAMETER = 20 (sleeps 20 sec), ordinalGroup: 'p1'
      project2: started after project1 with PARAMETER = 5, ordinalGroup: 'p2'
      project1: started after project2 with PARAMETER = 7 (sleeps 7 sec), ordinalGroup: 'p1'
      -> first p1 is aborted through 2nd p1 build, p2 runs through.

          [JENKINS-48510] Pipeline Milestone step: Adding parameter for milestone classification

          Jesse Glick added a comment -

          prevent running simultaneous deployments for the same app to the same environment

          This sort of logic should probably be moved out of Jenkins itself into a deployment tool.

          Jesse Glick added a comment - prevent running simultaneous deployments for the same app to the same environment This sort of logic should probably be moved out of Jenkins itself into a deployment tool.

          Markus Maga added a comment -

          jglick perhaps, suggestions welcome 

          But having milestones that can be bound to one or more parameters makes sense to me.

          But I don't see a reason not to offer this flexibility, it would give more super powers to parameterised builds

          Markus Maga added a comment - jglick  perhaps, suggestions welcome  But having milestones that can be bound to one or more parameters makes sense to me. But I don't see a reason not to offer this flexibility, it would give more super powers to parameterised builds

          To be more concrete, the best practice "one branch, one job" will not work in large companies with separated teams for SW development and infrastructure development due to job maintenance (E.g. are the latest released script versions used by the SW development teams). Here the build script version must be specified centrally. Please think about more than 1000 different SW development projects / repositories with at least 10 branches (note: multibranch is not possible, legacy SCM) but only 4 different workflows.

          It is a hard job to change some global configurations for more 10000 jobs  (even with Jobs DSL).

          To reduce the amount of jobs I want to use one job per released pipeline script version for all related projects / repositories (or 10000 jobs running in one Job). Unfortunately the milestone step cannot be used.

          flydiverny: Unfortunately not. I haven't found a real solution for this issue :/. E.g. the Job creation via Jobs DSL introduce additional infrastructure to handle the build parameters (some of them can be changed by the users and would be overwritten by the Jobs DSL)

          Tobias Peitzsch added a comment - To be more concrete, the best practice "one branch, one job" will not work in large companies with separated teams for SW development and infrastructure development due to job maintenance (E.g. are the latest released script versions used by the SW development teams). Here the build script version must be specified centrally. Please think about more than 1000 different SW development projects / repositories with at least 10 branches (note: multibranch is not possible, legacy SCM) but only 4 different workflows. It is a hard job to change some global configurations for more 10000 jobs  (even with Jobs DSL). To reduce the amount of jobs I want to use one job per released pipeline script version for all related projects / repositories (or 10000 jobs running in one Job). Unfortunately the milestone step cannot be used. flydiverny : Unfortunately not. I haven't found a real solution for this issue :/. E.g. the Job creation via Jobs DSL introduce additional infrastructure to handle the build parameters (some of them can be changed by the users and would be overwritten by the Jobs DSL)

          Jesse Glick added a comment -

          the latest released script versions [are] used by the SW development teams [so] the build script version must be specified centrally

          By the way this is easily supported in multibranch projects by moving the logic of the build into a library and then having a Jenkinsfile merely say something like

          library('standard-corporate-build@master').run()
          

          (The version identifier may be omitted if a default is specified in the library registration, and the administrator can control whether or not this version may be overridden, say to a tag or commit hash.)

          Jesse Glick added a comment - the latest released script versions [are] used by the SW development teams [so] the build script version must be specified centrally By the way this is easily supported in multibranch projects by moving the logic of the build into a library and then having a Jenkinsfile merely say something like library( 'standard-corporate-build@master' ).run() (The version identifier may be omitted if a default is specified in the library registration, and the administrator can control whether or not this version may be overridden, say to a tag or commit hash.)

          Timo Sand added a comment -

          Use case:
          We have a single job which does PR deployments. It is triggered by each PR's own build job, but with different params, like which branch to deploy etc.

          We would like to ensure that only the latest build gets deployed as it would be unnecessary to deploy older builds. Milestones and inverse locks seemed to offer a great solution to this, but milestones don't take parameters into account at all and thus it cancels jobs which have different parameters.

          Timo Sand added a comment - Use case: We have a single job which does PR deployments. It is triggered by each PR's own build job, but with different params, like which branch to deploy etc. We would like to ensure that only the latest build gets deployed as it would be unnecessary to deploy older builds. Milestones and inverse locks seemed to offer a great solution to this, but milestones don't take parameters into account at all and thus it cancels jobs which have different parameters.

          Henry Xu added a comment -

          I think this is an essential feature that needs to be implemented in milestone.

          Our pipeline is triggered by GitLab's Merge Request events. Whenever there are code updated or special comments, the job will be triggered. It is very common to have multiple builds that triggered by the same Merge Request in our pipeline.

          In this case, we would hope that old builds should be aborted if there are a new build of same Merge Request. If milestone can support classification, then we can simply write 

          milestone(groupLable: env.gitlab_merge_request_id)

          Henry Xu added a comment - I think this is an essential feature that needs to be implemented in milestone. Our pipeline is triggered by GitLab's Merge Request events. Whenever there are code updated or special comments, the job will be triggered. It is very common to have multiple builds that triggered by the same Merge Request in our pipeline. In this case, we would hope that old builds should be aborted if there are a new build of same Merge Request. If milestone can support classification, then we can simply write  milestone(groupLable: env.gitlab_merge_request_id)

          Jesse Glick added a comment -

          Our pipeline is triggered by GitLab's Merge Request events.

          Note that an alternate solution in this case is to use the GitLab branch source plugin, so that each MR corresponds to a distinct job. Obviously that is a big change with other impacts.

          Jesse Glick added a comment - Our pipeline is triggered by GitLab's Merge Request events. Note that an alternate solution in this case is to use the GitLab branch source plugin, so that each MR corresponds to a distinct job. Obviously that is a big change with other impacts.

          Henry Xu added a comment -

          jglick

          Actually I don't think the idea of one job per branch and the multi-branch pipeline is feasible for a large project. I think Jenkins 2.0 pipeline already grants us the ability to handle the multi-branch scenario and merge requests in a single pipeline script. Multi-branch pipeline may looks good if everything works ideally. But in the real world, we have already had to apply some global change to work around unexpected problems several times. And by using the one job per branch strategy, we still have to distinguish push events and merge events to do different things. Then why not just handle all of those conditions in a single pipeline script. It's hard for me to switch to multi-branch pipeline without any strong reason. 

          On the other hand, by providing a namespace mechanism for milestone, just as what we already had in `lock`, can be very useful and flexible in many cases.

          Henry Xu added a comment - jglick Actually I don't think the idea of one job per branch and the multi-branch pipeline is feasible for a large project. I think Jenkins 2.0 pipeline already grants us the ability to handle the multi-branch scenario and merge requests in a single pipeline script. Multi-branch pipeline may looks good if everything works ideally. But in the real world, we have already had to apply some global change to work around unexpected problems several times. And by using the one job per branch strategy, we still have to distinguish push events and merge events to do different things. Then why not just handle all of those conditions in a single pipeline script. It's hard for me to switch to multi-branch pipeline without any strong reason.  On the other hand, by providing a namespace mechanism for milestone, just as what we already had in `lock`, can be very useful and flexible in many cases.

          Stanislav Ovchar added a comment - Created PR  https://github.com/jenkinsci/pipeline-milestone-step-plugin/pull/17

          We're also currently running into a situation where we require some functionality similar the groups suggested here.

          We have a single job, triggered by various branches of a multibranch pipeline that fetches upstream artifacts, builds docker images from them and pushes them to an internal registry.

          To ensure that no outdated images are published on the registry we use a combination of the milestone and lock steps. The problem we're now facing is that (contrary to the lock step) the milestone step does not allow to identify milestones by means of a string label instead of an ordinal. If using a string label was possible we could dynamically create a label to disambiguate upstream branches. Using group labels - as suggested by this report - would also provide us with the means to solve the problem, but my personal opinion is that simply allowing for a textual milestone identifier would be a simpler, but equally as powerful solution.

          Kimon Hoffmann added a comment - We're also currently running into a situation where we require some functionality similar the groups suggested here. We have a single job, triggered by various branches of a multibranch pipeline that fetches upstream artifacts, builds docker images from them and pushes them to an internal registry. To ensure that no outdated images are published on the registry we use a combination of the milestone and lock steps. The problem we're now facing is that (contrary to the lock step) the milestone step does not allow to identify milestones by means of a string label instead of an ordinal. If using a string label was possible we could dynamically create a label to disambiguate upstream branches. Using group labels - as suggested by this report - would also provide us with the means to solve the problem, but my personal opinion is that simply allowing for a textual milestone identifier would be a simpler, but equally as powerful solution.

            amuniz Antonio Muñiz
            carlosrodlop Carlos Rodríguez López
            Votes:
            6 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: