-
Epic
-
Resolution: Unresolved
-
Major
-
Pipeline support for Promoted Builds Plugin
-
Powered by SuggestiMate
This enhancement request is for the Promoted Builds plugin to support Jenkins' 2.X style Pipelines.
Pipelines are the new paradigm that Jenkins 2.X is promoting heavily – with its ability to define the build process in code, automatic build discovery, and great integration with repository systems for automatic build / testing of pull requests.
There are many places within the new Pipeline style where the Promoted Plugin could be used to great effect:
- builds could dynamically https://issues.jenkins-ci.org/browse/JENKINS-36089#promote and save artifacts, pushing them into other pipeline jobs
- Users could manually promote pipeline builds that have been through some kind of QA process
- Permalinks provided by the Promoted Builds plugin could be used in pipeline builds
Basically, the same vast number of use cases that the Promoted Build Plugin provides today could be taken advantage of by builds defined via the Pipeline API and Jenkinsfile build definitions.
Reference here for the API changes that are required for a plugin to be available in a Pipeline build:
https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md
Original reference bug:
https://issues.jenkins-ci.org/browse/JENKINS-35376
- relates to
-
JENKINS-57634 GSoC '19: artifact promotion in Jenkins' pipeline. (Coding Phase-I)
-
- Resolved
-
[JENKINS-36089] Pipeline support for Promoted Builds Plugin
I am waiting on this feature, which will give me something close to promotion, using pipeline input:
Yes Please. It's very hard to have build pipelines without some amount of manual promotion and testing before the next step.
This support would be very useful. Currently this is our major blocking point for moving into pipelines.
jviolas As a alternative you could try to add hyperlink in build description(with GET parameters) to some promotion job under certain statement that is needed for promotion.
winotu: Thanks for the tip. I'll try that.
Right now we didn't have to use promotion since we migrate to pipelines, but the time will come. If the plugin has not been updated then, I think we will create some jobs dedicated to promotion (publish artifacts and other specific actions) that will get all necessary artifacts from the main building jobs.
-1 on this. The design of Promoted Builds is bad. There could be a separate plugin offering comparable functionality but done the right way.
Can you expand on why the design of promoted builds is bad?
I am using multibranch pipelines, and would like to be able to have automated deployments through different environments, but only with human approval. Is there a recommended approach for achieving this? I confess I did not understand the suggestion above of adding a hyperlink.
I just wanted to explain my approach for promoting pipeline builds, maybe it helps others.
I have several components and each has a separate build job. All jobs are orchestrated by a build chain job which triggers the other jobs, runs integration tests, etc.
After the build chain completed successfully, I use GroovyPostbuildAction.createBadge() to add a badge to the builds that were used during the integration tests.
Then I created a promotion job, in which I can select the component and the build number to promote. It then checks if that specific build contains the badge that was previously added (i.e. all tests were successful) and does some action (deploy on production server, upload artifacts, ...).
In the promotion job, I also included mechanisms to implement multi-level promotion by using different badges, so that, for example, a release first has to be deployed on the testing servers before it may be deployed to production systems.
I am using multibranch pipelines, and would like to be able to have automated deployments through different environments, but only with human approval. Is there a recommended approach for achieving this?
input step generally, assuming only the last build to pass tests is eligible for approval. Otherwise there is no equivalent currently. No plans to use the Promoted Builds plugin as is; needs to be redesigned from scratch.
In my case, I have some jobs building/testing various applications for different platform. I created one job specific for promotion. This job has parameters that let user choose the build for each job he wants to promote. The job then copy the artifacts of each build, do some checking to be sure the selected build is a valid one to be promoted and then publish the artifacts. This jobs also archive the artifacts as its own, so I can find promoted artifacts easily.
The only thing I lost comparing to before with the Promoted builds Plugin is the badge in the description of the promoted build that helps finding the last promoted build, but now I can find then in my promotion job, in which I have only the promoted artifacts.
I still disagree with jglick regarding the Promoted Builds feasibility for Pipeline. Honestly conditional promotion of releases is a PITA in Pipeline now, but I agree that the Promoted builds plugin ideally needs some redesign before adding the Pipeline support.
My vision:
- Get rid of the built-in promotion job, use standalone one (of any time, not just Freestyle-alike)
- In standalone jobs introduce the "Environment" entity to indicate where the build results are being deployed
- Get rid of the access to workspaces, allow promoting only artifacts (with some flexibility via build steps, of-course)
- Use Fingerprinting to trace artifacts between original and promotion jobs
- Retain the promotion functionality for cross-referencing builds and for triggering promotions from jobs
- Actually retain almost all existing UIs...
That's how I see it, the implementation would be more or less close to what is implemented in AppVeyor.
Bad news is that I won't have personal time to work on it in middle-term. Any contributions will be appreciated
Get rid of the built-in promotion job, use standalone one
Yes this is the key. The number of the upstream build being “promoted” should simply be passed as a build parameter to the downstream build. Of course you can do this today without the UI.
I do not think other concepts of the existing plugin are particularly problematic (though, as mentioned, certain common use cases are done more simply in a single Pipeline job).
Today, I am using "Keep this build forever" as a "poor man's" promotion button.
I have a pipeline script that runs every 15 minutes, and searches for builds that have been marked "Keep Forever" and then passes their information / build number to the next build in the promotion.
Its not as flexible as the Promoted Builds plugin was originally, but its working well.
My only wish is that I could trigger that build off new builds being marked "Keep Forever" instead of requiring that it run polling every 15 minutes.
This is just FYI – as I thought it might be interesting to someone else who finds this jira issue.
To give a use case, we have common ci builds for the following:
networking ci build (governs acl rules)
load balancing ci build (governs load balancer config)
platform image ci build (governs centos/ windows images)
developer specific ci build (code compilation).
Each of these ci builds have an associated promoted build parameter which when they are successful increments without teams having to manually update dependencies when a check-ins occur to networking, platform or load balancing versions.
These ci builds then roll up into a "package build" which generates the build artifact used at deployment time (we build a manifest json file using the artifacts linked to the latest promoted build parameter versions). This is generated every developer check-in and deployment pipelines are then invoked by the new manifest release candidate.
This manifest json then consumes the latest networking, load balancing, platform image using promoted build parameters to pull all the necessary artifacts from each ci build and arrange them as the 1st deployment step in the deployment pipeline.
Currently not having the promoted build parameter plugin integrated with the pipeline job type means we cant currently make the package build, currently a freestyle job, the 1st step of the jenkinsfile pipeline job and then trigger other jenkinsfile pipeline jobs from it using promotion stars, as it traverses environments.
With the promoted build parameter missing from the pipeline job we are missing the ability to:
1) Associate the promoted build parameters with the pipeline job type as part of the jenkinsfile meaning the pipeline jenkinsfile cant be stored in the code repository and used as the trigger event for the quality assurance pipeline.
2) Trigger a pipeline based on a state change to an artifact repository or another build. Currently we use thoughtworks go for pipelines, which allows us to push the manifest file to artifactory and then poll the repository for changes. The urltrigger plugin could provide this feature as well but it also hasnt been converted to the pipeline job type either meaning we currently have no way to trigger pipelines based on an artifact change. An updated promoted build number could be used to act as that trigger rather than the external artifact.
3) We will want to have a jenkinsfile for each pipeline stage so qa, integration, perf and prod by having association between multiple jenkinsfiles is necessary, the promoted build plugin could potentially do this link, so each pipeline promotes a package (promoted build number) that is allowed to be deployed to the next environment by users. This is particularly useful in continuous delivery when in production environments teams can select a promoted build number to deploy the latest build or trigger a previous promoted build number to roll back by selecting it from the dropdown list.
The above points are currently blocking us from moving to jenkins 2.x for pipelines from thoughtworks go and probably preventing many others too.
My use case is to flag a lot of plugins (about 50) before creating my application intranet. It's used to indicates a "developper release" which will be deployed on internal intranet, to have better user returns.
This is before QA steps and I can't use input step to accept every plugins builds.
My "poor man" solution will be to add some text in build description and parse it via REST.
Support of so-called "multi-day builds" is supposed to be a big selling point of Pipeline, so it seems strange that promotions are not a first-class citizen. Using an input step is definitely sub-optimal as this occupies the executor, requiring guards such as locks, milestones, timeouts so it doesn't use the executor for too long. All this hassle when we just need to set a 'promoted' flag on the job.
this occupies the executor
Only if you put input inside node, which generally you should not. An input paused at top level can sit there for days without occupying any resources on Jenkins (not even a native Thread).
OK, but I put most of my build inside node declarations. So if input can't run in node, I have to exclude that one step.
I would like to wrap the input in something like this, but it doesn't seem valid... is there another way to express?
{code:groovy}
stage('deploy') {
agent any
// some build seps
agent none {
// input...
}
}
{code}
Well you are using Declarative Pipeline. That is another RFE entirely.
This is the one thing still holding a majority of our devs from moving from Bamboo
Has anyone found an alternate way of handing deployment without using the promoted builds?
Same thing with Bamboo here, which holds us from moving most of projects to Jenkins.
We need something like Bamboo Deployment projects in Jenkins:
https://confluence.atlassian.com/bamboo/deployment-projects-338363438.html
We have used Promotions plug-in for years with great success. Our company is now moving towards pipeline builds which is fine. The pipeline will build and archive our artifacts but we have never “auto” deployed as it is not valuable. However, we constantly manually deploy to all environments.
Adding the Promotions capability in a pipeline build would be huge. Thank you for your support!
Recently we are trying Jenkins 2.X with declarative pipeline. If Promotions plug-in can support it, that would be great!
Only if you put input inside node, which generally you should not. An input paused at top level can sit there for days without occupying any resources on Jenkins (not even a native Thread).
The workspace of next stage will be reallocated if I put directive input into a standalone stage and set up agent as none, but my next stage need use the same workspace with the previous stage. this is my difficulty, otherwise, I need set up a pipeline global agent, then directive input will occupy executor.
Most of the Jenkins users (and users of another platforms like gitlab) are using the pipeline way because its simplicity and flexibility. If this plugin does not support pipelines, it becomes useless. Thanks for your understanding and support.
I totally agree with you, but I am not ready to implement it during my spare time anytime soon. It is a big chunk of work requiring several weekends to be done properly. I cannot dedicate time due to the personal stuff and other commitments in the Jenkins project. I'd guess the same applies to dnusbaum who is a new maintainer.
I have provided my vision here: https://issues.jenkins-ci.org/browse/JENKINS-36089?focusedCommentId=280959&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-280959 . If somebody is interested, please feel free to contribute. It is not that difficult to implement this functionality even for those ones who are not experts in Jenkins development, just requires time and some basic Java knowledge. If you do not have personal time, ask your employers to dedicate some time. It may be possible in many cases. This is how open-source works sometimes, fortunately or not
But, again, I am happy to provide reviews and help with delivering changes if somebody proposes a pull request
Hello, I I have added this project to the list of Google Summer of Code 2019 project ideas: https://docs.google.com/document/d/1UYi0jIYsKHE5IGS84B5W0XBoeMyF4yY_exu-21O99U8/edit?usp=sharing , so it may be implemented by one of the students this year.
We are also looking for mentors who would be interested in this project. If you would like to join the project, please let me know
I am not sure if this is the feature I am after but essentially, it is a cross between CopyArtifact and what this might offer.
Currently, I am publishing in htmlreports for a non-pipeline job and I want to promote that to the parent DSL pipeline.
I tried CopyArtifact but it cannot see the htmlreports directory (i.e. linked from lastSuccessfulBuild).
Perhaps by forcing a shared WORKSPACE is the simplest solution but generally, I wanted it to work without changing the legacy build jobs and cannot rely on the same WORKSPACE being used.
Any tips for an easily maintained solution welcome.
I am marking it as "In progress", because I am working on some bits as my Christmas hack
Any updates?We have freestyle jobs which use this promotion criterias due to which we are not able to convert these jobs to pipeline.Any idea by when we can expect this feature in pipeline.?
issac1993 nothing shippable unfortunately, no ETA. With COVID-19 I was unable to do real development for a while, and I have a huge queue of items to catch up. I do not expect to get to it this summer, but other contributors might contribute. Any patches are welcome
Hi Guys..
How I can use this plugin with Multibranch pipeline?
Multibranch pipeline uses Jenkinsfile and I am not sure how can I integrate this plugin to promote?
Any help is appreciated. Thanks.
amitrl you can't until this epic is complete. This would be very helpful for my team so I'm thinking of taking a stab at this. Do you think I could get your help/guidance oleg_nenashev?
silentsnowman with my current assignments at work and personal situation, it is unlikely that I will be able to seriously work on that anytime soon. I might find some time to answer questions if someone wants to implement it
In the meantime, is there any workaround or best practice to have manual promotion on build with multibranch pipeline job ?