-
New Feature
-
Resolution: Fixed
-
Minor
-
-
5.0.0-beta2
It would be great if the SCM would be queried for the reference build. If the builds are not in order (e.g. because the build are triggered by code review (e.g. Gerrit)), it doesn't make sense to use the last build as reference. The SCM (e.g. Git) or the Trigger (e.g. Gerrit) would know the previous commit which should be used as reference.
- depends on
-
JENKINS-21007 Add method to get parent revision
-
- Closed
-
-
-
- Resolved
-
- is duplicated by
-
-
- Resolved
-
-
JENKINS-48387 No job should be taken as a reference if it is triggered by Gerrit
-
- Closed
-
- is related to
-
-
- Open
-
- mentioned in
-
Page Loading...
[JENKINS-13056] Obtain reference build from SCM/Trigger
Roland Schulz
added a comment - I agree that the issue isn't specific to static analysis.
I think any SCM history which is not linear would benefit from it. The non linear history could be branches, changeset, etc. E.g. if versions B and C are compiled after each other, and they both have A as ancestor, then it is not correct if C is using B as reference build. E.g. if B has introduced a few warnings, than build C would report those warnings as fixed. But they are not fixed because these warning still exist in the separate branch/changeset.
If the threshold is set so that the count for B is above the limit, then relaying on the threshold works correctly. But in the general case it would not give the correct result. Also using the threshold makes it so that a build for version D with ancestor B doesn't report the warnings as fixed.
Also it seems (but I'm highly uncertain so please correct me if I'm wrong), that also unstable builds are considered for the reference build (and only failed ones are excluded). Thus it is not enough to set the status threshold (either new or total) for unstable but one has to set it for failed if one wants to affect the reference build. Also I assume that only the status threshold and not the health threshold effect the reference build.
Roland Schulz
added a comment - I agree that the issue isn't specific to static analysis.
I think any SCM history which is not linear would benefit from it. The non linear history could be branches, changeset, etc. E.g. if versions B and C are compiled after each other, and they both have A as ancestor, then it is not correct if C is using B as reference build. E.g. if B has introduced a few warnings, than build C would report those warnings as fixed. But they are not fixed because these warning still exist in the separate branch/changeset.
If the threshold is set so that the count for B is above the limit, then relaying on the threshold works correctly. But in the general case it would not give the correct result. Also using the threshold makes it so that a build for version D with ancestor B doesn't report the warnings as fixed.
Also it seems (but I'm highly uncertain so please correct me if I'm wrong), that also unstable builds are considered for the reference build (and only failed ones are excluded). Thus it is not enough to set the status threshold (either new or total) for unstable but one has to set it for failed if one wants to affect the reference build. Also I assume that only the status threshold and not the health threshold effect the reference build. Ok, I understand. I.e. we need some kind of ancestor of a build. This might be not trivial since not every SCM change will lead to a new build.
Also it seems (but I'm highly uncertain so please correct me if I'm wrong), that also unstable builds are considered for the reference build (and only failed ones are excluded). Thus it is not enough to set the status threshold (either new or total) for unstable but one has to set it for failed if one wants to affect the reference build. Also I assume that only the status threshold and not the health threshold effect the reference build.
Currently, a reference build is obtained from the last build that has a successful (not unstable) warnings result. The warnings result is independent from the overall build result.
Roland Schulz
added a comment - Ok, I understand. I.e. we need some kind of ancestor of a build. This might be not trivial since not every SCM change will lead to a new build.
Yes. It is true, in general not every SCM change has to lead to a build. But I think in many important cases it does. Thus supporting only this case would already be very nice. But I would imagine supporting the general case shouldn't be too hard either. If the SCM interface has an API to ask for the ancestor, the plugin could ask for the ancestor of the SCM version of the current build. If no build exist corresponding to that build, the same API could be used to recursively walk back in the history until a version is found which has a corresponding build.
A bit more tricky might be the case that more than one build exist for a certain SCM version. In that case one needs to decide which build should be picked, e.g. the most recent or the most recent successful one.
Roland Schulz
added a comment -
Ok, I understand. I.e. we need some kind of ancestor of a build. This might be not trivial since not every SCM change will lead to a new build.
Yes. It is true, in general not every SCM change has to lead to a build. But I think in many important cases it does. Thus supporting only this case would already be very nice. But I would imagine supporting the general case shouldn't be too hard either. If the SCM interface has an API to ask for the ancestor, the plugin could ask for the ancestor of the SCM version of the current build. If no build exist corresponding to that build, the same API could be used to recursively walk back in the history until a version is found which has a corresponding build.
A bit more tricky might be the case that more than one build exist for a certain SCM version. In that case one needs to decide which build should be picked, e.g. the most recent or the most recent successful one. Roland Schulz
added a comment - Is it possible to make any progress on this without the API change in the depending bug? E.g. by making it a git-only feature first and then adding other scm support when the API is added? Having this addressed would be very nice for us.
Roland Schulz
added a comment - Is it possible to make any progress on this without the API change in the depending bug? E.g. by making it a git-only feature first and then adding other scm support when the API is added? Having this addressed would be very nice for us. 
Sebastian Schuberth
added a comment - I'd also like to be able to specify an arbitrary build as the reference for computing the number of new warnings, basically for the same reason: We use a single Gerrit job which usually runs multiple builds for pending changes concurrently, and these builds do not need to finish in the order they were triggered. So computing new warnings wrt the absolute number of warnings in the last stable) build does not make sense, neither for analysis results nor test results etc.
However, I do not see what this has to do with querying the SCM. Instead, I'd propose to add a "build chooser" like in the "Copy artifacts from another project" build step provided by the Copy Artifacts Plugin. If a build referenced by this build chooser exposes warnings of the same type (e.g. FindBugs warnings), that is taken as a reference for computing the number of new warnings. In the Gerrit example, the reference build would be the latest stable post-merge build of our master branch.
Sebastian Schuberth
added a comment - I'd also like to be able to specify an arbitrary build as the reference for computing the number of new warnings, basically for the same reason: We use a single Gerrit job which usually runs multiple builds for pending changes concurrently, and these builds do not need to finish in the order they were triggered. So computing new warnings wrt the absolute number of warnings in the last stable) build does not make sense, neither for analysis results nor test results etc.
However, I do not see what this has to do with querying the SCM. Instead, I'd propose to add a "build chooser" like in the "Copy artifacts from another project" build step provided by the Copy Artifacts Plugin. If a build referenced by this build chooser exposes warnings of the same type (e.g. FindBugs warnings), that is taken as a reference for computing the number of new warnings. In the Gerrit example, the reference build would be the latest stable post-merge build of our master branch. Roland Schulz
added a comment - Now that the API for querying the parent in SCM is available is it possible to make progress on this issue? I have bought your Inca Trails game to show support 
.
Sebastian, your use case sounds different. For us the only logical way to get the reference build is to query the SCM (because of post submit code-review). You should open a separate issue.
Roland Schulz
added a comment - Now that the API for querying the parent in SCM is available is it possible to make progress on this issue? I have bought your Inca Trails game to show support .
Sebastian, your use case sounds different. For us the only logical way to get the reference build is to query the SCM (because of post submit code-review). You should open a separate issue. I don't see a chance to implement that in the near future. Interested in providing a pull request?
dirmeier
added a comment - Hi drulli, may you give me a hint how where to start? Maybe we could implement this because it is still a big problem for us - even worse since we use gerrit.
dirmeier
added a comment - Hi drulli , may you give me a hint how where to start? Maybe we could implement this because it is still a big problem for us - even worse since we use gerrit. Currently the computation of the reference build is hard-coded in BuildHistory (there are two options available). I think this class should be modified, so that the actual computation is delegated to a strategy. The two current implementations will then be the first two strategies. A SCM trigger could be the third one.
In the UI rather than two checkboxes the strategy should be selected.
Since we are currently developing a git-only solution for JENKINS-6748, this might be a good starting point.
Can someone please show in some more detail (example/sketch) which builds should be compared in the case of git?
dirmeier
added a comment - drulli I think it is quite simple. If there is a build for the parent of the current commit use it as reference. Otherwise proceed search with the parent commit (use the parent of the parent).
It the commit is a merge commit it has two parents. Maybe it could be configured which one should be use. But for our scenario it is always the first parent.
Example for merge commit:
You have a branch from master. When you merge back to master the new commit (merge) should have no more warning than the previous commit from master. This is the first parent of the merge commit.
Does this help?
dirmeier
added a comment - drulli I think it is quite simple. If there is a build for the parent of the current commit use it as reference. Otherwise proceed search with the parent commit (use the parent of the parent).
It the commit is a merge commit it has two parents. Maybe it could be configured which one should be use. But for our scenario it is always the first parent.
Example for merge commit:
You have a branch from master. When you merge back to master the new commit (merge) should have no more warning than the previous commit from master. This is the first parent of the merge commit.
Does this help? Sebastian Schuberth
added a comment - dirmeier I fear it's more complex. What if there are two feature branches A and B, A is based on master, but B is based on A? If the build for B finished before the build for A, your logic will take master as a reference for B, whereas it should probably be A.
Instead of doing something Git-specific at all, I'd prefer an implementation like mentioned in my previous comment that allows the user to select the reference build via a parameter. That would be way more flexible and allow e.g. to pass in information from Gerrit about the target branch to integrate to.
Sebastian Schuberth
added a comment - dirmeier I fear it's more complex. What if there are two feature branches A and B, A is based on master, but B is based on A? If the build for B finished before the build for A, your logic will take master as a reference for B, whereas it should probably be A.
Instead of doing something Git-specific at all, I'd prefer an implementation like mentioned in my previous comment that allows the user to select the reference build via a parameter. That would be way more flexible and allow e.g. to pass in information from Gerrit about the target branch to integrate to. Does this mean we need to have a user interaction? I would prefer if the plugin just would just pick the reference on its own.
Sebastian Schuberth
added a comment - I would not call it "interaction". It is an option to explicitly specify the reference build. IMO there simply is no reliable way to automatically determine the correct reference build in all cases.
Sebastian Schuberth
added a comment - I would not call it "interaction". It is an option to explicitly specify the reference build. IMO there simply is no reliable way to automatically determine the correct reference build in all cases. dirmeier
added a comment - sschuberth if B is build before A is finished there cannot be a correct reference build because A is not finished and hence cannot be used as reference. So best approach would be master or you need to throttle the second build. Or maybe it is possible to defere the warning analysis until the reference build is finished?
Of course it would be best if the reference build could be specified more detailed via parameters and I also think about an integration with gerrit. But for most cases in a git environment this should be the trick. Maybe as first approach or default behavior.
dirmeier
added a comment - sschuberth if B is build before A is finished there cannot be a correct reference build because A is not finished and hence cannot be used as reference. So best approach would be master or you need to throttle the second build. Or maybe it is possible to defere the warning analysis until the reference build is finished?
Of course it would be best if the reference build could be specified more detailed via parameters and I also think about an integration with gerrit. But for most cases in a git environment this should be the trick. Maybe as first approach or default behavior. 
mirza zeyrek
added a comment - I am having a similar issue. We are using two different jobs in Jenkins. One is for master branch and deployments. Other one is for PR's.
Since PR's could be cancelled or not merged instantly into the master, Unfortunately there is no useful case for counting new warnings from last successful PR or from previous PR.
Plug-in is taking last successful PR as reference. That's why if someone before you fixed unrelated files in PR; it is giving warnings with those files.
So I was wondering if an option like use last successful build from another Jenkins job would be possible or setting warning thresholds dynamically according to this ?
I really don't get why considering if this and that happens then implementation might not work. I am afraid current version is pretty much useless for any CI or CD integration if multiple developers are working different PR's involved simultaneously. So any implementation will be better than this even it's not 100% accurate 
Can someone point me the right direction for how may I edit / change plug-in in order to get reference build from another jenkins job.
mirza zeyrek
added a comment - I am having a similar issue. We are using two different jobs in Jenkins. One is for master branch and deployments. Other one is for PR's.
Since PR's could be cancelled or not merged instantly into the master, Unfortunately there is no useful case for counting new warnings from last successful PR or from previous PR.
Plug-in is taking last successful PR as reference. That's why if someone before you fixed unrelated files in PR; it is giving warnings with those files.
So I was wondering if an option like use last successful build from another Jenkins job would be possible or setting warning thresholds dynamically according to this ?
I really don't get why considering if this and that happens then implementation might not work. I am afraid current version is pretty much useless for any CI or CD integration if multiple developers are working different PR's involved simultaneously. So any implementation will be better than this even it's not 100% accurate
Can someone point me the right direction for how may I edit / change plug-in in order to get reference build from another jenkins job.
drulli sschuberth rschulz dirmeier I'm trying to consolidate the requirements for the static analysis suite in pipeline jobs in a wiki page. Can you please read it carefully and comment or change it accordingly?
mirzazeyrek It would be possible to also have an additional reference build computation that gets a Jenkins job name as property and returns the latest stable version (or whatever) from that build.
I see this issue is already 5 years old 
. Any advancement on the implementation design ?
I guess the easiest thing to do for the strategy is to have an additional mode: "For reference, take last stable build of Job:" for the plugin.
It increases the number of jobs, but is still flexible enough so that each use case above can be tackled.
In my case, i would have:
- job A gets triggered from Gerrit for pre-merge testing. It uses the configuration "Calculate new warnings" based on the strategy "For reference, take last stable build of Job B"
- job B would build on the same branch as job A, but only post-merge from the top of the branch (timer trigger).
By doing this, simultaneous job runs of A by different developers would not interfere between each others.
For other use cases, job B could either build from another branch or do some fancy stuff.
What do you think drulli mirzazeyrek dirmeier sschuberth rschulz ?
Actually nobody is working on this issue up to now. As I understood the discussion above it will be quite hard (or impossible) to provide a solution that works for all use cases.
Are you using a pipeline in your jobs? Maybe the planned pipeline features let you provide your own implementation as part of the pipeline script...
mirza zeyrek
added a comment - I ended up creating another job called warning-count-syncher. It contains a simple script based on casperjs.
After each successful commit to master, it gets the warning counts from latest successful build and sets it for PR job and master. I know it's a bit nasty but it does the job and works stable %99.9 of the cases.
You can take a look at it from here:
https://github.com/mirzazeyrek/jenkins-warning-count-synchronizer
mirza zeyrek
added a comment - chantivlad drulli
I ended up creating another job called warning-count-syncher. It contains a simple script based on casperjs.
After each successful commit to master, it gets the warning counts from latest successful build and sets it for PR job and master. I know it's a bit nasty but it does the job and works stable %99.9 of the cases.
You can take a look at it from here:
https://github.com/mirzazeyrek/jenkins-warning-count-synchronizer 
Michał Rapacz
added a comment - Any updates on this? It would be nice to get at least a feature mentioned by chantivlad:
"For reference, take last stable build of Job: " .
This doesn't sound like a big change, but it looks it would solve the problem for many users.
Michał Rapacz
added a comment - Any updates on this? It would be nice to get at least a feature mentioned by chantivlad :
"For reference, take last stable build of Job: " .
This doesn't sound like a big change, but it looks it would solve the problem for many users. 
Mark DeMichele
added a comment - I totally agree with the previous suggestion. Having a feature that allows you to reference another Jenkins job to compare against is all that's really needed to address the issue that usually happens when you have a Pull Request build.
Mark DeMichele
added a comment - I totally agree with the previous suggestion. Having a feature that allows you to reference another Jenkins job to compare against is all that's really needed to address the issue that usually happens when you have a Pull Request build. Ok, I will add this kind of reference selection to the new pipeline API. Is the reference 'job' sufficient? Or should also the build number of the other job be a variable?
Latest stable build from another job would be ok for now
But It would be nice to have something compatible with multi branch pipeline; being able to choose from:
1- (different or same) multi branch pipeline job
2- choosing branch from it and getting latest stable build as reference.
mirza zeyrek
added a comment - - edited Latest stable build from another job would be ok for now
But It would be nice to have something compatible with multi branch pipeline; being able to choose from:
1- (different or same) multi branch pipeline job
2- choosing branch from it and getting latest stable build as reference.
drulli Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/io/jenkins/plugins/analysis/core/JenkinsFacade.java
src/main/java/io/jenkins/plugins/analysis/core/graphs/AnnotationsByUserGraph.java
src/main/java/io/jenkins/plugins/analysis/core/history/BuildHistory.java
src/main/java/io/jenkins/plugins/analysis/core/history/OtherJobReferenceFinder.java
src/main/java/io/jenkins/plugins/analysis/core/history/ReferenceFinder.java
src/main/java/io/jenkins/plugins/analysis/core/history/ReferenceProvider.java
src/main/java/io/jenkins/plugins/analysis/core/history/ResultHistory.java
src/main/java/io/jenkins/plugins/analysis/core/model/AnalysisResult.java
src/main/java/io/jenkins/plugins/analysis/core/model/StaticAnalysisLabelProvider.java
src/main/java/io/jenkins/plugins/analysis/core/model/Summary.java
src/main/java/io/jenkins/plugins/analysis/core/steps/AnalysisExecution.java
src/main/java/io/jenkins/plugins/analysis/core/steps/PublishIssuesStep.java
src/main/java/io/jenkins/plugins/analysis/core/steps/ScanForIssuesStep.java
src/main/resources/io/jenkins/plugins/analysis/core/model/Messages.properties
src/main/resources/io/jenkins/plugins/analysis/core/views/InfoErrorDetail/index.jelly
src/main/resources/io/jenkins/plugins/analysis/core/views/IssuesDetail/index.jelly
src/test/java/io/jenkins/plugins/analysis/core/history/OtherJobReferenceFinderTest.java
src/test/java/io/jenkins/plugins/analysis/core/model/SummaryTest.java
http://jenkins-ci.org/commit/analysis-core-plugin/3429aead0352f32c2b2c2114bd868e2a8a110dcf
Log:
JENKINS-13056JENKINS-31812 Added option to specify a reference job.
The reference results will now be obtained either
a) from the current job
b) from a specified reference job (given by the name)
The reference job will be selected using the properties:
- ignoreAnalysisResult: ignores the result of the previous analysis run
- overallResultMustBeSuccess: picks only builds with overall SUCCESS
Code changed in jenkins
User: Ulli Hafner
Path:
src/test/java/io/jenkins/plugins/analysis/warnings/PipelineITest.java
src/test/java/io/jenkins/plugins/analysis/warnings/StepsITest.java
src/test/resources/io/jenkins/plugins/analysis/warnings/java-start.txt
http://jenkins-ci.org/commit/warnings-plugin/3c64ac6c68b433e3f3b4e960e02a99c414223b31
Log:
JENKINS-13056JENKINS-31812 Added option to specify a reference job.
The reference results will now be obtained either
a) from the current job
b) from a specified reference job (given by the name)
The reference job will be selected using the properties:
- ignoreAnalysisResult: ignores the result of the previous analysis run
- overallResultMustBeSuccess: picks only builds with overall SUCCESS
I think that feature is something that should be first added to core (SCM) as an extension point. E.g. in general the build stable results (test results, analysis results, etc.) should use that extension point to determine a reference build. Or is this only meaningful for static analysis?
When this extension point is provided, the static analysis plug-in can use that extension point to provide that behavior.
Can you please add some build sequences or examples to describe in more detail your requirements? Currently, in the warnings plug-in a reference build is the last build with results below a threshold. Should this also be done in your use case?