-
Bug
-
Resolution: Fixed
-
Major
-
Ubuntu 14.04.5 LTS (jenkins installed via debian)
Jenkins 2.18
Git plugin 3.0.0
Pipeline 2.4
Pipeline SCM Step 2.2
Pipeline Stage Step 2.2
I work in bitbucket which uses a hook to notify our jenkins server on commit (or if manually triggered in a pull request). I've noticed that, when I make a new pipeline job from SCM (git), onNotifyCommit won't look at the new job until after I've manually run the job and it has completed.
A bit more details on the job:
- I did specify an empty poll interval, which allowed my other maven jobs to be properly notified.
- I haven't specified any additional behaviors
Since the job has the SCM information built directly into the pipeline job, shouldn't onNotifyCommit properly find and notify the new job without needing execution?
[JENKINS-38669] SCM (Git) Will Not Notify Pipeline Before Execution
Angry Gami
added a comment - Hi I have same problem as dgomez
It is ridiculous how this bug still exists. I've checked Jenkins version 2.205 and Git plugin version 4.0.0.
Here is why:
package org.jenkinsci.plugins.workflow.job; ... public final class WorkflowJob extends Job<WorkflowJob,WorkflowRun> implements LazyBuildMixIn.LazyLoadingJob<WorkflowJob,WorkflowRun>, ParameterizedJobMixIn.ParameterizedJob<WorkflowJob, WorkflowRun>, TopLevelItem, Queue.FlyweightTask, SCMTriggerItem, BlockableResume { ... @Override public Collection<? extends SCM> getSCMs() { WorkflowRun b = getLastSuccessfulBuild(); if (b == null) { b = getLastCompletedBuild(); } if (b == null) { return Collections.emptySet(); } Map<String,SCM> scms = new LinkedHashMap<>(); for (WorkflowRun.SCMCheckout co : b.checkouts(null)) { scms.put(co.scm.getKey(), co.scm); } return scms.values(); }
Above is method getSCM from WorkflowJob class and as you can see that it will return empty collection when there is no successful or completed build for this workflow.
And here is code from GitStatus class that is responsible for processing notifications:
package hudson.plugins.git; ... public class GitStatus implements UnprotectedRootAction { ... public static class JenkinsAbstractProjectListener extends Listener { ... @Override public List<ResponseContributor> onNotifyCommit(String origin, URIish uri, String sha1, List<ParameterValue> buildParameters, String... branches) { ... for (final Item project : jenkins.getAllItems()) { SCMTriggerItem scmTriggerItem = SCMTriggerItem.SCMTriggerItems.asSCMTriggerItem(project); if (scmTriggerItem == null) { continue; } SCMS: for (SCM scm : scmTriggerItem.getSCMs()) { // do stuff ...
I.e. if getSCMs returns empty collection (and it will if no builds had happened yet) nothing will be checked.
Maybe better way would be to first check if definition of the scmTriggerItem has scm attached with something like this:
...
for (final Item project : jenkins.getAllItems()) {
SCMTriggerItem scmTriggerItem = SCMTriggerItem.SCMTriggerItems.asSCMTriggerItem(project);
if (scmTriggerItem == null) {
continue;
}
Collection<? extends SCM> scms = scmTriggerItem.getSCMs();
if (scms.isEmpty()){
if (scmTriggerItem.getDefinition() instanceof org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition) {
CpsScmFlowDefinition fd = (CpsScmFlowDefinition) scmTriggerItem.getDefinition();
scms = Arrays.asList(fd.getScm());
}
}
SCMS: for (SCM scm : scms) {
// do stuff
...
It is hard to tell if `org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition` will be available so maybe only real way to do that will be to do some reflection. Though main idea it NOT to rely on build happened at least once for receiving scms.
Angry Gami
added a comment - Hi I have same problem as dgomez
It is ridiculous how this bug still exists. I've checked Jenkins version 2.205 and Git plugin version 4.0.0.
Here is why:
package org.jenkinsci.plugins.workflow.job;
...
public final class WorkflowJob extends Job<WorkflowJob,WorkflowRun> implements LazyBuildMixIn.LazyLoadingJob<WorkflowJob,WorkflowRun>, ParameterizedJobMixIn.ParameterizedJob<WorkflowJob, WorkflowRun>, TopLevelItem, Queue.FlyweightTask, SCMTriggerItem, BlockableResume {
...
@Override public Collection<? extends SCM> getSCMs() {
WorkflowRun b = getLastSuccessfulBuild();
if (b == null ) {
b = getLastCompletedBuild();
}
if (b == null ) {
return Collections.emptySet();
}
Map< String ,SCM> scms = new LinkedHashMap<>();
for (WorkflowRun.SCMCheckout co : b.checkouts( null )) {
scms.put(co.scm.getKey(), co.scm);
}
return scms.values();
}
Above is method getSCM from WorkflowJob class and as you can see that it will return empty collection when there is no successful or completed build for this workflow.
And here is code from GitStatus class that is responsible for processing notifications:
package hudson.plugins.git;
...
public class GitStatus implements UnprotectedRootAction {
...
public static class JenkinsAbstractProjectListener extends Listener {
...
@Override
public List<ResponseContributor> onNotifyCommit( String origin, URIish uri, String sha1, List<ParameterValue> buildParameters, String ... branches) {
...
for ( final Item project : jenkins.getAllItems()) {
SCMTriggerItem scmTriggerItem = SCMTriggerItem.SCMTriggerItems.asSCMTriggerItem(project);
if (scmTriggerItem == null ) {
continue ;
}
SCMS: for (SCM scm : scmTriggerItem.getSCMs()) {
// do stuff
...
I.e. if getSCMs returns empty collection (and it will if no builds had happened yet) nothing will be checked.
Maybe better way would be to first check if definition of the scmTriggerItem has scm attached with something like this:
...
for ( final Item project : jenkins.getAllItems()) {
SCMTriggerItem scmTriggerItem = SCMTriggerItem.SCMTriggerItems.asSCMTriggerItem(project);
if (scmTriggerItem == null ) {
continue ;
}
Collection<? extends SCM> scms = scmTriggerItem.getSCMs();
if (scms.isEmpty()){
if (scmTriggerItem.getDefinition() instanceof org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition) {
CpsScmFlowDefinition fd = (CpsScmFlowDefinition) scmTriggerItem.getDefinition();
scms = Arrays.asList(fd.getScm());
}
}
SCMS: for (SCM scm : scms) {
// do stuff
...
It is hard to tell if `org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition` will be available so maybe only real way to do that will be to do some reflection. Though main idea it NOT to rely on build happened at least once for receiving scms. Thanks for the thorough investigation angrygami! Will you be submitting a pull request to resolve the issue, including tests that show the problem and can be used to confirm that the fix resolves the problem?
Angry Gami
added a comment - I don't have a fix yet. Suggested solution above won't compile because org.jenkinsci.plugins.workflow package is not a dependency of plugin project. I'll try to find working one.
Angry Gami
added a comment - I don't have a fix yet. Suggested solution above won't compile because org.jenkinsci.plugins.workflow package is not a dependency of plugin project. I'll try to find working one.
Ok, sorry for a delay. Now I have my fix working and it involves changes to 3 plugins:
workflow-api
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java b/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java index 55c29c7..ef71822 100644 --- a/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java +++ b/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java @@ -29,12 +29,15 @@ import hudson.Util; import hudson.model.AbstractDescribableImpl; import hudson.model.Action; import hudson.model.TaskListener; +import hudson.scm.SCM; import hudson.util.LogTaskListener; import javax.annotation.CheckForNull; import javax.annotation.Nonnull; import java.io.IOException; import java.util.List; +import java.util.Collection; +import java.util.Collections; import java.util.logging.Level; import java.util.logging.Logger; @@ -80,4 +83,7 @@ public abstract class FlowDefinition extends AbstractDescribableImpl<FlowDefinit return (FlowDefinitionDescriptor) super.getDescriptor(); } + public Collection<? extends SCM> getSCMs() { + return Collections.emptyList(); + } }
Here I've added new method to FlowDefinition class that could be overridden by subclasses and by default it returns empty collection. This is meant to help subclasses of FlowDefinition to provide list of SCMs that it might be aware of.
workflow-cps
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java b/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java index 6e665a8..6fcd729 100644 --- a/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java +++ b/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java @@ -46,6 +46,7 @@ import java.io.FileNotFoundException; import java.io.IOException; import java.io.InterruptedIOException; import java.util.Collection; +import java.util.Collections; import java.util.List; import jenkins.model.Jenkins; import jenkins.scm.api.SCMFileSystem; @@ -82,6 +83,11 @@ public class CpsScmFlowDefinition extends FlowDefinition { return scm; } + @Override + public Collection<? extends SCM> getSCMs() { + return Collections.singletonList(scm); + } + public String getScriptPath() { return scriptPath; }
Here I actually override getSCMs method from FlowDefinition for case when we are dealing with CpsScmFlowDefinition - i.e. definition of the flow that actually known to be based on SCM.
For definitions that are based of inline script we can't tell if there are any SCM used until first build and this behavior is preserved.
workflow-job
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java b/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java index 57ea992..3d0c84d 100644 --- a/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java +++ b/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java @@ -526,17 +526,23 @@ public final class WorkflowJob extends Job<WorkflowJob,WorkflowRun> implements L } @Override public Collection<? extends SCM> getSCMs() { + Collection<? extends SCM> definedSCMs = definition != null + ? definition.getSCMs() + : Collections.emptySet(); WorkflowRun b = getLastSuccessfulBuild(); if (b == null) { b = getLastCompletedBuild(); } if (b == null) { - return Collections.emptySet(); + return definedSCMs; } Map<String,SCM> scms = new LinkedHashMap<>(); for (WorkflowRun.SCMCheckout co : b.checkouts(null)) { scms.put(co.scm.getKey(), co.scm); } + for (SCM scm : definedSCMs) { + scms.put(scm.getKey(), scm); + } return scms.values(); }
And finally here I use getSCMs method from FlowDefinition to return known SCMs when there were no build happened yet.
I can try to make pull requests from all of the above, though I don't know how to do that for three projects simultaneously and consistently. Maybe markewaite can help me with that somehow?
I also don't understand how to write a test that cover three plugins at same time.
Angry Gami
added a comment - - edited Ok, sorry for a delay. Now I have my fix working and it involves changes to 3 plugins:
workflow-api
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java b/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java
index 55c29c7..ef71822 100644
--- a/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java
+++ b/src/main/java/org/jenkinsci/plugins/workflow/flow/FlowDefinition.java
@@ -29,12 +29,15 @@ import hudson.Util;
import hudson.model.AbstractDescribableImpl;
import hudson.model.Action;
import hudson.model.TaskListener;
+ import hudson.scm.SCM;
import hudson.util.LogTaskListener;
import javax.annotation.CheckForNull;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.List;
+ import java.util.Collection;
+ import java.util.Collections;
import java.util.logging.Level;
import java.util.logging.Logger;
@@ -80,4 +83,7 @@ public abstract class FlowDefinition extends AbstractDescribableImpl<FlowDefinit
return (FlowDefinitionDescriptor) super .getDescriptor();
}
+ public Collection<? extends SCM> getSCMs() {
+ return Collections.emptyList();
+ }
}
Here I've added new method to FlowDefinition class that could be overridden by subclasses and by default it returns empty collection. This is meant to help subclasses of FlowDefinition to provide list of SCMs that it might be aware of.
workflow-cps
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java b/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java
index 6e665a8..6fcd729 100644
--- a/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java
+++ b/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java
@@ -46,6 +46,7 @@ import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InterruptedIOException;
import java.util.Collection;
+ import java.util.Collections;
import java.util.List;
import jenkins.model.Jenkins;
import jenkins.scm.api.SCMFileSystem;
@@ -82,6 +83,11 @@ public class CpsScmFlowDefinition extends FlowDefinition {
return scm;
}
+ @Override
+ public Collection<? extends SCM> getSCMs() {
+ return Collections.singletonList(scm);
+ }
+
public String getScriptPath() {
return scriptPath;
}
Here I actually override getSCMs method from FlowDefinition for case when we are dealing with CpsScmFlowDefinition - i.e. definition of the flow that actually known to be based on SCM.
For definitions that are based of inline script we can't tell if there are any SCM used until first build and this behavior is preserved.
workflow-job
diff --git a/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java b/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
index 57ea992..3d0c84d 100644
--- a/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
+++ b/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
@@ -526,17 +526,23 @@ public final class WorkflowJob extends Job<WorkflowJob,WorkflowRun> implements L
}
@Override public Collection<? extends SCM> getSCMs() {
+ Collection<? extends SCM> definedSCMs = definition != null
+ ? definition.getSCMs()
+ : Collections.emptySet();
WorkflowRun b = getLastSuccessfulBuild();
if (b == null ) {
b = getLastCompletedBuild();
}
if (b == null ) {
- return Collections.emptySet();
+ return definedSCMs;
}
Map< String ,SCM> scms = new LinkedHashMap<>();
for (WorkflowRun.SCMCheckout co : b.checkouts( null )) {
scms.put(co.scm.getKey(), co.scm);
}
+ for (SCM scm : definedSCMs) {
+ scms.put(scm.getKey(), scm);
+ }
return scms.values();
}
And finally here I use getSCMs method from FlowDefinition to return known SCMs when there were no build happened yet.
I can try to make pull requests from all of the above, though I don't know how to do that for three projects simultaneously and consistently. Maybe markewaite can help me with that somehow?
I also don't understand how to write a test that cover three plugins at same time. Angry Gami
added a comment - I wrote a test for workflow-job project that shows the case and fix for it:
pom.xml
diff --git a/pom.xml b/pom.xml
index b9b8e64..f255e37 100644
--- a/pom.xml
+++ b/pom.xml
@@ -93,13 +93,13 @@
<!-- Satisfy upper bound dependencies -->
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>script-security</artifactId>
- <version>1.62</version>
+ <version>1.63</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-cps</artifactId>
- <version>2.74</version>
+ <version>2.76</version>
<scope>test</scope>
</dependency>
<dependency>
WorkflowJobTest.java
diff --git a/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java b/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java index 53b27de..b32130f 100644 --- a/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java +++ b/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java @@ -69,6 +69,15 @@ public class WorkflowJobTest { j.assertLogContains("second version", b2); } + @Issue("JENKINS-38669") + @Test public void nonEmptySCMListForGitSCMJobBeforeBuild() throws Exception { + WorkflowJob p = j.jenkins.createProject(WorkflowJob.class, "p"); + CpsScmFlowDefinition def = new CpsScmFlowDefinition(new GitSCM("I don't care"), "Jenkinsfile"); + assertEquals("Expecting one SCM for definition", 1, def.getSCMs().size()); + p.setDefinition(def); + assertEquals("Expecting one SCM", 1, p.getSCMs().size()); + } + @Test public void addAction() throws Exception { WorkflowJob p = j.jenkins.createProject(WorkflowJob.class, "p");
Though I'm not sure this is good for master branch, I was working from repo tag 'workflow-job-2.36'.
Angry Gami
added a comment - I wrote a test for workflow-job project that shows the case and fix for it:
pom.xml
diff --git a/pom.xml b/pom.xml
index b9b8e64..f255e37 100644
--- a/pom.xml
+++ b/pom.xml
@@ -93,13 +93,13 @@
<!-- Satisfy upper bound dependencies -->
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>script-security</artifactId>
- <version>1.62</version>
+ <version>1.63</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-cps</artifactId>
- <version>2.74</version>
+ <version>2.76</version>
<scope>test</scope>
</dependency>
<dependency>
WorkflowJobTest.java
diff --git a/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java b/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java
index 53b27de..b32130f 100644
--- a/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java
+++ b/src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowJobTest.java
@@ -69,6 +69,15 @@ public class WorkflowJobTest {
j.assertLogContains( "second version" , b2);
}
+ @Issue( "JENKINS-38669" )
+ @Test public void nonEmptySCMListForGitSCMJobBeforeBuild() throws Exception {
+ WorkflowJob p = j.jenkins.createProject(WorkflowJob.class, "p" );
+ CpsScmFlowDefinition def = new CpsScmFlowDefinition( new GitSCM( "I don't care" ), "Jenkinsfile" );
+ assertEquals( "Expecting one SCM for definition" , 1, def.getSCMs().size());
+ p.setDefinition(def);
+ assertEquals( "Expecting one SCM" , 1, p.getSCMs().size());
+ }
+
@Test
public void addAction() throws Exception {
WorkflowJob p = j.jenkins.createProject(WorkflowJob.class, "p" );
Though I'm not sure this is good for master branch, I was working from repo tag 'workflow-job-2.36'. Yes, please submit a pull request with your proposal. The "how to use incrementals" blog post from Jesse Glick and the Jenkins incrementals tools README are two good references that should allow you to more easily develop changes across multiple plugins.
I'm not expert enough in the details of the pipeline flow definition to comment on the viability of the change you're proposing. The pipeline maintainers will be much better suited to assess the impact of the proposed change than I am. They are most likely to prefer a pull request as the way to review the idea.
angrygami Thanks for looking into this. Note that WorkflowJob.getSCMs includes SCMs for things like shared libraries or uses of the checkout step from inside of the Pipeline, so IIUC changes to any of those SCMs will trigger a build after the first successful build has completed (and for one extra build even if the Pipeline was changed to no longer use those SCMs). The proposed API would only work for the SCM used for the Jenkinsfile, and only for CpsScmFlowDefinition, which is still better than today, but I think we'd want to make sure to be very clear about what would and would not work in the changelog and any documentation. CC bitwiseman in case he has any thoughts.
Angry Gami
added a comment - The proposed API would only work for the SCM used for the Jenkinsfile, and only for CpsScmFlowDefinition
This was my intention. I understand that some FlowDefintion subclasses are not using scm at all. I was annoyed that one that definitely do don't work as expected. I've seen your comment on github and will make other PRs as soon as I have time 
Thanks.
Angry Gami
added a comment - The proposed API would only work for the SCM used for the Jenkinsfile, and only for CpsScmFlowDefinition
This was my intention. I understand that some FlowDefintion subclasses are not using scm at all. I was annoyed that one that definitely do don't work as expected. I've seen your comment on github and will make other PRs as soon as I have time Thanks.
Ok, all PRs are now done. Though https://github.com/jenkinsci/workflow-job-plugin/pull/147 fails and I don't understand why. Please 0u812 could you take a look?
Angry Gami
added a comment - - edited Ok, all PRs are now done. Though https://github.com/jenkinsci/workflow-job-plugin/pull/147 fails and I don't understand why. Please 0u812 could you take a look? 
Does posting the config.xml help?
In English: it's just a pipeline job using a script from SCM using a single branch (develop).
Duplication is pretty straight forward:
node { stage("checkout") { checkout scm } }At this point, the one pipeline job that's been executed will get triggered, but the other will not even write any log messages. If you delete the executed job and repeat the steps, the non-executed job will still not be notified.
Not sure about a fresh jenkins instance.