-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Jenkins 1.480.1 LTS
My job dependency graph looks like this:
A starts B. B starts C and D and 'joins' with E.
Expected behavior: [assuming everything runs successfully] A should never start before E has completed.
Reality: sometimes it does. And as it performs clean-up of some common folders, this messes up E completely.
Potentially relevant info:
- E can be run as a 'join' job from other job flows, not just A-B-C-D.
- All triggering is performed via "Trigger parameterized builds" (I'm including the parameterized-trigger component to the report as well).
[JENKINS-16384] Blocking on downstream projects
Yes, I forgot to mention that "Block when downstream is building" is of course enabled in A.
Posting <publishers> for all 5 jobs:
job_A:
<publishers>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_B</projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>false</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_B:
<publishers>
<hudson.plugins.logparser.LogParserPublisher>
<unstableOnWarning>false</unstableOnWarning>
<failBuildOnError>true</failBuildOnError>
<parsingRulesPath>make.error</parsingRulesPath>
</hudson.plugins.logparser.LogParserPublisher>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_C, job_D</projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
<join.JoinTrigger>
<joinProjects></joinProjects>
<joinPublishers>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
<properties>A_CUSTOM_PARAMETER=HELLO</properties>
</hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_E, </projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
</joinPublishers>
<evenIfDownstreamUnstable>false</evenIfDownstreamUnstable>
</join.JoinTrigger>
<hudson.notify.UpDownStreamNumberSychronizedNotify>
<times>10</times>
<itimes>10</itimes>
</hudson.notify.UpDownStreamNumberSychronizedNotify>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers/>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_C:
<publishers>
<hudson.tasks.ArtifactArchiver>
<artifacts>**/test/**/*.log</artifacts>
<latestOnly>false</latestOnly>
</hudson.tasks.ArtifactArchiver>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>true</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_D:
<publishers>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers/>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_E:
<publishers>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$JENKINS_MAIL_LIST</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>false</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>[Jenkins-Maintenance] Warning, job failed!</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
Alex Vesely
added a comment - - edited Yes, I forgot to mention that "Block when downstream is building" is of course enabled in A.
Posting <publishers> for all 5 jobs:
job_A:
<publishers>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_B</projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>false</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_B:
<publishers>
<hudson.plugins.logparser.LogParserPublisher>
<unstableOnWarning>false</unstableOnWarning>
<failBuildOnError>true</failBuildOnError>
<parsingRulesPath>make.error</parsingRulesPath>
</hudson.plugins.logparser.LogParserPublisher>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_C, job_D</projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
<join.JoinTrigger>
<joinProjects></joinProjects>
<joinPublishers>
<hudson.plugins.parameterizedtrigger.BuildTrigger>
<configs>
<hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
<configs>
<hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
<properties>A_CUSTOM_PARAMETER=HELLO</properties>
</hudson.plugins.parameterizedtrigger.PredefinedBuildParameters>
<hudson.plugins.parameterizedtrigger.CurrentBuildParameters/>
</configs>
<projects>job_E, </projects>
<condition>SUCCESS</condition>
<triggerWithNoParameters>false</triggerWithNoParameters>
</hudson.plugins.parameterizedtrigger.BuildTriggerConfig>
</configs>
</hudson.plugins.parameterizedtrigger.BuildTrigger>
</joinPublishers>
<evenIfDownstreamUnstable>false</evenIfDownstreamUnstable>
</join.JoinTrigger>
<hudson.notify.UpDownStreamNumberSychronizedNotify>
<times>10</times>
<itimes>10</itimes>
</hudson.notify.UpDownStreamNumberSychronizedNotify>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers/>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_C:
<publishers>
<hudson.tasks.ArtifactArchiver>
<artifacts>**/test/**/*.log</artifacts>
<latestOnly>false</latestOnly>
</hudson.tasks.ArtifactArchiver>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>true</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_D:
<publishers>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$DEFAULT_RECIPIENTS</recipientList>
<configuredTriggers/>
<contentType>default</contentType>
<defaultSubject>$DEFAULT_SUBJECT</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
<matrixTriggerMode>ONLY_PARENT</matrixTriggerMode>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
job_E:
<publishers>
<hudson.plugins.emailext.ExtendedEmailPublisher>
<recipientList>$JENKINS_MAIL_LIST</recipientList>
<configuredTriggers>
<hudson.plugins.emailext.plugins.trigger.FailureTrigger>
<email>
<recipientList></recipientList>
<subject>$PROJECT_DEFAULT_SUBJECT</subject>
<body>$PROJECT_DEFAULT_CONTENT</body>
<sendToDevelopers>true</sendToDevelopers>
<sendToRequester>false</sendToRequester>
<includeCulprits>false</includeCulprits>
<sendToRecipientList>true</sendToRecipientList>
</email>
</hudson.plugins.emailext.plugins.trigger.FailureTrigger>
</configuredTriggers>
<contentType>default</contentType>
<defaultSubject>[Jenkins-Maintenance] Warning, job failed!</defaultSubject>
<defaultContent>$DEFAULT_CONTENT</defaultContent>
<attachmentsPattern></attachmentsPattern>
<presendScript></presendScript>
</hudson.plugins.emailext.ExtendedEmailPublisher>
</publishers>
cjo9900
added a comment - I have managed to reproduce what I think is the issue.
and the steps are
Trigger A1
A1 triggers B1
Other Jobs triggers E1
Trigger A2
B1 triggers C1, D1
C1, D1 trigger E2
E1 runs
A2 runs
E2 waits until B2,C2,D2 have finished.
In this case I have enabled the Block when upstream is building on project E.
and seem to see the same issue as JENKINS-14918
Where E2 is seen as Blocked because E1 is building
The the queue is processed in order with A2 checked first and E2 is still classed as blocked so A2 is run.
Then E2 is checked and still blocked due A2 running.
Can you check the settings for Job E and see if this is the case.
cjo9900
added a comment - I have managed to reproduce what I think is the issue.
and the steps are
Trigger A1
A1 triggers B1
Other Jobs triggers E1
Trigger A2
B1 triggers C1, D1
C1, D1 trigger E2
E1 runs
A2 runs
E2 waits until B2,C2,D2 have finished.
In this case I have enabled the Block when upstream is building on project E.
and seem to see the same issue as JENKINS-14918
Where E2 is seen as Blocked because E1 is building
The the queue is processed in order with A2 checked first and E2 is still classed as blocked so A2 is run.
Then E2 is checked and still blocked due A2 running.
Can you check the settings for Job E and see if this is the case.
Alex Vesely
added a comment - Do you mean if job E has 'block if upstream is building' enabled? No, it has both the 'block' flags unchecked.
Can the issue be connected to the "Check job prerequisites" script that job E runs before finding a good slave for execution?
Alex Vesely
added a comment - Do you mean if job E has 'block if upstream is building' enabled? No, it has both the 'block' flags unchecked.
Can the issue be connected to the "Check job prerequisites" script that job E runs before finding a good slave for execution? cjo9900
added a comment - I have added the Slave Prerequisites plugin to Job E (without the block upstream enabled) and that seems to block job A as expected even if job E is started from other places.
This is because the plugin blocks at the executor level and the block on downstream check does include queued jobs.
So cannot see a particular cause of why Job A should run if Job E is in the queue, unless it is a timing issue in that when the join trigger schedules job E it is not seen by Job A when it does its checks which is probably Queue related item.
Which I cannot really investigate unless there is a particular case that can be seen causing job A to run instead of job E.
If you can get logs on why Job E is being blocked in the queue, before Job A is started then a particular case might be found that can be invistigated.
cjo9900
added a comment - I have added the Slave Prerequisites plugin to Job E (without the block upstream enabled) and that seems to block job A as expected even if job E is started from other places.
This is because the plugin blocks at the executor level and the block on downstream check does include queued jobs.
So cannot see a particular cause of why Job A should run if Job E is in the queue, unless it is a timing issue in that when the join trigger schedules job E it is not seen by Job A when it does its checks which is probably Queue related item.
Which I cannot really investigate unless there is a particular case that can be seen causing job A to run instead of job E.
If you can get logs on why Job E is being blocked in the queue, before Job A is started then a particular case might be found that can be invistigated.
Alex Vesely
added a comment - The issue happens very rarely (so rarely that I even thought I'd solved it by rolling back to the LTS), so these logs seem impossible to catch. All I get is a failed job every once in a while, when it's already too late to collect blocking logs.
Is there a way to collect the required kind of logs automatically?
Alex Vesely
added a comment - The issue happens very rarely (so rarely that I even thought I'd solved it by rolling back to the LTS), so these logs seem impossible to catch. All I get is a failed job every once in a while, when it's already too late to collect blocking logs.
Is there a way to collect the required kind of logs automatically? 
ikedam
added a comment - Sounds an issue of join plugin. Removed parameterized-plugin.
You'd better to update the description as "Block when downstream is building" is very important for this problem.
Throttle concurrent build plugin may help you.
ikedam
added a comment - Sounds an issue of join plugin. Removed parameterized-plugin.
You'd better to update the description as "Block when downstream is building" is very important for this problem.
Throttle concurrent build plugin may help you. 
Can you add the config.xml files from the 4 jobs, in particular the <publishers> ... </publishers> section.
And as the jobs are separate and there is no actual lick between them other than the upstream/downstream relationship there is no cause to prevent the jobs from running at the same time unless that advanced option is enabled.
See Advanced section in project A config and enable the block when downstream is building item and see if that fixes the problem.