[JENKINS-26616] Multi-config project concurrent build race - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Duplicate
Component/s: gerrit-trigger-plugin, matrix-project-plugin
Labels:
None
Environment:
Jenkins ver. 1.597
Matrix project plugin ver. 1.4
Gerrit trigger ver. 2.12.0
Gerrit ver. 2.8.3
Master running on an Ubuntu 10.04.1 machine
2 slaves running on Ubuntu 14.04.1 machines

Similar Issues:

Show

Description

I have a multi-config job (4 configurations) with "Execute concurrent builds if necessary" enabled. The job is triggered by new patch sets being pushed to gerrit (using the Gerrit trigger plugin).

This seems to work fine when only pushing a single new patch set to gerrit at a time. When two new patch sets are pushed at the same time (git add, git commit, git add, git commit, git push) all 4 sub-builds will fail for one of the patch sets with errors like this (and bring down the Jenkins thread they are running on):

java.lang.IllegalStateException: cannot create a build with number 252 since that (or higher) is already in use among [151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 253]
	at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
	at hudson.model.RunMap.put(RunMap.java:189)
	at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
	at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
	at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1200)
	at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
	at hudson.model.Executor.run(Executor.java:213)

In this case build 252 was the build number for the bad patch set, 253 for the good patch set. My guess as to what happened is that the parent builds started in one order (252, then 253) but the sub-builds started in the opposite order (253, then 252). The Jenkins log is missing messages about the 252 sub-builds though so I can't confirm this (presumably the sub-build started messages are missing because the threads died before writing the messages).

I don't know if this is just a race condition as suggested in the title or if it is a problem with how the gerrit trigger plugin launches jobs when more than one new patch set is introduced by a single push to gerrit.

Attachments

Issue Links

duplicates

JENKINS-26739 ISE from AbstractLazyLoadRunMap.proposeNewNumber for concurrent matrix builds

Closed

Activity

Descending order - Click to sort in ascending order

Hide

Permalink

Jesse Glick added a comment - 2015-03-02 22:58

Tracking as ~~JENKINS-26739~~.

Show

Jesse Glick added a comment - 2015-03-02 22:58 Tracking as JENKINS-26739 .

Hide

Permalink

Daniel Beck added a comment - 2015-02-18 11:01

In fact, it's probably a core issue and unrelated to matrix projects, see linked ~~JENKINS-26739~~.

Show

Daniel Beck added a comment - 2015-02-18 11:01 In fact, it's probably a core issue and unrelated to matrix projects, see linked JENKINS-26739 .

Hide

Permalink

Daniel Beck added a comment - 2015-02-18 10:57

Since the build number isn't duplicated, but skipped, this seems to be a different failure mode than Gerrit Trigger.

Show

Daniel Beck added a comment - 2015-02-18 10:57 Since the build number isn't duplicated, but skipped, this seems to be a different failure mode than Gerrit Trigger.

Hide

Permalink

Dmitry Maslakov added a comment - 2015-02-18 10:40

I do not use gerrit. More likely, stack trace shows, this problem in matrix plug-in.

Show

Dmitry Maslakov added a comment - 2015-02-18 10:40 I do not use gerrit. More likely, stack trace shows, this problem in matrix plug-in.

Hide

Permalink

rsandell added a comment - 2015-02-18 10:31

Of the gerrit trigger plugin.

Show

rsandell added a comment - 2015-02-18 10:31 Of the gerrit trigger plugin.

Hide

Permalink

Dmitry Maslakov added a comment - 2015-02-18 10:08

@rsandell, beta release of what product?

Show

Dmitry Maslakov added a comment - 2015-02-18 10:08 @rsandell, beta release of what product?

Hide

Permalink

rsandell added a comment - 2015-02-18 09:50

This sounds like having the same root cause as ~~JENKINS-23152~~, have you tried the new beta release?

Show

rsandell added a comment - 2015-02-18 09:50 This sounds like having the same root cause as JENKINS-23152 , have you tried the new beta release?

Hide

Permalink

Dmitry Maslakov added a comment - 2015-02-18 03:23

After upgrade from 1.596 to 1.599 I got same issues. It happened on matrix job.

Here is a log, see how same build numbers are probed several times and failed:

Feb 18, 2015 6:55:09 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Feb 18, 2015 6:55:09 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Feb 18, 2015 6:55:14 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Feb 18, 2015 6:55:15 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Feb 18, 2015 6:55:19 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Feb 18, 2015 6:55:20 AM hudson.model.Executor run
SEVERE: Unexpected executor death
java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441]
        at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361)
        at hudson.model.RunMap.put(RunMap.java:189)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284)
        at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
        at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
        at hudson.model.Executor.run(Executor.java:213)

Show

Dmitry Maslakov added a comment - 2015-02-18 03:23 After upgrade from 1.596 to 1.599 I got same issues. It happened on matrix job. Here is a log, see how same build numbers are probed several times and failed: Feb 18, 2015 6:55:09 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213) Feb 18, 2015 6:55:09 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213) Feb 18, 2015 6:55:14 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213) Feb 18, 2015 6:55:15 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213) Feb 18, 2015 6:55:19 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2439 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213) Feb 18, 2015 6:55:20 AM hudson.model.Executor run SEVERE: Unexpected executor death java.lang.IllegalStateException: cannot create a build with number 2440 since that (or higher) is already in use among [2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2441] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:361) at hudson.model.RunMap.put(RunMap.java:189) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:284) at hudson.matrix.MatrixConfiguration.newBuild(MatrixConfiguration.java:74) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144) at hudson.model.Executor.run(Executor.java:213)

People

Assignee:: Jesse Glick

Reporter:: David Emett

Votes:: 4 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2015-01-26 20:22

Updated:: 2015-03-02 22:58

Resolved:: 2015-03-02 22:57