-
Improvement
-
Resolution: Unresolved
-
Major
I use a retry block to perform requests on a remote server that I don't control (e.g., Apple's Code Signing or Notarization service), and sometimes that remote server is offline.
Currently retry immediately retries upon failure, so when the remote server is offline for an extended period of time, my pipeline project quickly retries and then the build fails.
It would be nice to have the ability to make retry wait after each failure, with optional exponential backoff (doubling the wait time after each failure), to give these builds a greater chance of success. For example:
retry(tries:5, waitSecondsBetweenFailures:30, doubleWaitTimeAfterEachFailure:true)
- is related to
-
-
- Open
-
[JENKINS-54735] Pipeline retry clause: optionally delay between retries
Adam Brousseau
added a comment - in response to the workaround, what about this...?
ret = false retry(5) { if (ret) { sleep time: 30, unit: 'SECONDS' } else { ret = true } sh 'some-failure-prone-script' }
Adam Brousseau
added a comment - in response to the workaround, what about this...?
ret = false
retry(5) {
if (ret) {
sleep time: 30, unit: 'SECONDS'
} else {
ret = true
}
sh 'some-failure-prone-script'
}
Here's a different approach:
timeout(10) {
waitUntil {
script {
def r = sh returnStatus: true, script: 'command'
return r == 0
}
}
}
This will retry for up to 10 minutes. waitUntil automatically adds delays between the attempts.
asgeirn
added a comment - - edited Here's a different approach:
timeout(10) {
waitUntil {
script {
def r = sh returnStatus: true , script: 'command'
return r == 0
}
}
}
This will retry for up to 10 minutes. waitUntil automatically adds delays between the attempts. 
Matt Weisel
added a comment - I created a retry with exponential backoff and jitter method:
def retryWithBackoff(int retries = 1, int backoffSeconds = 5, int jitterSeconds = 30, Closure op) { def random = new Random(); int backoffSecondsCurrent = 0; retry(retries) { if (backoffSecondsCurrent > 0) { sleep backoffSecondsCurrent } backoffSecondsCurrent += backoffSeconds + random.nextInt(jitterSeconds) if (backoffSecondsCurrent < 0) { Log.warn("Backoff seconds rolled over, got ${backoffSecondsCurrent}, resetting to ${backoffSeconds}") backoffSecondsCurrent = backoffSeconds } op() } }
Matt Weisel
added a comment - I created a retry with exponential backoff and jitter method:
def retryWithBackoff( int retries = 1, int backoffSeconds = 5, int jitterSeconds = 30, Closure op) {
def random = new Random();
int backoffSecondsCurrent = 0;
retry(retries) {
if (backoffSecondsCurrent > 0) {
sleep backoffSecondsCurrent
}
backoffSecondsCurrent += backoffSeconds + random.nextInt(jitterSeconds)
if (backoffSecondsCurrent < 0) {
Log.warn( "Backoff seconds rolled over, got ${backoffSecondsCurrent}, resetting to ${backoffSeconds}" )
backoffSecondsCurrent = backoffSeconds
}
op()
}
}
Francisco Navarro
made changes -
| Assignee | New: Francisco Navarro [ grg121 ] |
Francisco Navarro
made changes -
| Assignee | Original: Francisco Navarro [ grg121 ] |
Francisco Navarro
made changes -
| Comment | [ Hello there, I'm interested in this issue. I'm going to try doing this. ] |
A Pull Request exists to add this functionality and an associated issue (JENKINS-59678) was created, however it seems to have stalled. I'm linking it here in hopes of reviving it.
João Pinto
added a comment - - edited A Pull Request exists to add this functionality and an associated issue ( JENKINS-59678 ) was created, however it seems to have stalled. I'm linking it here in hopes of reviving it. João Pinto
made changes -
| Link | New: This issue is related to JENKINS-59678 [ JENKINS-59678 ] |
A workaround in the meantime:
retry(5) { ret = sh returnStatus: true, script: 'some-failure-prone-script' if (ret) { sleep time: 30, unit: 'SECONDS' error 'some-failure-prone-script failed' } }…but there's no way to know when we're on the last retry attempt (see JENKINS-49341), so it unnecessarily waits after the last retry attempt fails.