Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50260

lock step interacts badly with timeout step

XMLWordPrintable

      I have a job which takes a lock {} inside of a timeout {} block.

      If the timeout expires, then the lock {} is not freed, which results in future taking of the lock becoming a deadlock.

      It seems like timeout {} should properly free the lock when forcing an exit from the lock() step? Is this just a bad pipeline/workflow pattern?

          [JENKINS-50260] lock step interacts badly with timeout step

          Jacob Keller added a comment -

           

          This is with Lockable Resource Plugin 2.0

          I'm not 100% sure if the issue is a timeout() killing the lock() or not. I looked at the pipeline graph, and I see a timeout set to 15 minutes and the lock is blocked for 13 minutes, so it's possible this isn't the issue. I also had 2 other branches of parallel pipeline that ended up blocking for 4 days before we killed the job.

          Specifically, the lock is only used inside a single build to limit parallelism of a section that cannot be run concurrently during a parallel{} chunk. Since the lockable resource name includes the BUILD_TAG, it should be a resource unique to the individual build, and thus the only places it should become locked are in other parallel threads.

          Any other ideas what might cause the deadlock?

          Jacob Keller added a comment -   This is with Lockable Resource Plugin 2.0 I'm not 100% sure if the issue is a timeout() killing the lock() or not. I looked at the pipeline graph, and I see a timeout set to 15 minutes and the lock is blocked for 13 minutes, so it's possible this isn't the issue. I also had 2 other branches of parallel pipeline that ended up blocking for 4 days before we killed the job. Specifically, the lock is only used inside a single build to limit parallelism of a section that cannot be run concurrently during a parallel{} chunk. Since the lockable resource name includes the BUILD_TAG, it should be a resource unique to the individual build, and thus the only places it should become locked are in other parallel threads. Any other ideas what might cause the deadlock?

          Tobias Getrost added a comment -

          It looks as if the timeout is the reason why the lock is not released. I created a test to reproduce the behaviour: https://github.com/getrostt/lockable-resources-plugin/tree/jenkins-50260

          Tobias Getrost added a comment - It looks as if the timeout is the reason why the lock is not released. I created a test to reproduce the behaviour:  https://github.com/getrostt/lockable-resources-plugin/tree/jenkins-50260

          Tomas Bjerre added a comment -

          I tried using:

          Build Timeout Plugin 1.19

          Lockable Resource Plugin 2.8-SNAPSHOT

          And with pipeline:

           

          lock('myresource') {
 println "locked"
 timeout(time: 2, unit: 'SECONDS') {
  println "sleep..."
  sleep 10
  println "sleep done !!"
 }
 println "leaving timeout"
}
println "after lock"
timeout(time: 2, unit: 'SECONDS') {
 println "locking..."
 lock('myresource') {
  println "locked, sleeping..."
  sleep 10
  println "sleep done !!"
 }
 println "left lock"
}
println "left timeout"

          The log shows:

           

           

          Started by user unknown or anonymous
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] lock
Trying to acquire lock on [myresource]
Resource [myresource] did not exist. Created.
Lock acquired on [myresource]
[Pipeline] {
[Pipeline] echo
locked
[Pipeline] timeout
Timeout set to expire in 2 sec
[Pipeline] {
[Pipeline] echo
sleep...
[Pipeline] sleep
Sleeping for 10 sec
Cancelling nested steps due to timeout
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
Lock released on resource [myresource]
[Pipeline] // lock
[Pipeline] End of Pipeline
Timeout has been exceeded
Finished: ABORTED

           

           

          Would help if anyone can reproduce it with a sample pipeline and plugin versions.

          Tomas Bjerre added a comment - I tried using: Build Timeout Plugin 1.19 Lockable Resource Plugin 2.8-SNAPSHOT And with pipeline:   lock( 'myresource' ) { println "locked" timeout(time: 2, unit: 'SECONDS' ) { println "sleep..." sleep 10 println "sleep done !!" } println "leaving timeout" } println "after lock" timeout(time: 2, unit: 'SECONDS' ) { println "locking..." lock( 'myresource' ) { println "locked, sleeping..." sleep 10 println "sleep done !!" } println "left lock" } println "left timeout" The log shows:     Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] lock Trying to acquire lock on [myresource] Resource [myresource] did not exist. Created. Lock acquired on [myresource] [Pipeline] { [Pipeline] echo locked [Pipeline] timeout Timeout set to expire in 2 sec [Pipeline] { [Pipeline] echo sleep... [Pipeline] sleep Sleeping for 10 sec Cancelling nested steps due to timeout [Pipeline] } [Pipeline] // timeout [Pipeline] } Lock released on resource [myresource] [Pipeline] // lock [Pipeline] End of Pipeline Timeout has been exceeded Finished: ABORTED     Would help if anyone can reproduce it with a sample pipeline and plugin versions.

            Unassigned Unassigned
            jekeller Jacob Keller
            Votes:
            4 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: