Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43599

Distributed locks occasionally failing across nodes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • lockable-resources-plugin

      Are pipeline locks expected to behave when used by two nodes simultaneously?

      My Jenkinsfile defines the following helper -

      Jenkinsfile
      def log(msg) {
  echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
}

def lockDB(number, callback) {
  try {
    log "${number}: Acquiring lock ..."
    lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
      log "${number}: Acquired!"
      callback()
      log "${number}: Releasing..."
    }
  } finally {
    log "${number}: Released!"
  }
}

      I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

      Job1
      [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
[featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
[Pipeline] [featureTests] lock
[featureTests] Trying to acquire lock on [boom-test-database-1]
[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
[featureTests] [boom-test-database-1] is locked, waiting...
[featureTests] Lock acquired on [boom-test-database-1]
[featureTests] 2017-04-07T14:45:57: 1: Acquired!
... running tests
[featureTests] Lock released on resource [boom-test-database-1]
[featureTests] 2017-04-07T14:52:04: 1: Released!
      Job2
      [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
[featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
[Pipeline] [featureTests] lock
[featureTests] Trying to acquire lock on [boom-test-database-1]
[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
[featureTests] [boom-test-database-1] is locked, waiting...
[featureTests] Lock acquired on [boom-test-database-1]
[featureTests] 2017-04-07T14:51:24: 1: Acquired!
... running tests
[featureTests] Lock released on resource [boom-test-database-1]
[featureTests] 2017-04-07T14:56:43: 1: Released!

      Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

            Unassigned Unassigned
            jdelstrother Jonathan del Strother
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: