Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43599

Distributed locks occasionally failing across nodes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Are pipeline locks expected to behave when used by two nodes simultaneously?

      My Jenkinsfile defines the following helper -

      Jenkinsfile
      def log(msg) {
        echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
      }
      
      def lockDB(number, callback) {
        try {
          log "${number}: Acquiring lock ..."
          lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
            log "${number}: Acquired!"
            callback()
            log "${number}: Releasing..."
          }
        } finally {
          log "${number}: Released!"
        }
      }
      

      I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

      Job1
      [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
      [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
      [Pipeline] [featureTests] lock
      [featureTests] Trying to acquire lock on [boom-test-database-1]
      [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
      [featureTests] [boom-test-database-1] is locked, waiting...
      [featureTests] Lock acquired on [boom-test-database-1]
      [featureTests] 2017-04-07T14:45:57: 1: Acquired!
      ... running tests
      [featureTests] Lock released on resource [boom-test-database-1]
      [featureTests] 2017-04-07T14:52:04: 1: Released!
      
      Job2
      [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
      [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
      [Pipeline] [featureTests] lock
      [featureTests] Trying to acquire lock on [boom-test-database-1]
      [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
      [featureTests] [boom-test-database-1] is locked, waiting...
      [featureTests] Lock acquired on [boom-test-database-1]
      [featureTests] 2017-04-07T14:51:24: 1: Acquired!
      ... running tests
      [featureTests] Lock released on resource [boom-test-database-1]
      [featureTests] 2017-04-07T14:56:43: 1: Released!
      

      Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jdelstrother Jonathan del Strother
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: