Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43599

Distributed locks occasionally failing across nodes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Are pipeline locks expected to behave when used by two nodes simultaneously?

      My Jenkinsfile defines the following helper -

      Jenkinsfile
      def log(msg) {
        echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
      }
      
      def lockDB(number, callback) {
        try {
          log "${number}: Acquiring lock ..."
          lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
            log "${number}: Acquired!"
            callback()
            log "${number}: Releasing..."
          }
        } finally {
          log "${number}: Released!"
        }
      }
      

      I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

      Job1
      [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
      [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
      [Pipeline] [featureTests] lock
      [featureTests] Trying to acquire lock on [boom-test-database-1]
      [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
      [featureTests] [boom-test-database-1] is locked, waiting...
      [featureTests] Lock acquired on [boom-test-database-1]
      [featureTests] 2017-04-07T14:45:57: 1: Acquired!
      ... running tests
      [featureTests] Lock released on resource [boom-test-database-1]
      [featureTests] 2017-04-07T14:52:04: 1: Released!
      
      Job2
      [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
      [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
      [Pipeline] [featureTests] lock
      [featureTests] Trying to acquire lock on [boom-test-database-1]
      [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
      [featureTests] [boom-test-database-1] is locked, waiting...
      [featureTests] Lock acquired on [boom-test-database-1]
      [featureTests] 2017-04-07T14:51:24: 1: Acquired!
      ... running tests
      [featureTests] Lock released on resource [boom-test-database-1]
      [featureTests] 2017-04-07T14:56:43: 1: Released!
      

      Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

        Attachments

          Activity

          jdelstrother Jonathan del Strother created issue -
          jdelstrother Jonathan del Strother made changes -
          Field Original Value New Value
          Description Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }


          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.


          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!




          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!



          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          {{
          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }
          }}

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

          {{
          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!
          }}

          {{
          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!
          }}


          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          jdelstrother Jonathan del Strother made changes -
          Description Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          {{
          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }
          }}

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

          {{
          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!
          }}

          {{
          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!
          }}


          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -


          {{def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }}}


          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

          {{
          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!
          }}

          {{
          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!
          }}


          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          jdelstrother Jonathan del Strother made changes -
          Description Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -


          {{def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }}}


          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

          {{
          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!
          }}

          {{
          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!
          }}


          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          Are pipeline locks expected to behave when used by two nodes simultaneously?

           

          {{def log(msg) \{}}
          {{  echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)}}
          {{}}}

          {{def lockDB(number, callback) \{}}
          {{  try \{}}
          {{    log "$\{number}: Acquiring lock ..."}}
          {{    lock(resource: "boom-test-database-$\{number}", inversePrecedence: true) \{}}
          {{      log "$\{number}: Acquired!"}}
          {{      callback()}}
          {{      log "$\{number}: Releasing..."}}
          {{    }}}
          {{  } finally \{}}
          {{    log "$\{number}: Released!"}}
          {{  }}}
          {{}}}

           

          My Jenkinsfile defines the following helper -

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

           

          Job 1:


          {{ [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA}}
          {{ [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...}}
          {{ [Pipeline] [featureTests] lock}}
          {{ [featureTests] Trying to acquire lock on [boom-test-database-1]}}
          {{ [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.}}
          {{ [featureTests] [boom-test-database-1] is locked, waiting...}}
          {{ [featureTests] Lock acquired on [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:45:57: 1: Acquired!}}
          {{ ... running tests}}
          {{ [featureTests] Lock released on resource [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:52:04: 1: Released!}}

           

          {{Job2 :}}


          {{ [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ}}
          {{ [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...}}
          {{ [Pipeline] [featureTests] lock}}
          {{ [featureTests] Trying to acquire lock on [boom-test-database-1]}}
          {{ [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.}}
          {{ [featureTests] [boom-test-database-1] is locked, waiting...}}
          {{ [featureTests] Lock acquired on [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:51:24: 1: Acquired!}}
          {{ ... running tests}}
          {{ [featureTests] Lock released on resource [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:56:43: 1: Released!}}

          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          jdelstrother Jonathan del Strother made changes -
          Description Are pipeline locks expected to behave when used by two nodes simultaneously?

           

          {{def log(msg) \{}}
          {{  echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)}}
          {{}}}

          {{def lockDB(number, callback) \{}}
          {{  try \{}}
          {{    log "$\{number}: Acquiring lock ..."}}
          {{    lock(resource: "boom-test-database-$\{number}", inversePrecedence: true) \{}}
          {{      log "$\{number}: Acquired!"}}
          {{      callback()}}
          {{      log "$\{number}: Releasing..."}}
          {{    }}}
          {{  } finally \{}}
          {{    log "$\{number}: Released!"}}
          {{  }}}
          {{}}}

           

          My Jenkinsfile defines the following helper -

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

           

          Job 1:


          {{ [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA}}
          {{ [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...}}
          {{ [Pipeline] [featureTests] lock}}
          {{ [featureTests] Trying to acquire lock on [boom-test-database-1]}}
          {{ [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.}}
          {{ [featureTests] [boom-test-database-1] is locked, waiting...}}
          {{ [featureTests] Lock acquired on [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:45:57: 1: Acquired!}}
          {{ ... running tests}}
          {{ [featureTests] Lock released on resource [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:52:04: 1: Released!}}

           

          {{Job2 :}}


          {{ [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ}}
          {{ [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...}}
          {{ [Pipeline] [featureTests] lock}}
          {{ [featureTests] Trying to acquire lock on [boom-test-database-1]}}
          {{ [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.}}
          {{ [featureTests] [boom-test-database-1] is locked, waiting...}}
          {{ [featureTests] Lock acquired on [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:51:24: 1: Acquired!}}
          {{ ... running tests}}
          {{ [featureTests] Lock released on resource [boom-test-database-1]}}
          {{ [featureTests] 2017-04-07T14:56:43: 1: Released!}}

          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          {code:title=Jenkinsfile}
          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }
          {code}

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.


          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!




          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!



          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          jdelstrother Jonathan del Strother made changes -
          Description Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          {code:title=Jenkinsfile}
          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }
          {code}

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.


          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!




          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!



          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?
          Are pipeline locks expected to behave when used by two nodes simultaneously?

          My Jenkinsfile defines the following helper -

          {code:title=Jenkinsfile}
          def log(msg) {
            echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)
          }

          def lockDB(number, callback) {
            try {
              log "${number}: Acquiring lock ..."
              lock(resource: "boom-test-database-${number}", inversePrecedence: true) {
                log "${number}: Acquired!"
                callback()
                log "${number}: Releasing..."
              }
            } finally {
              log "${number}: Released!"
            }
          }
          {code}

          I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

          {noformat:title=Job1}
          [featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA
          [featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:45:57: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:52:04: 1: Released!
          {noformat}


          {noformat:title=Job2}
          [featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ
          [featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...
          [Pipeline] [featureTests] lock
          [featureTests] Trying to acquire lock on [boom-test-database-1]
          [featureTests] Found 0 available resource(s). Waiting for correct amount: 1.
          [featureTests] [boom-test-database-1] is locked, waiting...
          [featureTests] Lock acquired on [boom-test-database-1]
          [featureTests] 2017-04-07T14:51:24: 1: Acquired!
          ... running tests
          [featureTests] Lock released on resource [boom-test-database-1]
          [featureTests] 2017-04-07T14:56:43: 1: Released!
          {noformat}


          Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jdelstrother Jonathan del Strother
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: