Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36479

Locked resources not freed up by Pipeline job hard kill

    XMLWordPrintable

Details

    Description

      Since LockStepExecution.Callback.finished(context) never gets called in the case of a hard kill, resources can be locked forever when a build is hard killed. It's possible to manually unlock those resources from the UI, but it'd be preferable to have some behavior that detects this scenario and is able to unlock resources locked by defunct builds.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-40368 Locked resources are not always freed up on Pipeline hard kill when there are other pipelines waiting on the Resource

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-28183 Hard killed job's stage blocks stage in following jobs

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link CloudBees Internal OSS-762

          Web Link PR #34

          Web Link PR #35

          Activity

            Ascending order - Click to sort in descending order
            View 7 older comments
            abayer Andrew Bayer added a comment -

            I can't see a way to clear the lock without either waiting for a new lock request to come in (as my PR does) or having an async recurring task running periodically checking every locked resource for defunct locks. LockRunListener doesn't seem to fire on hard kill or while-running build deletion, so far as I can tell from my experiments...

            abayer Andrew Bayer added a comment - I can't see a way to clear the lock without either waiting for a new lock request to come in (as my PR does) or having an async recurring task running periodically checking every locked resource for defunct locks. LockRunListener doesn't seem to fire on hard kill or while-running build deletion, so far as I can tell from my experiments...
            abayer Andrew Bayer added a comment -

            New PR (https://github.com/jenkinsci/lockable-resources-plugin/pull/35) probably supersedes #34 - updating LockRunListener to listen on Run rather than AbstractBuild seems to, well, fix everything!

            abayer Andrew Bayer added a comment - New PR ( https://github.com/jenkinsci/lockable-resources-plugin/pull/35 ) probably supersedes #34 - updating LockRunListener to listen on Run rather than AbstractBuild seems to, well, fix everything!
            abayer Andrew Bayer added a comment -

            Fixed as of next release (presumably 1.10), which should be coming shortly, I think.

            The fix makes LockRunListener fire correctly on Run not just AbstractBuild. That alone did the trick for both hard killed builds and deleted-while-in-progress builds, and doesn't require queuing up a new lock request to clear the defunct lock. Woo.

            abayer Andrew Bayer added a comment - Fixed as of next release (presumably 1.10), which should be coming shortly, I think. The fix makes LockRunListener fire correctly on Run not just AbstractBuild . That alone did the trick for both hard killed builds and deleted-while-in-progress builds, and doesn't require queuing up a new lock request to clear the defunct lock. Woo.
            aheritier Arnaud Héritier added a comment -

            Cool abayer
            Could we also add some documentations/samples about it. It's always not clear for me how this feature should be used and what is its interest.

            Thx

            aheritier Arnaud Héritier added a comment - Cool abayer Could we also add some documentations/samples about it. It's always not clear for me how this feature should be used and what is its interest. Thx
            abayer Andrew Bayer added a comment -

            Talk to amuniz in re docs/samples - I honestly don't know much about how to use the plugin, I just decided to fix the bug. =)

            abayer Andrew Bayer added a comment - Talk to amuniz in re docs/samples - I honestly don't know much about how to use the plugin, I just decided to fix the bug. =)

            People

              abayer Andrew Bayer
              abayer Andrew Bayer
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: