Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43574

Lockable resources plugin does not take effect

    XMLWordPrintable

Details

    Description

      We are doing automated regression testing with jenkins and using the lockable resources plugin to run one test job at a time. But sometimes it fails to acquire the lock on the testing system and one or more test job starts before the first one finished depending on when the lockable resource plugin starts to work again. Example wrong behavior:
      17:09:23 Started by timer

      17:09:23 [lockable-resources] acquired lock on []

      17:09:23 [EnvInject] - Loading node environment variables.
      Please tell me what kind of logs do you need to debug this issue.

      Attachments

        Issue Links

          Activity

            szanics Bela Szanics added a comment - - edited

            Tried it with resource name and label but the same issue happens. The problem started to happen after the upgrade to 2.0 so we plan to downgrade to 1.12.1 and will check what happens.

            szanics Bela Szanics added a comment - - edited Tried it with resource name and label but the same issue happens. The problem started to happen after the upgrade to 2.0 so we plan to downgrade to 1.12.1 and will check what happens.
            wheredevel Pavel Bekkerman added a comment - - edited

            I believe this is connected to: https://github.com/jenkinsci/lockable-resources-plugin/issues/24 .

            (The strange thing to me is: I've never had problems before with Lockable Resources, till I started working with Generated Jobs - and try to lock 'em.)

            Also I upgraded recently to 2.46.1. Before, Lockables worked fine with 2.x versions - not with Generated Jobs though - haven't tried back than. 

            Any idea how to solve it? A workaround? 

            wheredevel Pavel Bekkerman added a comment - - edited I believe this is connected to: https://github.com/jenkinsci/lockable-resources-plugin/issues/24  . (The strange thing to me is: I've never had problems before with Lockable Resources, till I started working with Generated Jobs - and try to lock 'em.) Also I upgraded recently to 2.46.1. Before, Lockables worked fine with 2.x versions - not with Generated Jobs though - haven't tried back than.  Any idea how to solve it? A workaround? 
            wheredevel Pavel Bekkerman added a comment - - edited

            Workaround: downgrade to ver. 1.11.2 of the plugin.

            Now timeout doesn't occur after 1 minute.

            wheredevel Pavel Bekkerman added a comment - - edited Workaround: downgrade to ver. 1.11.2 of the plugin. Now timeout doesn't occur after 1 minute.
            szanics Bela Szanics added a comment - - edited

            Hi,

            >I believe this is connected to: https://github.com/jenkinsci/lockable-resources-plugin/issues/24 .

            Yes, I've also found this bug but this bug is really old and my problems started after I upgraded to the latest version. I checked the diff between the releases and noticed that the lock queue related methods got some refactoring. Probably a bug was introduced there.

            My findings on this issue so far:

            This issue is  really weird because we have 3 resource for our 3 different release and this problem only happens with one of them but it happens constantly with that one after the jenkins/plugin upgrade(we also did both). Tried to harmonize  the job configuration for the three release(the faulty one used resource name instead of label and the resource name contained underscore instead of spaces) but the bug still occurs. So I tried to downgrade the plugin but the problem still occurred with labels so I tested it with resource name and it seems like it works(so far). Although I had to revert the changes on the resource name(the job seems to be split the resource name along the spaces so it couldn't find it when space is used. This seems to be another bug in the plugin since as I said the other resources are have this naming convention and the plugin could find it with the label.)

             Br,

            Bela

             

            szanics Bela Szanics added a comment - - edited Hi, >I believe this is connected to: https://github.com/jenkinsci/lockable-resources-plugin/issues/24  . Yes, I've also found this bug but this bug is really old and my problems started after I upgraded to the latest version. I checked the diff between the releases and noticed that the lock queue related methods got some refactoring. Probably a bug was introduced there. My findings on this issue so far: This issue is  really weird because we have 3 resource for our 3 different release and this problem only happens with one of them but it happens constantly with that one after the jenkins/plugin upgrade(we also did both). Tried to harmonize  the job configuration for the three release(the faulty one used resource name instead of label and the resource name contained underscore instead of spaces) but the bug still occurs. So I tried to downgrade the plugin but the problem still occurred with labels so I tested it with resource name and it seems like it works(so far). Although I had to revert the changes on the resource name(the job seems to be split the resource name along the spaces so it couldn't find it when space is used. This seems to be another bug in the plugin since as I said the other resources are have this naming convention and the plugin could find it with the label.)  Br, Bela  

            I had only one resource, and no special chars in either resource/label name.

            So, the downgrade to 1.11.2 just brought it back to normal.

            wheredevel Pavel Bekkerman added a comment - I had only one resource, and no special chars in either resource/label name. So, the downgrade to 1.11.2 just brought it back to normal.

            We had the same problem, with one of our 5 lockable resources. We downgraded to 1.11.2 and updated all configuration using the plugin and everything is working like before.

            webpatrick Patrick Visser added a comment - We had the same problem, with one of our 5 lockable resources. We downgraded to 1.11.2 and updated all configuration using the plugin and everything is working like before.

            Also same problem with 2.0, generated jobs.

            At first, one job holds the resource, the other one waits, resource is showing as queued.

            After a short delay, the second job suddenly starts, the resource itself switches to "unlocked" state, and the second job shows the empty brackets: [].

            Now both jobs are running simulataniously and the resource is incorrectly unlocked.

            ext3h Andreas Ringlstetter added a comment - Also same problem with 2.0, generated jobs. At first, one job holds the resource, the other one waits, resource is showing as queued. After a short delay, the second job suddenly starts, the resource itself switches to "unlocked" state, and the second job shows the empty brackets: []. Now both jobs are running simulataniously and the resource is incorrectly unlocked.
            ext3h Andreas Ringlstetter added a comment - - edited

            For cross reference: https://github.com/jenkinsci/lockable-resources-plugin/issues/54

            debfx pointed out that version 2.0 creates invalid configurations:

                <org.jenkins.plugins.lockableresources.RequiredResourcesProperty plugin="lockable-resources@1.11.2">
                  <resourceNames>ResX ResY</resourceNames>
                  <resourceNamesVar></resourceNamesVar>
                  <resourceNumber></resourceNumber>
                  <labelName></labelName>
                </org.jenkins.plugins.lockableresources.RequiredResourcesProperty

            while the previous version serialized into a configuration like this:

                <org.jenkins.plugins.lockableresources.RequiredResourcesProperty plugin="lockable-resources@1.11.2">
                  <resourceNames>ResX ResY</resourceNames>
                </org.jenkins.plugins.lockableresources.RequiredResourcesProperty>

            Note how the missing (nulled) options were replaced by empty options instead?

            Unbreaking the configuration also works again with 2.0. Using the broken 2.0 configuration with 1.11.2 also triggers the bug.

            ext3h Andreas Ringlstetter added a comment - - edited For cross reference: https://github.com/jenkinsci/lockable-resources-plugin/issues/54 debfx pointed out that version 2.0 creates invalid configurations: <org.jenkins.plugins.lockableresources.RequiredResourcesProperty plugin= "lockable-resources@1.11.2" > <resourceNames>ResX ResY</resourceNames> <resourceNamesVar></resourceNamesVar> <resourceNumber></resourceNumber> <labelName></labelName> </org.jenkins.plugins.lockableresources.RequiredResourcesProperty while the previous version serialized into a configuration like this: <org.jenkins.plugins.lockableresources.RequiredResourcesProperty plugin= "lockable-resources@1.11.2" > <resourceNames>ResX ResY</resourceNames> </org.jenkins.plugins.lockableresources.RequiredResourcesProperty> Note how the missing (nulled) options were replaced by empty options instead? Unbreaking the configuration also works again with 2.0. Using the broken 2.0 configuration with 1.11.2 also triggers the bug.
            aishak01 Alec Ishak added a comment - - edited

            I believe I found a solution for this in the plugin code. In RequiredResourcesProperty.java there is this snippet in the RequiredResourcesProperty constructor. 

            this.resourceNames = resourceNames;
            this.resourceNamesVar = resourceNamesVar;
            this.resourceNumber = resourceNumber;

            The label name is set a little further down in the same way. It looks like these are getting set to an empty string when they should be getting set to null. I just added a little type checking and it works for me. 

            this.resourceNames = resourceNames;
            if (resourceNamesVar.equals("")) {
            this.resourceNamesVar = null
            } else {
            this.resourceNamesVar = resourceNamesVar;
            }
            if (resourceNumber.equals("")) {
            this.resourceNumber = null
            } else {
            this.resourceNumber = resourceNumber;
            }

            This fixes the job config issue pointed out by Andreas. 

            aishak01 Alec Ishak added a comment - - edited I believe I found a solution for this in the plugin code. In RequiredResourcesProperty.java there is this snippet in the RequiredResourcesProperty constructor.  this.resourceNames = resourceNames; this.resourceNamesVar = resourceNamesVar; this.resourceNumber = resourceNumber; The label name is set a little further down in the same way. It looks like these are getting set to an empty string when they should be getting set to null. I just added a little type checking and it works for me.  this.resourceNames = resourceNames; if (resourceNamesVar.equals("")) { this.resourceNamesVar = null } else { this.resourceNamesVar = resourceNamesVar; } if (resourceNumber.equals("")) { this.resourceNumber = null } else { this.resourceNumber = resourceNumber; } This fixes the job config issue pointed out by Andreas. 
            krulls Stephan Krull added a comment -

            @abayer: Are you aware of this? We still have this issue in version 2.1

            krulls Stephan Krull added a comment - @ abayer : Are you aware of this? We still have this issue in version 2.1
            krulls Stephan Krull added a comment -

            Set up a PR to get that one done.

            krulls Stephan Krull added a comment - Set up a PR to get that one done.

            Code changed in jenkins
            User: krull
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java
            src/test/java/org/jenkins/plugins/lockableresources/BasicIntegrationTest.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/1e440bfbbd803400065c062ae6bba9b86a7e45ac
            Log:
            JENKINS-43574: fix construction of RequiredResourcesProperty

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: krull Path: src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java src/test/java/org/jenkins/plugins/lockableresources/BasicIntegrationTest.java http://jenkins-ci.org/commit/lockable-resources-plugin/1e440bfbbd803400065c062ae6bba9b86a7e45ac Log: JENKINS-43574 : fix construction of RequiredResourcesProperty

            Code changed in jenkins
            User: krull
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/90bee03e8972183210ca5510f4e276bc8097cdb7
            Log:
            JENKINS-43574: replace whitespaces with tabs

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: krull Path: src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java http://jenkins-ci.org/commit/lockable-resources-plugin/90bee03e8972183210ca5510f4e276bc8097cdb7 Log: JENKINS-43574 : replace whitespaces with tabs

            Code changed in jenkins
            User: Antonio Muniz
            Path:
            src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java
            src/test/java/org/jenkins/plugins/lockableresources/BasicIntegrationTest.java
            http://jenkins-ci.org/commit/lockable-resources-plugin/67a01b44f4a995955e0888db62f927fcc740697a
            Log:
            Merge pull request #89 from krulls/master

            JENKINS-43574: fix construction of RequiredResourcesProperty

            Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/3b9717e389f7...67a01b44f4a9

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Antonio Muniz Path: src/main/java/org/jenkins/plugins/lockableresources/RequiredResourcesProperty.java src/test/java/org/jenkins/plugins/lockableresources/BasicIntegrationTest.java http://jenkins-ci.org/commit/lockable-resources-plugin/67a01b44f4a995955e0888db62f927fcc740697a Log: Merge pull request #89 from krulls/master JENKINS-43574 : fix construction of RequiredResourcesProperty Compare: https://github.com/jenkinsci/lockable-resources-plugin/compare/3b9717e389f7...67a01b44f4a9

            Released as 2.2

            amuniz Antonio Muñiz added a comment - Released as 2.2

            People

              Unassigned Unassigned
              szanics Bela Szanics
              Votes:
              11 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: