Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18786

Port allocation blocks jobs from executing concurrently

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • port-allocator-plugin
    • None
    • Jenkins 1.523, port-allocator 1.6

      I'm assigning one port in configuration, called Arquillian. When using version 1.6 of port allocator, only one job starts execution and others are blocked by it.

      Thread dump had following:

      "Executor #1 for virt-slave1 : executing Project #3464" Id=38 Group=main WAITING on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Native Method)
              -  waiting on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Object.java:503)
              at hudson.model.ResourceController.execute(ResourceController.java:80)
              at hudson.model.Executor.run(Executor.java:247)
      
      
      "Executor #5 for virt-slave1 : executing Project #3465" Id=42 Group=main WAITING on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Native Method)
              -  waiting on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Object.java:503)
              at hudson.model.ResourceController.execute(ResourceController.java:80)
              at hudson.model.Executor.run(Executor.java:247)
      
      "Executor #4 for virt-slave2 : executing Project #3463 / waiting for hudson.remoting.Channel@279eb40d:virt-slave2" Id=48 Group=main TIMED_WAITING on hudson.remoting.UserRequest@20f46461
              at java.lang.Object.wait(Native Method)
              -  waiting on hudson.remoting.UserRequest@20f46461
              at hudson.remoting.Request.call(Request.java:146)
              at hudson.remoting.Channel.call(Channel.java:722)
              at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:162)
              at com.sun.proxy.$Proxy32.join(Unknown Source)
              at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:925)
              at hudson.Launcher$ProcStarter.join(Launcher.java:360)
              at hudson.tasks.Maven.perform(Maven.java:327)
              at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
              at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
              at hudson.model.Build$BuildExecution.build(Build.java:199)
              at hudson.model.Build$BuildExecution.doRun(Build.java:160)
              at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
              at hudson.model.Run.execute(Run.java:1593)
              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
              at hudson.model.ResourceController.execute(ResourceController.java:88)
              at hudson.model.Executor.run(Executor.java:247)
      
      Executor #5 for virt-slave1 : executing Project #3465
      
      "Executor #5 for virt-slave1 : executing Project #3465" Id=42 Group=main WAITING on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Native Method)
              -  waiting on hudson.model.Queue@67a7efcb
              at java.lang.Object.wait(Object.java:503)
              at hudson.model.ResourceController.execute(ResourceController.java:80)
              at hudson.model.Executor.run(Executor.java:247)
      

      After adding https://github.com/jenkinsci/jenkins/pull/853, log got following lines:

      Jul 17, 2013 7:23:45 AM hudson.model.ResourceList _getConflict
      INFO: Collision with standalone port ARQUILLIAN(1)=1 and standalone port ARQUILLIAN(1)
      

      Downgrading to version 1.5 of port allocator solved the problem

          [JENKINS-18786] Port allocation blocks jobs from executing concurrently

          Jyrki Puttonen created issue -
          Jyrki Puttonen made changes -
          Environment Original: Jenkins .1523, port-allocator 1.6 New: Jenkins 1.523, port-allocator 1.6
          Summary Original: Port allocation blocks all jobs from executing New: Port allocation blocks jobs from executing concurrently

          This is related to the fix for JENKINS-11255. That change blocks makes the build queue rather than just block mid-build waiting for a port to become available.

          It is not clear to me from your report what problem this is causing you.

          • Do the queued builds continue once the build holding the blocked port has completed?
          • Is a build on one slave being blocked because a port is being used on another slave?
          • Do you have a use case where you need all of the slaves to start running but then block during the build?

          I would help greatly if you could describe a simple test case that allows your problem to be reproduced/tested.

          Richard Mortimer added a comment - This is related to the fix for JENKINS-11255 . That change blocks makes the build queue rather than just block mid-build waiting for a port to become available. It is not clear to me from your report what problem this is causing you. Do the queued builds continue once the build holding the blocked port has completed? Is a build on one slave being blocked because a port is being used on another slave? Do you have a use case where you need all of the slaves to start running but then block during the build? I would help greatly if you could describe a simple test case that allows your problem to be reproduced/tested.

          I have three slaves and each of them have multiple executors. There's two different jobs which both allocate a port with name ARQUILLIAN. Both of these jobs have concurrent builds.

          Now only one build gets executed at a time, leaving other slaves and executors unused. Other builds stay in the queue with text "Blocked by <running build>". When the running build is completed, next one starts.

          At some times, multiple builds are assigned to executors, but only one of them is actually running. Others are just waiting with empty consoles.

          So, answers to your questions:
          Do the queued builds continue once the build holding the blocked port has completed?
          Yes, they do

          Is a build on one slave being blocked because a port is being used on another slave?
          Yes

          Do you have a use case where you need all of the slaves to start running but then block during the build?
          No, I just need one random port assigned to one job.

          Jyrki Puttonen added a comment - I have three slaves and each of them have multiple executors. There's two different jobs which both allocate a port with name ARQUILLIAN. Both of these jobs have concurrent builds. Now only one build gets executed at a time, leaving other slaves and executors unused. Other builds stay in the queue with text "Blocked by <running build>". When the running build is completed, next one starts. At some times, multiple builds are assigned to executors, but only one of them is actually running. Others are just waiting with empty consoles. So, answers to your questions: Do the queued builds continue once the build holding the blocked port has completed ? Yes, they do Is a build on one slave being blocked because a port is being used on another slave ? Yes Do you have a use case where you need all of the slaves to start running but then block during the build ? No, I just need one random port assigned to one job.

          Thanks for the information. That all makes sense. I've reproduced the problem here and have a proposed fix.

          https://github.com/jenkinsci/port-allocator-plugin/pull/3

          Can you test the auto-build version with that pull requested included and confirm that it fixes the problem for you.

          https://buildhive.cloudbees.com/job/jenkinsci/job/port-allocator-plugin/12/org.jenkins-ci.plugins$port-allocator/artifact/org.jenkins-ci.plugins/port-allocator/1.7-SNAPSHOT/port-allocator-1.7-SNAPSHOT.hpi

          Richard Mortimer added a comment - Thanks for the information. That all makes sense. I've reproduced the problem here and have a proposed fix. https://github.com/jenkinsci/port-allocator-plugin/pull/3 Can you test the auto-build version with that pull requested included and confirm that it fixes the problem for you. https://buildhive.cloudbees.com/job/jenkinsci/job/port-allocator-plugin/12/org.jenkins-ci.plugins$port-allocator/artifact/org.jenkins-ci.plugins/port-allocator/1.7-SNAPSHOT/port-allocator-1.7-SNAPSHOT.hpi
          Richard Mortimer made changes -
          Assignee Original: ramapulavarthi [ ramapulavarthi ] New: Richard Mortimer [ oldelvet ]

          Great, I'm able to test this on next Monday, I'll report results then.

          Jyrki Puttonen added a comment - Great, I'm able to test this on next Monday, I'll report results then.

          Seems to be working properly, thanks a lot!

          Jyrki Puttonen added a comment - Seems to be working properly, thanks a lot!

          Thanks for testing.

          Richard Mortimer added a comment - Thanks for testing.

          Assign to Peter (who fixed JENKINS-11255) to check that he has no objections to the proposed fix for this issue.

          I don't think the fix will break pooled ports but you never know!

          Richard Mortimer added a comment - Assign to Peter (who fixed JENKINS-11255 ) to check that he has no objections to the proposed fix for this issue. I don't think the fix will break pooled ports but you never know!
          Richard Mortimer made changes -
          Assignee Original: Richard Mortimer [ oldelvet ] New: Peter Wilcsinszky [ pepov ]

            oldelvet Richard Mortimer
            jyrkip Jyrki Puttonen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: