Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54948

Dynamic creation of cloud instance for every new trigger irrespective of free executors in existing slave cloud instance

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • openstack-cloud-plugin
    • None
    • Jenkins 2.138

      I have a multiphase job.i wanted to run both slave and master job in same node.the parent job can be built concurrently.I am creating node using Jenkins openstack cloud plugin with 6 executors.so my requirement is that each trigger of  the parentjob should create new instance and should not use existing slave node even if executors are free...

          [JENKINS-54948] Dynamic creation of cloud instance for every new trigger irrespective of free executors in existing slave cloud instance

          I am sorry but I do not understand the request.

          • Is that a pair of upstream/downstram jobs that should share a node?
          • The parent can run concurrently (on the same node I presume) with another parents, children, its child or other jobs?
          • What is the motivation to enforce new node creation for particular build execution? Is that not what all your builds want?

          Oliver Gondža added a comment - I am sorry but I do not understand the request. Is that a pair of upstream/downstram jobs that should share a node? The parent can run concurrently (on the same node I presume) with another parents, children, its child or other jobs? What is the motivation to enforce new node creation for particular build execution? Is that not what all your builds want?

          Daniel Issac added a comment -

          I have a parent job which will be calling 2 slave jobs sequentially:

          1)build job

          2)test execution job(multiconfig)

           

          Since build and test execution job need to run in same workspace,I need all these 3 jobs to run in same node...so I will use openstack cloud plugin to create instance with 6 executors.and correcting my typo,both parent job and slave jobs support concurrent builds....so if there is free executor in slave node launched in previous run of parent job,new run of the job is not creating new instance and taking free executor in old instance...so as a result,my jobs are going in queue waiting for executors and in same node when multiple build happens,I run out of disk space in slave node.

          Daniel Issac added a comment - I have a parent job which will be calling 2 slave jobs sequentially: 1)build job 2)test execution job(multiconfig)   Since build and test execution job need to run in same workspace,I need all these 3 jobs to run in same node...so I will use openstack cloud plugin to create instance with 6 executors.and correcting my typo,both parent job and slave jobs support concurrent builds....so if there is free executor in slave node launched in previous run of parent job,new run of the job is not creating new instance and taking free executor in old instance...so as a result,my jobs are going in queue waiting for executors and in same node when multiple build happens,I run out of disk space in slave node.

          Ok, I think I get that now. To me, the culprit seems to be the node permits 2+ such "executions" in parallel (nodes with given label have enough executors) but the nodes can not really handle that (running out of disk space when used simultaneously). Trying to address that by enforcing the jobs you know can clash this way to always provision new nodes (with excessive executors that cannot be used freely) is something I consider clumsy. Here are some better suggestions:

          • Rewrite your "execution" into single pipeline job so you can orchestrate the way you like while sharing single node for parent and slave build - including workspace. Restrict executors of such nodes to 1 to prevent parallelism.
          • Or use dedicated OpenStack cloud template for this kind of execution with reduced number of executors so multiple "executions" can not clash this way as there is enough executors for one.
          • Or fight the inevitable by https://wiki.jenkins.io/display/JENKINS/Throttle+Concurrent+Builds+Plugin or similar.

          Oliver Gondža added a comment - Ok, I think I get that now. To me, the culprit seems to be the node permits 2+ such "executions" in parallel (nodes with given label have enough executors) but the nodes can not really handle that (running out of disk space when used simultaneously). Trying to address that by enforcing the jobs you know can clash this way to always provision new nodes (with excessive executors that cannot be used freely) is something I consider clumsy. Here are some better suggestions: Rewrite your "execution" into single pipeline job so you can orchestrate the way you like while sharing single node for parent and slave build - including workspace. Restrict executors of such nodes to 1 to prevent parallelism. Or use dedicated OpenStack cloud template for this kind of execution with reduced number of executors so multiple "executions" can not clash this way as there is enough executors for one. Or fight the inevitable by https://wiki.jenkins.io/display/JENKINS/Throttle+Concurrent+Builds+Plugin or similar.

          Daniel Issac added a comment -

          Can you explain the second option in detail...I didn't get it.

          Daniel Issac added a comment - Can you explain the second option in detail...I didn't get it.

          Sure, since your nodes has more executors than this greedy jobs needs, I presume it is not a problem for you in general to run multiple build on as instance at a time but rather running multiple builds of particular type is what is causing a problem. Hence I suggest to have a different OS template to create nodes for this kind only so all your build would run wherever they are running but the offending job would use dedicated set of machines when the number of executors would not prevent it to run multiple builds at a time. IOW, your nodes have 6 executors but those would have just as many as the single execution of your pipeline needs needs.

          Oliver Gondža added a comment - Sure, since your nodes has more executors than this greedy jobs needs, I presume it is not a problem for you in general to run multiple build on as instance at a time but rather running multiple builds of particular type is what is causing a problem. Hence I suggest to have a different OS template to create nodes for this kind only so all your build would run wherever they are running but the offending job would use dedicated set of machines when the number of executors would not prevent it to run multiple builds at a time. IOW, your nodes have 6 executors but those would have just as many as the single execution of your pipeline needs needs.

            olivergondza Oliver Gondža
            issac1993 Daniel Issac
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: