Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36919

Docker Instance Capacity counted across templates

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: docker-plugin
    • Labels:
      None
    • Environment:
      SUSE Linux Enterprise 12 SP1
      jdk1.8.0_60
      Jenkins ver. 1.635
      apache tomcat 7.0.65
      docker-plugin 0.16.0
    • Similar Issues:

      Description

      Hi folks,

      after some detailed investigations I am quite sure that we have an issue with counting the running slaves on the docker-plugin with multiple templates. Here's how you may reproduce it:

      1. Setup a Jenkins with docker-plugin
      2. Configure a cloud type 'docker', put the Capacity limit to a high value (let's say 50 or so)
      3. Configure two templates: A and B.
      4. Set for template A to accept label A
      5. Set for template B to accept label B
      6. Use the same image (some simple image)
      7. Set instance limit of template A to 5
      8. Set instance limit of template B to 2
      9. Create 10 jobs assigned to label A, implementation "sleep 60"
      10. Create 5 jobs assigned to label B, implementation "sleep 60"
      11. Start all the jobs of label A at once (you may run a small groovy script for that)
      12. Wait 10s
      13. Start all the jobs of label B at once (you may run a small groovy script for that)

      What you will observe is the following:

      • The jobs of label A will request new slaves up to the instance limit.
      • Jobs with label B will remain in the queue. No new slaves are created
      • The Jobs with label B will be executed, once all the jobs in label A have been completed (and the slaves are taken offline again)

      If you look into the system log you will read the following messages:

      Asked to provision 23 slave(s) for: labelB
      Jul 25, 2016 4:46:52 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
      Will provision 'image', for label: 'labelB', in cloud: 'docker'
      Jul 25, 2016 4:46:52 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
      Not Provisioning 'labelB'. Instance limit of '2' reached on server 'docker'

      Please note that during that error message, not a single slave of the second template was up and running (however, 5 of the first template were up).

      Repeat the same activity with setting the instance limit of template B to 6. Repeat the same kind of load. You will observe that exactly one slave of template B will be created.

      Alas: The instance limits of two different templates are not counted separately (which is what the configuration UI suggests).

      Impact: Though capacity is available on the docker server, the different loads are not executed in parallel.

      PS: I also tried to configure a second cloud provider (of type docker), thus separating the templates into two sections. However, this did not change the situation either: Apparently, the "instances used" are counted per URL and not per template...

      Thanks for checking!

        Attachments

          Activity

          Hide
          eagle_rainbow Nico Schmoigl added a comment -

          For a proposal how to fix this issue, see also https://github.com/jenkinsci/docker-plugin/pull/409

          Show
          eagle_rainbow Nico Schmoigl added a comment - For a proposal how to fix this issue, see also https://github.com/jenkinsci/docker-plugin/pull/409
          Hide
          ndeloof Nicolas De Loof added a comment -

          countCurrentDockerSlaves is based on identifying container based on image running, which was the sole option when docker was 0.x and this plugin has been created, but in 2017 is well better implemented using labels. 

          Show
          ndeloof Nicolas De Loof added a comment - countCurrentDockerSlaves is based on identifying container based on image running, which was the sole option when docker was 0.x and this plugin has been created, but in 2017 is well better implemented using labels. 
          Hide
          ndeloof Nicolas De Loof added a comment -

          unfortunately docker-java implementation for label filters (com.github.dockerjava.core.util.FiltersEncoder) relies on JacksonJaxbJsonProvider and here comes dependency hell :'(

           

          Show
          ndeloof Nicolas De Loof added a comment - unfortunately docker-java implementation for label filters (com.github.dockerjava.core.util.FiltersEncoder) relies on JacksonJaxbJsonProvider and here comes dependency hell :'(  
          Hide
          eagle_rainbow Nico Schmoigl added a comment -

          What about the (ugly) idea to fetch all containers, and filtering the containers then manually by its labels? Or would retrieving of all containers each time be too costly...?

          NB PR #409  was closed (and rejected) by you some time back... Should we revisit the suggestion there once more?

           

          Show
          eagle_rainbow Nico Schmoigl added a comment - What about the (ugly) idea to fetch all containers, and filtering the containers then manually by its labels? Or would retrieving of all containers each time be too costly...? NB PR #409  was closed (and rejected) by you some time back... Should we revisit the suggestion there once more?  
          Hide
          pjdarton pjdarton added a comment -

          FYI: I coded a workaround to the FiltersEncoder issue mentioned above, albeit by using a local definition of that code in the plugin. This is basically a bug in docker-java, and it's not limited to just that class, but I'm not convinced it's worthwhile trying to fix docker-java.

          I spent a bit of time looking into this myself (but for other reasons).
          The cause of the problem is that the plugin is using the template's image name as a means of identifying the template, but the image name is not unique.
          What we need is to give each template its own unique name.
          We could then set a label on each container we start specifying the "template name" so that we can recognize them later.
          Lastly, we change the `countCurrentDockerSlaves()` code to use the "template name" instead of "image name".

          Note: Ideally, we'd also name the docker slaves (both in Jenkins and in docker) after that name too, instead of calling them all "docker-.............." - I have multiple docker hosts and templates that I'd like to differentiate by name, which is why I started looking into this.

          Show
          pjdarton pjdarton added a comment - FYI: I coded a workaround to the FiltersEncoder issue mentioned above , albeit by using a local definition of that code in the plugin. This is basically a bug in docker-java, and it's not limited to just that class, but I'm not convinced it's worthwhile trying to fix docker-java. I spent a bit of time looking into this myself (but for other reasons). The cause of the problem is that the plugin is using the template's image name as a means of identifying the template, but the image name is not unique. What we need is to give each template its own unique name. We could then set a label on each container we start specifying the "template name" so that we can recognize them later. Lastly, we change the `countCurrentDockerSlaves()` code to use the "template name" instead of "image name". Note: Ideally, we'd also name the docker slaves (both in Jenkins and in docker) after that name too, instead of calling them all "docker-.............." - I have multiple docker hosts and templates that I'd like to differentiate by name, which is why I started looking into this.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            eagle_rainbow Nico Schmoigl
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: