Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-3099

ci.jenkins.io: Only use High Memory instance when it's needed

    XMLWordPrintable

    Details

    • Similar Issues:
    • Epic Link:

      Description

      Why

      • Read the EPIC (AWS cost decrease)
      • shorter feedback loop for Jenkins Core PRs

      What

      The high memory instances VMs are beefy machines that are expensive on ci.jenkins.io

      As for today, when a job requests a agent with labels `docker`, `linux` or a combination of these keywords, the high meme instance can be picked as well as "standard sized" VM instances.

      The reason is because the high-mem instances also have these labels.

      This issue tracks the work related to fixing this behavior, to decrease cost and allow big builds such as ATH or core PR to benefit from more available executors (instead of being queued in competition with smaller jobs).

      Another consequence is that the "docker" jobs are hitting the API rate limit when being run accidentally by a highmem instance, as these costly instances are kept around for reusability comapred to other instances

      How

      Proposal to fix is the following:

      Please note that, in short term, we can update jenkins-infra/jenkins.io and jenkinsci/docker* from "docker && linux" to "docker && linux && !highmem" (ping Mark WaiteTim Jacomb)

        Attachments

          Activity

          Hide
          hlemeur Hervé Le Meur added a comment - - edited

          Next:

          • add 'docker-windows' label (instead of 'windock'), then use this one in the different Jenkinfiles, finally remove 'windock' label
          • remove the labels 'docker', 'linux' and 'java' on hieradata highmem machines
          • check if AWS has a machine size closer to the Azure one:
          Show
          hlemeur Hervé Le Meur added a comment - - edited Next: add 'docker-windows' label (instead of 'windock'), then use this one in the different Jenkinfiles, finally remove 'windock' label remove the labels 'docker', 'linux' and 'java' on hieradata highmem machines check if AWS has a machine size closer to the Azure one: the T3Xlarge AWS machine has the same vCPU count as the Azure one, it's only the comments in hieradata which are wrong https://github.com/jenkins-infra/jenkins-infra/pull/1938
          Show
          hlemeur Hervé Le Meur added a comment - PRs for docker-windows label: https://github.com/jenkins-infra/jenkins-infra/pull/1936 https://github.com/jenkinsci/jnlp-agents/pull/29 https://github.com/jenkinsci/lib-durable-task/pull/25 https://github.com/jenkinsci/docker-ssh-agent/pull/88 https://github.com/jenkinsci/docker-inbound-agent/pull/245 https://github.com/jenkinsci/docker-agent/pull/209 https://github.com/jenkinsci/docker/pull/1222 https://github.com/jenkins-infra/charts/pull/1575 https://github.com/jenkins-infra/documentation/pull/19
          Hide
          hlemeur Hervé Le Meur added a comment -

          PR to remove 'windock' label from hieradata: https://github.com/jenkins-infra/jenkins-infra/pull/1939

          Show
          hlemeur Hervé Le Meur added a comment - PR to remove 'windock' label from hieradata: https://github.com/jenkins-infra/jenkins-infra/pull/1939
          Hide
          hlemeur Hervé Le Meur added a comment - - edited

          Draft PR to remove 'docker', 'linux' and 'java' labels on hieradata highmem machines: https://github.com/jenkins-infra/jenkins-infra/pull/1940

          (On hold until related issues are resolved)

          Show
          hlemeur Hervé Le Meur added a comment - - edited Draft PR to remove 'docker', 'linux' and 'java' labels on hieradata highmem machines: https://github.com/jenkins-infra/jenkins-infra/pull/1940 (On hold until related issues are resolved)
          Hide
          dduportal Damien Duportal added a comment -

          Last PR merged and approved \o/

          Show
          dduportal Damien Duportal added a comment - Last PR merged and approved \o/

            People

            Assignee:
            hlemeur Hervé Le Meur
            Reporter:
            dduportal Damien Duportal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: