Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65169

Update Mesos Plugin to be compliant with agent naming requirements in Jenkins 2.263.2+

    XMLWordPrintable

Details

    Description

      Mesos cloud labels can contain the char ":" in order to specify custom docker images (which also contain the char ":" for the docker image tag separator)

       

      Was working on LTS 2.263.1 but not 2.277.1, hitting a SEVERE error with stack trace:

       

       2021-03-18 16:02:45.512+0000 [id=66]	SEVERE	hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected uncaught exception encountered while processing agent MesosCloud
      hudson.model.Failure: ‘:’ is an unsafe character
      	at jenkins.model.Jenkins.checkGoodName(Jenkins.java:4063)
      	at jenkins.model.Nodes.addNode(Nodes.java:139)
      	at jenkins.model.Jenkins.addNode(Jenkins.java:2157)
      	at hudson.slaves.NodeProvisioner.lambda$update$6(NodeProvisioner.java:256)
      	at hudson.model.Queue._withLock(Queue.java:1384)
      	at hudson.model.Queue.withLock(Queue.java:1261)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:225)
      	at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:64)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
      	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)

       

      example label that is used in the mesos config, and the agent fails to start:

      worker:REDACTEDREPO/qe/jira-dependency-graph:1

       

       

      Attachments

        Activity

          sbeaulie Samuel Beaulieu created issue -

          looking at the code I'll try to set

          jenkins.model.Nodes.enforceNameRestrictions=false

          sbeaulie Samuel Beaulieu added a comment - looking at the code I'll try to set jenkins.model.Nodes.enforceNameRestrictions=false
          sbeaulie Samuel Beaulieu added a comment - https://github.com/jenkinsci/jenkins/blame/master/core/src/main/java/jenkins/model/Jenkins.java#L4057
          sbeaulie Samuel Beaulieu made changes -
          Field Original Value New Value
          Priority Critical [ 2 ] Major [ 3 ]
          sbeaulie Samuel Beaulieu made changes -
          Labels SECURITY-2021
          sbeaulie Samuel Beaulieu made changes -
          Link This issue is caused by SECURITY-2021 [ SECURITY-2021 ]
          oleg_nenashev Oleg Nenashev added a comment -

          Another issue to document in upgrade guidelines, right markewaite danielbeck wfollonier?

          oleg_nenashev Oleg Nenashev added a comment - Another issue to document in upgrade guidelines, right markewaite danielbeck wfollonier ?
          danielbeck Daniel Beck added a comment - - edited

          Yes, in the upgrade guide for 2.263.2, because this has been around for more than two months now: https://www.jenkins.io/security/advisory/2021-01-13/#SECURITY-2021

          My recommendation would be to use different labels that result in safe agent names instead. Based on my understanding of this issue, this is a cloud configuration side issue. Disabling the fix should only be a last resort. This is how I would document this as well.

           

          danielbeck Daniel Beck added a comment - - edited Yes, in the upgrade guide for 2.263.2, because this has been around for more than two months now: https://www.jenkins.io/security/advisory/2021-01-13/#SECURITY-2021 My recommendation would be to use different labels that result in safe agent names instead. Based on my understanding of this issue, this is a cloud configuration side issue. Disabling the fix should only be a last resort. This is how I would document this as well.  
          markewaite Mark Waite made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          markewaite Mark Waite made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          markewaite Mark Waite added a comment - - edited

          Submitted a documentation pull request to add more description to the security hardening that was applied to Jenkins 2,263.2 for label names.

          markewaite Mark Waite added a comment - - edited Submitted a documentation pull request to add more description to the security hardening that was applied to Jenkins 2,263.2 for label names.
          markewaite Mark Waite made changes -
          Remote Link This issue links to "PR-4196 to extend upgrade guide description of label name security hardening (Web Link)" [ 26582 ]
          danielbeck Daniel Beck added a comment -

          This is the result of a deliberate change that now has been documented in the upgrade guide at https://www.jenkins.io/doc/upgrade-guide/2.263/#SECURITY-2021

          We recommend you change your agent naming scheme to no longer include characters that would not be legal in job names either.

          Closing as Won't Fix.

          danielbeck Daniel Beck added a comment - This is the result of a deliberate change that now has been documented in the upgrade guide at https://www.jenkins.io/doc/upgrade-guide/2.263/#SECURITY-2021 We recommend you change your agent naming scheme to no longer include characters that would not be legal in job names either. Closing as Won't Fix.
          danielbeck Daniel Beck made changes -
          Resolution Won't Fix [ 2 ]
          Status In Review [ 10005 ] Closed [ 6 ]

          FWIW thats not possible without a change to the mesos cloud plugin.

           

          Again the issue is not about setting a label name and making sure no illegal chars are in it, but rather a functionality of the plugin which enables you to run the mesos workload on a "custom docker image". In order to do that you specifyin this format: [LABEL]:[DOCKERIMAGE]

           

          So while I could keep the LABEL free of illegal chars, you have to separate it with a colon, and then the DOCKERIMAGE name also usualy includes a tag, which is also separated by a colon eg. ruby:3.0.0-buster otherwise it pulls to latest tag only. In production I have for example labels in the job equal to production:ruby:3.0.0-buster and the plugin finds the "production" label, then pulls the custom image for the workload.

           

          It would also be useful for newcomers if the form validation for label names would trigger a red message that the label name is invalid.

          sbeaulie Samuel Beaulieu added a comment - FWIW thats not possible without a change to the mesos cloud plugin.   Again the issue is not about setting a label name and making sure no illegal chars are in it, but rather a functionality of the plugin which enables you to run the mesos workload on a "custom docker image". In order to do that you specifyin this format: [LABEL] : [DOCKERIMAGE]   So while I could keep the LABEL free of illegal chars, you have to separate it with a colon, and then the DOCKERIMAGE name also usualy includes a tag, which is also separated by a colon eg. ruby:3.0.0-buster otherwise it pulls to latest tag only. In production I have for example labels in the job equal to production:ruby:3.0.0-buster and the plugin finds the "production" label, then pulls the custom image for the workload.   It would also be useful for newcomers if the form validation for label names would trigger a red message that the label name is invalid.
          danielbeck Daniel Beck added a comment -

          It would also be useful for newcomers if the form validation for label names would trigger a red message that the label name is invalid.

          Unfortunately that form is provided by the plugin so this is not something we could have done in core And with 2000+ plugins it's impossible to even keep track of them when we prepare a fix, much less adapt them to work nicely.


          In the short term, the mesos plugin needs to be changed, or affected users need to apply the escape hatch. This is not ideal, but it is where we are.

          Mesos plugin could use names with a more basic character set to pass the new restriction, and implement Computer#getDisplayName to implement a richer name to show on the UI.

          danielbeck Daniel Beck added a comment - It would also be useful for newcomers if the form validation for label names would trigger a red message that the label name is invalid. Unfortunately that form is provided by the plugin so this is not something we could have done in core And with 2000+ plugins it's impossible to even keep track of them when we prepare a fix, much less adapt them to work nicely. In the short term, the mesos plugin needs to be changed, or affected users need to apply the escape hatch. This is not ideal, but it is where we are. Mesos plugin could use names with a more basic character set to pass the new restriction, and implement Computer#getDisplayName to implement a richer name to show on the UI.
          oleg_nenashev Oleg Nenashev made changes -
          Component/s mesos-plugin [ 18122 ]
          Component/s core [ 15593 ]
          oleg_nenashev Oleg Nenashev made changes -
          Summary SEVERE error Cloud agent not starting because of checkGoodname() regression in 2.277.1 Update Mesos Plugin to be compliant with agent naming requirements in checkGoodname() regression in 2.277.1
          oleg_nenashev Oleg Nenashev made changes -
          Summary Update Mesos Plugin to be compliant with agent naming requirements in checkGoodname() regression in 2.277.1 Update Mesos Plugin to be compliant with agent naming requirements in Jenkins 2.263.2+
          oleg_nenashev Oleg Nenashev added a comment -

          Reopening the issue as a Mesos Plugin one.

           

          oleg_nenashev Oleg Nenashev added a comment - Reopening the issue as a Mesos Plugin one.  
          oleg_nenashev Oleg Nenashev made changes -
          Resolution Won't Fix [ 2 ]
          Status Closed [ 6 ] Reopened [ 4 ]

          People

            Unassigned Unassigned
            sbeaulie Samuel Beaulieu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: