Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31866

Attempts to restrict Pipeline jobs from running on master result in job hanging

      Trying the job-restrictions-plugin with Workflow jobs - I added a restriction to master that only jobs matching a regex could run on there. At first glance, that looked perfect - a Workflow job that didn't match that regex stayed in queue while another Workflow job that did match ran fine.

      But then I realized that the non-matching job was still sitting in queue, even after another executor not on the master was available. The job is just stuck - it's not even getting to the node { }, etc. If it can't run on master (when master has executors), it just flops around. Ow.

          [JENKINS-31866] Attempts to restrict Pipeline jobs from running on master result in job hanging

          Andrew Bayer added a comment -

          And cc danielbeck

          Andrew Bayer added a comment - And cc danielbeck

          Oleg Nenashev added a comment -

          Well, Job Restrictions are still do not officially support Pipeline. I've made some progress in JENKINS-38644, which works around the blockage cause. There is also a patch for JENKINS-36626, which addresses some extra cases.

          The work still needs to be done.

          Oleg Nenashev added a comment - Well, Job Restrictions are still do not officially support Pipeline. I've made some progress in JENKINS-38644 , which works around the blockage cause. There is also a patch for JENKINS-36626 , which addresses some extra cases. The work still needs to be done.

          Greg Smith added a comment -

          I recently ran into this problem too:

          It seems that the webhook that causes the build to be run – which then kicks off the scan of the repo – runs on master.

          So say you have a github org definition of "MyCompany" and all projects are under that, like "MyCompany/aproject/master"

          Now you set the restrictions so that "master will not run any builds under 'MyCompany'" with a regex like:
          ^(?!MyCompany\/)

          That successfully stops any of those builds from using a node("master") – but it also stops the incoming webhook from running a scan – so all builds in that github org "hang"

          This plugin needs some kind of special allowance for the scan, or maybe some way to define that scans are OK.

          Greg Smith added a comment - I recently ran into this problem too: It seems that the webhook that causes the build to be run – which then kicks off the scan of the repo – runs on master. So say you have a github org definition of "MyCompany" and all projects are under that, like "MyCompany/aproject/master" Now you set the restrictions so that "master will not run any builds under 'MyCompany'" with a regex like: ^(?!MyCompany\/) That successfully stops any of those builds from using a node("master") – but it also stops the incoming webhook from running a scan – so all builds in that github org "hang" This plugin needs some kind of special allowance for the scan, or maybe some way to define that scans are OK.

          Oleg Nenashev added a comment -

          gregcovertsmith as designed in the current implementation. You will have to use a more complex Restriction expression in order to allow such task classes (I've added Job Class Restriction for such purpose in 0.6).

          I would be happy if somebody contributes a better solution

          Oleg Nenashev added a comment - gregcovertsmith as designed in the current implementation. You will have to use a more complex Restriction expression in order to allow such task classes (I've added Job Class Restriction for such purpose in 0.6). I would be happy if somebody contributes a better solution

          Oleg Nenashev added a comment -

          abayer jglick

          I think we should just close it as "as designed". Here is a configuration example for Job Restrictions, which enables Pipeline builds: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/blob/master/init_scripts/src/main/groovy/MasterComputer.groovy#L29-L42

          Oleg Nenashev added a comment - abayer jglick I think we should just close it as "as designed". Here is a configuration example for Job Restrictions, which enables Pipeline builds: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/blob/master/init_scripts/src/main/groovy/MasterComputer.groovy#L29-L42

          oleg_nenashev The link to your example is broken. I think it has moved here: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/blob/master/init_scripts/src/main/groovy/scripts/MasterComputer.groovy

          To be specific, is the class restriction to add "org.jenkinsci.plugins.workflow.job.WorkflowJob" to allow the light-weight executors of Pipeline jobs to run on the Jenkins Master node?

          Claes Buckwalter added a comment - oleg_nenashev The link to your example is broken. I think it has moved here: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/blob/master/init_scripts/src/main/groovy/scripts/MasterComputer.groovy To be specific, is the class restriction to add "org.jenkinsci.plugins.workflow.job.WorkflowJob" to allow the light-weight executors of Pipeline jobs to run on the Jenkins Master node?

          Oleg Nenashev added a comment -

          Sorry, missed the comment. Yes, this is a change which makes the restriction to work properly.
          There is also JENKINS-46652 which fixes it on the core's side. I think the ticket can be closed

          Oleg Nenashev added a comment - Sorry, missed the comment. Yes, this is a change which makes the restriction to work properly. There is also JENKINS-46652 which fixes it on the core's side. I think the ticket can be closed

          Sam Gleske added a comment - - edited

          Question about your class filter configuration.

          Can normal pipeline users arbitrarily execute code on the master node if a user defines in their Jenkinsfile the following?

          node('master') {
              //user does something malicious on master
          }

          I basically want to reserve the master executors to be available to admins only.

          Sam Gleske added a comment - - edited Question about your class filter configuration. Can normal pipeline users arbitrarily execute code on the master node if a user defines in their Jenkinsfile the following? node('master') { //user does something malicious on master } I basically want to reserve the master executors to be available to admins only.

          Sam Gleske added a comment -

          I ran a quick test. Looks like the solution works. Does not allow users to define master in their Jenkinsfile. The pipeline runs but hangs on the master node step indefinitely. Looks good to me. A build timeout could cancel hung jobs. In the future it would be nice if pipelines were officially supported.

          Some support could include:

          • JENKINS-50644 - allow admins to define master node steps in global shared libraries.
          • Throw a security warning to the admin when a user attempts to run on the master node via a Jenkinsfile.
          • Prevent the job from ever starting so it doesn't hang indefinitely taking up resources. It could just kill the job as a failure.

          Sam Gleske added a comment - I ran a quick test. Looks like the solution works. Does not allow users to define master in their Jenkinsfile. The pipeline runs but hangs on the master node step indefinitely. Looks good to me. A build timeout could cancel hung jobs. In the future it would be nice if pipelines were officially supported. Some support could include: JENKINS-50644 - allow admins to define master node steps in global shared libraries. Throw a security warning to the admin when a user attempts to run on the master node via a Jenkinsfile. Prevent the job from ever starting so it doesn't hang indefinitely taking up resources. It could just kill the job as a failure.

          Sam Gleske added a comment -

          For multibranch pipelines I had to whitelist org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject in addition to org.jenkinsci.plugins.workflow.job.WorkflowJob.

          Sam Gleske added a comment - For multibranch pipelines I had to whitelist org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject in addition to org.jenkinsci.plugins.workflow.job.WorkflowJob .

          Sam Gleske added a comment - - edited

          Sam Gleske added a comment - - edited Here's my script https://github.com/samrocketman/jenkins-bootstrap-shared/blob/21f76bb94f89dbbd56e8697482c861266c2b456a/scripts/configure-job-restrictions-master.groovy

          Sam Gleske added a comment - - edited

          Pipelines now being a core feature of Jenkins, I recommend keeping this issue open until pipelines are natively supported without any additional configuration on the user.

          Sam Gleske added a comment - - edited Pipelines now being a core feature of Jenkins, I recommend keeping this issue open until pipelines are natively supported without any additional configuration on the user.

          This solution is great!  How do I configure a build timeout to cancel the hung job?

          michael jackson added a comment - This solution is great!  How do I configure a build timeout to cancel the hung job?

          Sam Gleske added a comment -

          You could run a nightly system groovy script (Groovy plugin) which kills all jobs which have taken longer than 24 hours. Here's an example script https://github.com/samrocketman/jenkins-script-console-scripts/blob/master/kill-all-day-old-builds.groovy

          Sam Gleske added a comment - You could run a nightly system groovy script (Groovy plugin) which kills all jobs which have taken longer than 24 hours. Here's an example script https://github.com/samrocketman/jenkins-script-console-scripts/blob/master/kill-all-day-old-builds.groovy

          Great idea!  Thanks.

          michael jackson added a comment - Great idea!  Thanks.

          Hi,

          i think this is not a minor issue. Because of reasons I don't want to run pipelines on master node, I want to run it on slave. Piepeline is always waiting for a node master, no matter if I specify.

          pipeline

          {     agent <label> ... }

           or 

           

          pipeline

          {     agent any ... }

           

          I found no simple solution for this. Is there any?

           

          Martin Cimbalek added a comment - Hi, i think this is not a minor issue. Because of reasons I don't want to run pipelines on master node, I want to run it on slave. Piepeline is always waiting for a node master, no matter if I specify. pipeline {     agent <label> ... }  or    pipeline {     agent any ... }   I found no simple solution for this. Is there any?  

          Artur Szostak added a comment -

          A possible workaround: Use the "Job class" restriction with "org.jenkinsci.plugins.workflow.job.WorkflowJob" for the class name in a "Or" combination with other restriction expressions.

          In my particular case I want to reserve the executors on the master to jobs owned by an account called jenkins and for jobs from an Admin folder. So I have the configuration as indicated in the following image:

          Note that apparently one can ignore the warning indicated in the GUI. Its yet another Jenkins bug. Things seem to work in my case anyway.

          Artur Szostak added a comment - A possible workaround: Use the "Job class" restriction with "org.jenkinsci.plugins.workflow.job.WorkflowJob" for the class name in a "Or" combination with other restriction expressions. In my particular case I want to reserve the executors on the master to jobs owned by an account called jenkins and for jobs from an Admin folder. So I have the configuration as indicated in the following image: Note that apparently one can ignore the warning indicated in the GUI. Its yet another Jenkins bug. Things seem to work in my case anyway.

          Alexandre Martel added a comment - - edited

          Pipeline dispatch to slave node seems to always execute in the master first. The workaround seems to create a local slave node for jobs that should run on the master machine and never use the master node for running any jobs. I set the master node usage to "Only builds jobs with label expressions matching this node".

          But i think that dispatching pipeline job to nodes shouldn't be restricted by the master number of executor.

          Alexandre Martel added a comment - - edited Pipeline dispatch to slave node seems to always execute in the master first. The workaround seems to create a local slave node for jobs that should run on the master machine and never use the master node for running any jobs. I set the master node usage to "Only builds jobs with label expressions matching this node". But i think that dispatching pipeline job to nodes shouldn't be restricted by the master number of executor.

          Sam Gleske added a comment - - edited

          Pipelines orchestrate from the master runtime. There's nothing you can change that. You can set the master executors to 0 and it will still be running the pipeline orchestration within the master.

          In general, a major concern for me was users abusing the master by adding to their Jenkinsfiles.

          node('master') {
              sh 'im-a-bad-person.sh'
          }
          

          However, with suggestions from this thread one can easily restrict the master by only allowing the Jenkins pipeline orchestration classes (and not the WorkflowRun class which is the actual pipeline build that runs on agents). sdalex check out some of my earlier comments in this issue. I share some of my scripts and methods for handling pipelines and cleanup as well as automatically configuring.

          Sam Gleske added a comment - - edited Pipelines orchestrate from the master runtime. There's nothing you can change that. You can set the master executors to 0 and it will still be running the pipeline orchestration within the master. In general, a major concern for me was users abusing the master by adding to their Jenkinsfiles. node('master') { sh 'im-a-bad-person.sh' } However, with suggestions from this thread one can easily restrict the master by only allowing the Jenkins pipeline orchestration classes (and not the WorkflowRun class which is the actual pipeline build that runs on agents). sdalex check out some of my earlier comments in this issue. I share some of my scripts and methods for handling pipelines and cleanup as well as automatically configuring.

          Sam Gleske added a comment -

          aszostak I get that warning too. The primary reason for the error is the class loader for plugins is only populated with related plugins in which it depends. Since the job-restrictions plugin does not directly depend on pipeline plugins, the class loader for it does not include the workflow set of classes. You're right that you can ignore the error and it will "just work". Can probably be considered a bug. It should probably be using the uber classloader which exists in the plugin manager.

          Sam Gleske added a comment - aszostak I get that warning too. The primary reason for the error is the class loader for plugins is only populated with related plugins in which it depends. Since the job-restrictions plugin does not directly depend on pipeline plugins, the class loader for it does not include the workflow set of classes. You're right that you can ignore the error and it will "just work". Can probably be considered a bug. It should probably be using the uber classloader which exists in the plugin manager.

          Sam Gleske added a comment -

          aszostak I opened a PR addressing the warning you experienced https://github.com/jenkinsci/job-restrictions-plugin/pull/23

          Sam Gleske added a comment - aszostak I opened a PR addressing the warning you experienced https://github.com/jenkinsci/job-restrictions-plugin/pull/23

          Artur Szostak added a comment -

          Thank you for the warning fix. Looking forward to seeing the warning disappear in some future upgrade of our Jenkins instance.

          Artur Szostak added a comment - Thank you for the warning fix. Looking forward to seeing the warning disappear in some future upgrade of our Jenkins instance.

          Adrian Vlad added a comment -

          I have a similar problem. I have a pipeline called "test_pipeline" and inside it does:

          node('some_node') {
          ...
          }

          On "some_node" I use Restrict jobs execution at node -> Regular Expression (Job Name) with value test_pipeline.

          This configuration doesn't work (the build hangs). But if I change the value to test_pipeline.* then it works.

          What is the problem? Is job-restrictions using different pipeline names with some suffix? (test_pipeline/.* also works)

          Adrian Vlad added a comment - I have a similar problem. I have a pipeline called "test_pipeline" and inside it does: node( 'some_node' ) { ... } On "some_node" I use  Restrict jobs execution at node ->  Regular Expression (Job Name) with value  test_pipeline . This configuration doesn't work (the build hangs). But if I change the value to  test_pipeline.* then it works. What is the problem? Is  job-restrictions using different pipeline names with some suffix? ( test_pipeline/.* also works)

          Oleg Nenashev added a comment -

          As stated in https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ , I have no bandwidth to work on this issue anytime soon. I have marked the plugin for adoption. But I will be happy to review/release anything related to this issue while there is no other maintainer.

           

          Oleg Nenashev added a comment - As stated in https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ , I have no bandwidth to work on this issue anytime soon. I have marked the plugin for adoption. But I will be happy to review/release anything related to this issue while there is no other maintainer.  

            Unassigned Unassigned
            abayer Andrew Bayer
            Votes:
            11 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated: