Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15094

Parameters: NPE in canTake() procedures may kill all executors

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Fixed
    • core
    • Jenkins ver. 1.466.1
      Linux buildserver 2.6.18-164.11.1.el5PAE #1 SMP Wed Jan 6 13:43:57 EST 2010 i686 i686 i386 GNU/Linux

    Description

      Jenkins executors are not robust against nulls returned by createValue() and createDefaultValue(). Such return value is valid according to Javadoc.

      NPE in canTake() procedure kills the executor thread. Unfortunately, Jenkins tries other executors after that => any NPE in canTake() terminates ALL active Jenkins executors.

      The initial description from Rotem:

      when using jenkins-multijob-plugin and parameter from type “File Parameter” the Jenkins crashes.
      When i start to run a Multijob with a multijob phase with values in "Advanced: "parameters", if the next job has a parameter from type “File Parameter” Jenkins is eating up all the executers with - ALL available Build executors are turning into zombies with the status 'Dead ' with the exception:

      java.lang.NullPointerException
      at hudson.model.ParametersAction.getAssignedLabel(ParametersAction.java:126)
      at hudson.model.Queue$Item.getAssignedLabel(Queue.java:1265)
      at hudson.model.Node.canTake(Node.java:308)
      at hudson.model.Queue$JobOffer.canTake(Queue.java:210)
      at hudson.model.Queue.maintain(Queue.java:952)
      at hudson.model.Queue.pop(Queue.java:783)
      at hudson.model.Executor.grabJob(Executor.java:287)
      at hudson.model.Executor.run(Executor.java:208)

      Even if at this stage I go to the settings and enlarge the number of Executors, they immediately turn into zombies as well.
      When I remove the specific “File Parameter” parameter all works fine.
      Please advice.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-22925 [Custom Tools] - Parameter definition does not return the default value

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-11541 NPE in Throttle Concurrent Builds Plugin kills the executor

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-15036 [jenkins-multijob-plugin] Multijob Freeze after first phase and all Executors are becoming Dead (!)

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            hagzag hagzag added a comment -

            Rotem thanks
            Our team is checking this issue.

            hagzag hagzag added a comment - Rotem thanks Our team is checking this issue.
            hagzag hagzag added a comment -

            Could you please assign this to someone ASAP - looks like a huge show stopper ...

            hagzag hagzag added a comment - Could you please assign this to someone ASAP - looks like a huge show stopper ...
            doronshai Doron Shai added a comment -

            This issue is a real pain since when using the MultiJob Plugin there is no real way to use the Parametrized Build Plugin and therefore passing parameters from MultiJob Phase to another MultiJob Phase is not possible.

            doronshai Doron Shai added a comment - This issue is a real pain since when using the MultiJob Plugin there is no real way to use the Parametrized Build Plugin and therefore passing parameters from MultiJob Phase to another MultiJob Phase is not possible.
            oleg_nenashev Oleg Nenashev added a comment -

            I see the issue after the failed logrotate procedure

            oleg_nenashev Oleg Nenashev added a comment - I see the issue after the failed logrotate procedure
            oleg_nenashev Oleg Nenashev added a comment - - edited

            The issue is inside the core

            • ParameterDefinition:createValue() can return null value according to Javadoc.
            • BuildCommand::run() creates a list of parameter values without null checks
            • hudson.model.ParametersAction.getAssignedLabel() iterates parameter values, but there is no null checks

            Any null parameter value in the task (e.g. in CLI call) may lead to the complete DoS of the service. Marked the issue as a blocker

            oleg_nenashev Oleg Nenashev added a comment - - edited The issue is inside the core ParameterDefinition:createValue() can return null value according to Javadoc. BuildCommand::run() creates a list of parameter values without null checks hudson.model.ParametersAction.getAssignedLabel() iterates parameter values, but there is no null checks Any null parameter value in the task (e.g. in CLI call) may lead to the complete DoS of the service. Marked the issue as a blocker
            oleg_nenashev Oleg Nenashev added a comment - - edited

            In my case the issue has been caused by JENKINS-22925, but there're many other possible failure causes.
            Created a PR to the core: https://github.com/jenkinsci/jenkins/pull/1229

            I'll also investigate & re-submit the initial bug to another issue.

            oleg_nenashev Oleg Nenashev added a comment - - edited In my case the issue has been caused by JENKINS-22925 , but there're many other possible failure causes. Created a PR to the core: https://github.com/jenkinsci/jenkins/pull/1229 I'll also investigate & re-submit the initial bug to another issue.
            oleg_nenashev Oleg Nenashev added a comment -

            Updated the issue's description. Probably, workaround in jenkins-multijob-plugin would be useful as well

            oleg_nenashev Oleg Nenashev added a comment - Updated the issue's description. Probably, workaround in jenkins-multijob-plugin would be useful as well
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/com/synopsys/arc/jenkinsci/plugins/customtools/versions/ToolVersionParameterDefinition.java
            src/test/java/com/cloudbees/jenkins/plugins/customtools/CustomToolInstallerTest.java
            src/test/java/com/synopsys/arc/jenkins/plugins/customtools/util/CLICommandInvoker.java
            src/test/java/com/synopsys/arc/jenkins/plugins/customtools/versions/ToolVersionParameterDefinitionTest.java
            http://jenkins-ci.org/commit/customtools-plugin/317adc58e636548e5f3619d74ce1fbed7ff95d3c
            Log:
            [FIXED JENKINS-22925] - Properly return default values to avoid JENKINS-15094

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/com/synopsys/arc/jenkinsci/plugins/customtools/versions/ToolVersionParameterDefinition.java src/test/java/com/cloudbees/jenkins/plugins/customtools/CustomToolInstallerTest.java src/test/java/com/synopsys/arc/jenkins/plugins/customtools/util/CLICommandInvoker.java src/test/java/com/synopsys/arc/jenkins/plugins/customtools/versions/ToolVersionParameterDefinitionTest.java http://jenkins-ci.org/commit/customtools-plugin/317adc58e636548e5f3619d74ce1fbed7ff95d3c Log: [FIXED JENKINS-22925] - Properly return default values to avoid JENKINS-15094 Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/test/java/hudson/model/ParametersActionTest.java
            test/src/test/groovy/hudson/cli/BuildCommandTest.groovy
            http://jenkins-ci.org/commit/jenkins/cd23fac9950c1c0762731372df2c109d4bb80db3
            Log:
            JENKINS-15094 - Unit tests for the issue

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/test/java/hudson/model/ParametersActionTest.java test/src/test/groovy/hudson/cli/BuildCommandTest.groovy http://jenkins-ci.org/commit/jenkins/cd23fac9950c1c0762731372df2c109d4bb80db3 Log: JENKINS-15094 - Unit tests for the issue Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/cli/BuildCommand.java
            core/src/main/java/hudson/model/ParameterDefinition.java
            core/src/main/java/hudson/model/ParametersAction.java
            core/src/main/java/hudson/model/ParametersDefinitionProperty.java
            http://jenkins-ci.org/commit/jenkins/7b894f57b78be0e4d847fe1c971955d2cbb08e58
            Log:
            [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI).

            The change adds missing checks for null ParameterValue, which may be passed from the CLI.
            It prevents the death of Jenkins executors during the canTake() check of node labels.

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/cli/BuildCommand.java core/src/main/java/hudson/model/ParameterDefinition.java core/src/main/java/hudson/model/ParametersAction.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java http://jenkins-ci.org/commit/jenkins/7b894f57b78be0e4d847fe1c971955d2cbb08e58 Log: [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI). The change adds missing checks for null ParameterValue, which may be passed from the CLI. It prevents the death of Jenkins executors during the canTake() check of node labels. Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            core/src/main/java/hudson/cli/BuildCommand.java
            core/src/main/java/hudson/model/ParameterDefinition.java
            core/src/main/java/hudson/model/ParametersAction.java
            core/src/main/java/hudson/model/ParametersDefinitionProperty.java
            core/src/test/java/hudson/model/ParametersActionTest.java
            test/src/test/groovy/hudson/cli/BuildCommandTest.groovy
            http://jenkins-ci.org/commit/jenkins/9402bd82cdafdde8b28a99c2cd6fede9409499a1
            Log:
            Merge pull request #1229 from synopsys-arc-oss/CLI_null_ParameteValue

            [READY][FIXED JENKINS-15094] - Handle null parameter values to avoid massive executor deaths

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: core/src/main/java/hudson/cli/BuildCommand.java core/src/main/java/hudson/model/ParameterDefinition.java core/src/main/java/hudson/model/ParametersAction.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java core/src/test/java/hudson/model/ParametersActionTest.java test/src/test/groovy/hudson/cli/BuildCommandTest.groovy http://jenkins-ci.org/commit/jenkins/9402bd82cdafdde8b28a99c2cd6fede9409499a1 Log: Merge pull request #1229 from synopsys-arc-oss/CLI_null_ParameteValue [READY] [FIXED JENKINS-15094] - Handle null parameter values to avoid massive executor deaths
            oleg_nenashev Oleg Nenashev added a comment -

            Marked the issue as lts-candidate

            oleg_nenashev Oleg Nenashev added a comment - Marked the issue as lts-candidate
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/00292ff44c38a4d6c1423020f13f97e48efd7bad
            Log:
            Marking JENKINS-15094 major bug

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: changelog.html http://jenkins-ci.org/commit/jenkins/00292ff44c38a4d6c1423020f13f97e48efd7bad Log: Marking JENKINS-15094 major bug
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3425
            JENKINS-15094 - Unit tests for the issue (Revision cd23fac9950c1c0762731372df2c109d4bb80db3)
            [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI). (Revision 7b894f57b78be0e4d847fe1c971955d2cbb08e58)

            Result = SUCCESS
            o.v.nenashev : cd23fac9950c1c0762731372df2c109d4bb80db3
            Files :

            • core/src/test/java/hudson/model/ParametersActionTest.java
            • test/src/test/groovy/hudson/cli/BuildCommandTest.groovy

            o.v.nenashev : 7b894f57b78be0e4d847fe1c971955d2cbb08e58
            Files :

            • core/src/main/java/hudson/model/ParametersAction.java
            • core/src/main/java/hudson/cli/BuildCommand.java
            • core/src/main/java/hudson/model/ParametersDefinitionProperty.java
            • core/src/main/java/hudson/model/ParameterDefinition.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3425 JENKINS-15094 - Unit tests for the issue (Revision cd23fac9950c1c0762731372df2c109d4bb80db3) [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI). (Revision 7b894f57b78be0e4d847fe1c971955d2cbb08e58) Result = SUCCESS o.v.nenashev : cd23fac9950c1c0762731372df2c109d4bb80db3 Files : core/src/test/java/hudson/model/ParametersActionTest.java test/src/test/groovy/hudson/cli/BuildCommandTest.groovy o.v.nenashev : 7b894f57b78be0e4d847fe1c971955d2cbb08e58 Files : core/src/main/java/hudson/model/ParametersAction.java core/src/main/java/hudson/cli/BuildCommand.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java core/src/main/java/hudson/model/ParameterDefinition.java
            oleg_nenashev Oleg Nenashev added a comment - - edited

            For backporting:
            https://github.com/jenkinsci/jenkins/pull/1229

            Commits:

            oleg_nenashev Oleg Nenashev added a comment - - edited For backporting: https://github.com/jenkinsci/jenkins/pull/1229 Commits: Unit test: https://github.com/jenkinsci/jenkins/commit/cd23fac9950c1c0762731372df2c109d4bb80db3 The issue fix: https://github.com/jenkinsci/jenkins/commit/7b894f57b78be0e4d847fe1c971955d2cbb08e58
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/test/java/hudson/model/ParametersActionTest.java
            test/src/test/groovy/hudson/cli/BuildCommandTest.groovy
            http://jenkins-ci.org/commit/jenkins/5cfcebf722ed027cb9dbbd1b5b3142196fc58161
            Log:
            JENKINS-15094 - Unit tests for the issue

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            (cherry picked from commit cd23fac9950c1c0762731372df2c109d4bb80db3)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/test/java/hudson/model/ParametersActionTest.java test/src/test/groovy/hudson/cli/BuildCommandTest.groovy http://jenkins-ci.org/commit/jenkins/5cfcebf722ed027cb9dbbd1b5b3142196fc58161 Log: JENKINS-15094 - Unit tests for the issue Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com> (cherry picked from commit cd23fac9950c1c0762731372df2c109d4bb80db3)
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/cli/BuildCommand.java
            core/src/main/java/hudson/model/ParameterDefinition.java
            core/src/main/java/hudson/model/ParametersAction.java
            core/src/main/java/hudson/model/ParametersDefinitionProperty.java
            http://jenkins-ci.org/commit/jenkins/b6b94ea5e7a89068728c0a65e6323ca6991c7ddb
            Log:
            [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI).

            The change adds missing checks for null ParameterValue, which may be passed from the CLI.
            It prevents the death of Jenkins executors during the canTake() check of node labels.

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            (cherry picked from commit 7b894f57b78be0e4d847fe1c971955d2cbb08e58)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/cli/BuildCommand.java core/src/main/java/hudson/model/ParameterDefinition.java core/src/main/java/hudson/model/ParametersAction.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java http://jenkins-ci.org/commit/jenkins/b6b94ea5e7a89068728c0a65e6323ca6991c7ddb Log: [FIXED JENKINS-15094] - Handle null parameter values in BuildCommand (CLI) and Parameter classes (CLI). The change adds missing checks for null ParameterValue, which may be passed from the CLI. It prevents the death of Jenkins executors during the canTake() check of node labels. Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com> (cherry picked from commit 7b894f57b78be0e4d847fe1c971955d2cbb08e58)

            People

              oleg_nenashev Oleg Nenashev
              rotem Rotem G
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: