Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31582

Log / document the plugin usage in the flow nodes

      Jenkins adds the plugin names and versions to the XML configuration, e.g. like plugin="groovy@1.27". Analyzing config.xml files it is possible to see what pluins are used by a job.

      With workflow this is not possible, since the information what plugin is used, is in the script file. It is probably difficult to analyze the script statically and extract plugin information.

      However, when a script is executed, workflow has the needed information. The steps are already logged to builds/<job-id>/workflow/<step-id>.xml.

      For a git checkout, via the generic SCM API it logs:

      <?xml version='1.0' encoding='UTF-8'?>
      <org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage_-Tag plugin="workflow-support@1.11">
        <node class="org.jenkinsci.plugins.workflow.cps.nodes.StepAtomNode" plugin="workflow-cps@1.11">
          <parents class="com.google.common.collect.ImmutableList">
            <org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode>5</org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode>
          </parents>
          <id>6</id>
          <descriptorId>org.jenkinsci.plugins.workflow.steps.scm.GenericSCMStep</descriptorId>
        </node>
      ...
      

      Improvement:

      Each step node contains also the used plugin(s) and version. In the above case it needs to log workflow-scm-step and git.

          [JENKINS-31582] Log / document the plugin usage in the flow nodes

          Jesse Glick added a comment -

          Suggested generalization of API: a new Action to be attached to a StepNode which records the step function name and its configuration in the format understood by DescribableModel. Helper methods could try to reconstruct the actual Step object, or produce a list of plugins involved in its definition.

          Would lead to more storage being used by the flow graph, but I think it is not too bad, so long as people refrain from putting ten-page shell scripts inline in their Jenkinsfile (anything complex should anyway be moved to a separate file). There are a lot of potential features related to visualization and analysis which could benefit from access to the detailed step definition.

          Jesse Glick added a comment - Suggested generalization of API: a new Action to be attached to a StepNode which records the step function name and its configuration in the format understood by DescribableModel . Helper methods could try to reconstruct the actual Step object, or produce a list of plugins involved in its definition. Would lead to more storage being used by the flow graph, but I think it is not too bad, so long as people refrain from putting ten-page shell scripts inline in their Jenkinsfile (anything complex should anyway be moved to a separate file). There are a lot of potential features related to visualization and analysis which could benefit from access to the detailed step definition.

          Jesse Glick added a comment -

          (Currently StepAtomNode and StepStartNode record a descriptorId, which could be deprecated in favor of the new action, without breaking compatibility for callers of StepNode.getDescriptor().)

          Jesse Glick added a comment - (Currently StepAtomNode and StepStartNode record a descriptorId , which could be deprecated in favor of the new action, without breaking compatibility for callers of StepNode.getDescriptor() .)

          Michael Neale added a comment -

          sounds interesting. I think peoeple may put several line scripts, but not pages, so this may not be too bad.

          Regarding display, would be nice to show a summary of what it did (with ability to display the whole script invoked if warranted), could be quite neat.

          Michael Neale added a comment - sounds interesting. I think peoeple may put several line scripts, but not pages, so this may not be too bad. Regarding display, would be nice to show a summary of what it did (with ability to display the whole script invoked if warranted), could be quite neat.

          Jesse Glick added a comment -

          Would suffice to deprecate StepNode and instead introduce a [FlowNode]Action into workflow-api which serializes (via XStream) the actual Step being run. (Details of how to do such a migration compatibly are TBD.) This metadata could be used for many purposes. For JENKINS-34002 we could simply scan for plugin="…" attributes. For JENKINS-37324 (vague as it is) various visualizations which show step-level details could show particular configuration; would also allow us to solve the problem of metasteps. For JENKINS-33995 we would be able to reconstruct the originally requested label expression, not just the node name this ultimately resolved to.

          Jesse Glick added a comment - Would suffice to deprecate StepNode and instead introduce a [FlowNode]Action into workflow-api which serializes (via XStream) the actual Step being run. (Details of how to do such a migration compatibly are TBD.) This metadata could be used for many purposes. For JENKINS-34002 we could simply scan for plugin="…" attributes. For JENKINS-37324 (vague as it is) various visualizations which show step-level details could show particular configuration; would also allow us to solve the problem of metasteps . For JENKINS-33995 we would be able to reconstruct the originally requested label expression, not just the node name this ultimately resolved to.

          Sam Van Oort added a comment -

          I'll pick this one up initially with a proposal to implement Step serialization in an Action (though reserving the right to simplify where needed to make the API easier to use or sidestep compatibility issue). Explicitly not committing to an ETA on this, just interest.

          May classify that work under JENKINS-37324 or here, depending on scope (I'd like to deal with step parameters first since it's far more necessary than plugin use reporting and I think much simpler too, surprisingly)

          Sam Van Oort added a comment - I'll pick this one up initially with a proposal to implement Step serialization in an Action (though reserving the right to simplify where needed to make the API easier to use or sidestep compatibility issue). Explicitly not committing to an ETA on this, just interest. May classify that work under JENKINS-37324 or here, depending on scope (I'd like to deal with step parameters first since it's far more necessary than plugin use reporting and I think much simpler too, surprisingly)

          Jesse Glick added a comment -

          I'd like to deal with step parameters first since it's […] much simpler

          Seems doubtful. A Step is a Describable. So are non-trivial step parameters.

          Jesse Glick added a comment - I'd like to deal with step parameters first since it's […] much simpler Seems doubtful. A Step is a Describable . So are non-trivial step parameters.

          Jesse Glick added a comment -

          This is sort of fixed, except the original requirement to record plugin version was apparently dropped.

          Jesse Glick added a comment - This is sort of fixed, except the original requirement to record plugin version was apparently dropped.

          Jesse Glick added a comment -

          would also allow us to solve the problem of metasteps

          Tracking as JENKINS-45109.

          Jesse Glick added a comment - would also allow us to solve the problem of metasteps Tracking as  JENKINS-45109 .

          Jesse Glick added a comment -

          Unfortunately there are problems with the API beyond the lack of plugin version information; it is tricky to reconstruct what the actual Step object looked like. For example, after running junit '*.xml' the singleton value of the map from ArgumentsAction.getArguments is an UninstantiatedDescribable (this seems to not be documented anywhere); yet the model is null, so you have to redo some of the work of determining that the junit symbol referred to JUnitResultArchiver. Presumably if the $class syntax had been used, you would get something else, even though these are purely aspects of surface syntax that is of no relevance after the build completes.

          Jesse Glick added a comment - Unfortunately there are problems with the API beyond the lack of plugin version information; it is tricky to reconstruct what the actual Step object looked like. For example, after running junit '*.xml' the singleton value of the map from ArgumentsAction.getArguments is an UninstantiatedDescribable (this seems to not be documented anywhere); yet the model is null, so you have to redo some of the work of determining that the junit symbol referred to JUnitResultArchiver . Presumably if the $class syntax had been used, you would get something else, even though these are purely aspects of surface syntax that is of no relevance after the build completes.

          Jesse Glick added a comment -

          The tricky part is reconstructing what the arguments actually referred to, since what is stored is not the fully-qualified names of the classes that the running build actually used, but symbols and abbreviated $class names as written in the script—both of which are potentially ambiguous without the context of the parameter using the nested object. Which we have, but the API does not make it straightforward to process this. Perhaps ArgumentsAction should offer a new API like getNormalizedArguments which would, for each nested object, produce an UninstantiatedDescribable whose model has been set, if the required classes are actually still available; from DescribableModel.type you could call PluginManager.whichPlugin or whatever you needed. (Doing this automatically in the existing getArguments is also an option, though the resolution is potentially expensive and not all clients will necessarily need it—though for JENKINS-45109 and JENKINS-34002 it is known to be needed.)

          Jesse Glick added a comment - The tricky part is reconstructing what the arguments actually referred to, since what is stored is not the fully-qualified names of the classes that the running build actually used, but symbols and abbreviated $class names as written in the script—both of which are potentially ambiguous without the context of the parameter using the nested object. Which we have, but the API does not make it straightforward to process this. Perhaps ArgumentsAction  should offer a new API like getNormalizedArguments which would, for each nested object, produce an UninstantiatedDescribable whose model has been set, if the required classes are actually still available; from DescribableModel.type you could call PluginManager.whichPlugin or whatever you needed. (Doing this automatically in the existing getArguments is also an option, though the resolution is potentially expensive and not all clients will necessarily need it—though for JENKINS-45109 and JENKINS-34002 it is known to be needed.)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java
          http://jenkins-ci.org/commit/workflow-api-plugin/8b2ef48c68b6adf1c2af0e0e53cdfaf5e8de1268
          Log:
          JENKINS-31582 Introduced ArgumentsAction.getResolvedArguments.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/8b2ef48c68b6adf1c2af0e0e53cdfaf5e8de1268 Log: JENKINS-31582 Introduced ArgumentsAction.getResolvedArguments.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java
          http://jenkins-ci.org/commit/workflow-api-plugin/f105670fc62c365da3fc4ea0fda21a80253d511f
          Log:
          Merge pull request #44 from jglick/ArgumentsAction.getResolvedArguments-JENKINS-31582

          JENKINS-31582 ArgumentsAction.getResolvedArguments

          Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/78744ae7045c...f105670fc62c

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/actions/ArgumentsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/f105670fc62c365da3fc4ea0fda21a80253d511f Log: Merge pull request #44 from jglick/ArgumentsAction.getResolvedArguments- JENKINS-31582 JENKINS-31582 ArgumentsAction.getResolvedArguments Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/78744ae7045c...f105670fc62c

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/0a0e05093fecc8adcf8ba1eaf541d655ba7a7464
          Log:
          JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/0a0e05093fecc8adcf8ba1eaf541d655ba7a7464 Log: JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109 .

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/b78991523e2ff0868610b770bd3719452452fb2d
          Log:
          Merge pull request #151 from jglick/ArgumentsAction.getResolvedArguments-JENKINS-31582

          JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4f703cf1b2c6...b78991523e2f

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/nodes/StepAtomNode.java src/test/java/org/jenkinsci/plugins/workflow/cps/actions/ArgumentsActionImplTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/nodes/StepNodeTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/b78991523e2ff0868610b770bd3719452452fb2d Log: Merge pull request #151 from jglick/ArgumentsAction.getResolvedArguments- JENKINS-31582 JENKINS-31582 Using ArgumentsAction.getResolvedArguments to better handle JENKINS-45109 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4f703cf1b2c6...b78991523e2f

          Jesse Glick added a comment -

          API committed which gives you access to plugin names used in step configurations. No version numbers available. That would require a complete redesign of the API; too late now.

          Jesse Glick added a comment - API committed which gives you access to plugin names used in step configurations. No version numbers available. That would require a complete redesign of the API; too late now.

          It would have a bit better possibly, but I don't think it's such a big issue. It's easier to get the plugin version, than the plugin's artifactId from the pluginManager's UI anyway.

          Baptiste Mathus added a comment - It would have a bit better possibly, but I don't think it's such a big issue. It's easier to get the plugin version, than the plugin's artifactId from the pluginManager's UI anyway.

            jglick Jesse Glick
            cruftex Jens Wilke
            Votes:
            4 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: