Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-607

Extract pipeline step implementations from all the plugins

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Similar Issues:

      Description

      We want to generate the reference manual for the pipeline plugin to be put on the website (see WEBSITE-129.) This requires an ability to automatically scan all the plugins for step implementations and generate them into some document format, such as asciidoc, HTML, etc.

      IIUC, the current thinking is to extend backend-extension-indexer for this job, given that it's already used to scan all the plugins.

      Another approach discussed in the past includes loading them in classpath and reusing existing reference generator from the pipeline plugin.

        Attachments

          Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-35348 Pipeline job loses connection with agent node

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is blocking

          Improvement - An improvement or enhancement to an existing feature or task. WEBSITE-129 Pipeline reference documentation

          • Major - Major loss of function.
          • Done
          links to

          Web Link PipelineStepsDoc job

          Web Link PR342

            Activity

            Ascending order - Click to sort in descending order
            16 older comments
            Hide
            Permalink
            rtyler R. Tyler Croy added a comment -

            Recalling where I last left off on this. ci.jenkins.io has agents coming online with limits at 4096 file descriptors, which is clearly not enough for this :-P

            Caused by: java.io.FileNotFoundException: /home/azureuser/workspace/Infra/pipeline-steps-doc-generator/ci-jenkinsfile/docFolder/./plugins/clang-scanbuild-plugin-1.7/WEB-INF/lib/clang-scanbuild-plugin.jar (Too many open files)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:219)
	at java.util.zip.ZipFile.<init>(ZipFile.java:149)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:103)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URL.openStream(URL.java:1045)
	at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:151)
	... 9 more
            Show
            rtyler R. Tyler Croy added a comment - Recalling where I last left off on this. ci.jenkins.io has agents coming online with limits at 4096 file descriptors, which is clearly not enough for this :-P Caused by: java.io.FileNotFoundException: /home/azureuser/workspace/Infra/pipeline-steps-doc-generator/ci-jenkinsfile/docFolder/./plugins/clang-scanbuild-plugin-1.7/WEB-INF/lib/clang-scanbuild-plugin.jar (Too many open files) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:103) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) at java.net.URL.openStream(URL.java:1045) at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:151) ... 9 more
            Hide
            Permalink
            rtyler R. Tyler Croy added a comment -

            Huin, looks like even when running on some Puppet-managed infrastructure our ulimit is too low

            Unable to obtain resource from ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar: 
java.io.FileNotFoundException: ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar (Too many open files)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:219)
	at java.util.zip.ZipFile.<init>(ZipFile.java:149)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:130)
	at jenkins.util.AntClassLoader.getResourceURL(AntClassLoader.java:1017)
	at jenkins.util.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:157)
	at jenkins.util.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:118)
	at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:964)
	at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:946)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44)
	at jenkins.ClassLoaderReflectionToolkit._findResources(ClassLoaderReflectionToolkit.java:103)
	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$UberPlusClassLoader.findResources(HyperLocalPluginManger.java:325)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
	at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:142)
	at net.java.sezpoz.Index$LazyIndexIterator.hasNext(Index.java:185)
	at com.google.common.collect.Lists.newArrayList(Lists.java:138)
	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256)
	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217)
	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.getIndices(HyperLocalPluginManger.java:397)
	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.scout(HyperLocalPluginManger.java:460)
	at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$ModClassicPluginStrategy.findComponents(HyperLocalPluginManger.java:358)
	at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.findSteps(PipelineStepExtractor.java:69)
	at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.main(PipelineStepExtractor.java:49)
            Show
            rtyler R. Tyler Croy added a comment - Huin, looks like even when running on some Puppet-managed infrastructure our ulimit is too low Unable to obtain resource from ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar: java.io.FileNotFoundException: ./plugins/async-http-client-1.7.24.1/WEB-INF/lib/async-http-client-1.7.24.jar (Too many open files) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.jar.JarFile.<init>(JarFile.java:166) at java.util.jar.JarFile.<init>(JarFile.java:130) at jenkins.util.AntClassLoader.getResourceURL(AntClassLoader.java:1017) at jenkins.util.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:157) at jenkins.util.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:118) at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:964) at jenkins.util.AntClassLoader.findResources(AntClassLoader.java:946) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44) at jenkins.ClassLoaderReflectionToolkit._findResources(ClassLoaderReflectionToolkit.java:103) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$UberPlusClassLoader.findResources(HyperLocalPluginManger.java:325) at java.lang. ClassLoader .getResources( ClassLoader .java:1142) at net.java.sezpoz.Index$LazyIndexIterator.peek(Index.java:142) at net.java.sezpoz.Index$LazyIndexIterator.hasNext(Index.java:185) at com.google.common.collect.Lists.newArrayList(Lists.java:138) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.getIndices(HyperLocalPluginManger.java:397) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$SmallSezpoz.scout(HyperLocalPluginManger.java:460) at org.jenkinsci.pipeline_steps_doc_generator.HyperLocalPluginManger$ModClassicPluginStrategy.findComponents(HyperLocalPluginManger.java:358) at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.findSteps(PipelineStepExtractor.java:69) at org.jenkinsci.pipeline_steps_doc_generator.PipelineStepExtractor.main(PipelineStepExtractor.java:49)
            Hide
            Permalink
            kwhetstone Kristin Whetstone added a comment -

            I had to bump this to a really high number in order to get this to work. My one test system is at 32768 if that helps any.

            Show
            kwhetstone Kristin Whetstone added a comment - I had to bump this to a really high number in order to get this to work. My one test system is at 32768 if that helps any.
            Hide
            Permalink
            danielbeck Daniel Beck added a comment -

            JarUrlConnection leaks. We recently had the problem in Jenkins. Oleg's PR: https://github.com/jenkinsci/jenkins/pull/2516

            Show
            danielbeck Daniel Beck added a comment - JarUrlConnection leaks. We recently had the problem in Jenkins. Oleg's PR: https://github.com/jenkinsci/jenkins/pull/2516
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: R. Tyler Croy
            Path:
            .fixtures.yml
            Puppetfile
            Vagrantfile
            dist/profile/manifests/buildslave.pp
            hieradata/common.yaml
            hieradata/roles/buildslave.yaml
            hieradata/roles/jenkins::agent.yaml
            spec/classes/profile/buildslave_spec.rb
            spec/classes/role/cucumber_spec.rb
            http://jenkins-ci.org/commit/jenkins-infra/eacd0706d79cf4171eee484ca8fbf635baaee713
            Log:
            Add support for tweaking limits for various users via the erwbgy/limits module

            References INFRA-607

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: .fixtures.yml Puppetfile Vagrantfile dist/profile/manifests/buildslave.pp hieradata/common.yaml hieradata/roles/buildslave.yaml hieradata/roles/jenkins::agent.yaml spec/classes/profile/buildslave_spec.rb spec/classes/role/cucumber_spec.rb http://jenkins-ci.org/commit/jenkins-infra/eacd0706d79cf4171eee484ca8fbf635baaee713 Log: Add support for tweaking limits for various users via the erwbgy/limits module References INFRA-607

              People

              Assignee:
              rtyler R. Tyler Croy
              Reporter:
              kohsuke Kohsuke Kawaguchi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: