Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58167

Add option to not store affected files

    XMLWordPrintable

Details

    Description

      We are using the warnings-ng plugin to analyze some of our projects with the following code

      recordIssues enabledForFailure: true, blameDisabled: true,tools: [java(), mavenConsole(), taskScanner(highTags: 'FIXME', normalTags: 'TODO, CHECKSTYLE IGNORE')]
      

      This leads to very large folders on the Jenkins master (around 250MB for a single build) compared to the old Static Analysis Plugin.

      An option to disable this behaviour or to specify retention times separatly from build retention times would be appreciated.

      Attachments

        Issue Links

          Activity

            pmr Philipp Moeller created issue -
            drulli Ulli Hafner made changes -
            Field Original Value New Value
            Description We are using the warnings-ng plugin to analyze some of our projects with the following code
            {code:java}
            recordIssues enabledForFailure: true, blameDisabled: true,tools: [java(), mavenConsole(), taskScanner(highTags: 'FIXME', normalTags: 'TODO, CHECKSTYLE IGNORE')]{code}

            This leads to very large folders on the Jenkins master (around 250MB for a single build) compared to the old Static Analysis Plugin.

            An option to disable this behaviour or to specify retention times separatly from build retention times would be appreciated.

            We are using the warnings-ng plugin to analyze some of our projects with the following code
            {code:java}
            recordIssues enabledForFailure: true, blameDisabled: true,tools: [java(), mavenConsole(), taskScanner(highTags: 'FIXME', normalTags: 'TODO, CHECKSTYLE IGNORE')]
            {code}

            This leads to very large folders on the Jenkins master (around 250MB for a single build) compared to the old Static Analysis Plugin.

            An option to disable this behaviour or to specify retention times separatly from build retention times would be appreciated.

            Issue Type Bug [ 1 ] Improvement [ 4 ]
            Summary files-with-issues folders excessively large Add option to not store affected files
            drulli Ulli Hafner made changes -
            Assignee Ulli Hafner [ drulli ]
            Labels help-wanted newbie-friendly
            drulli Ulli Hafner added a comment -

            Maybe we should provide the same options as the coverage API plugin.

            drulli Ulli Hafner added a comment - Maybe we should provide the same options as the coverage API plugin.
            sladyn98 Sladyn Nunes added a comment -

            drulli I would like to get started with this issue, any pointers on getting started with this ?

            sladyn98 Sladyn Nunes added a comment - drulli I would like to get started with this issue, any pointers on getting started with this ?
            drulli Ulli Hafner added a comment -

            One thing to start with is to have a look at the coverage-api plugin and see which options they provide for the user.

            Then we can discuss if we are heading for the simple solution (boolean flag) or a more sophisticating approach. One question for the latter one is: should this be solved in my plugin? Or should this be part of a source code renderer plugin that handles such a thing for all Jenkins plugins that show source code in some form. I discussed this in the coverage API Gitter channel a couple of weeks ago. I think providing a plugin would definitely be the best way but also the most complex one. I think this would be a GSoC project on its own...

            drulli Ulli Hafner added a comment - One thing to start with is to have a look at the coverage-api plugin and see which options they provide for the user. Then we can discuss if we are heading for the simple solution (boolean flag) or a more sophisticating approach. One question for the latter one is: should this be solved in my plugin? Or should this be part of a source code renderer plugin that handles such a thing for all Jenkins plugins that show source code in some form. I discussed this in the coverage API Gitter channel a couple of weeks ago. I think providing a plugin would definitely be the best way but also the most complex one. I think this would be a GSoC project on its own...
            sladyn98 Sladyn Nunes added a comment -

            Yeah I seen the thread on gitter, well then I guess I should move on to picking another issue, since the scope for the implementation is pretty wide.

            sladyn98 Sladyn Nunes added a comment - Yeah I seen the thread on gitter, well then I guess I should move on to picking another issue, since the scope for the implementation is pretty wide.
            drulli Ulli Hafner made changes -
            Link This issue relates to JENKINS-62151 [ JENKINS-62151 ]
            drulli Ulli Hafner made changes -
            Epic Link JENKINS-62153 [ 206075 ]
            romanz Roman Zwi added a comment -

            Quick workaround that worked for me:
            Configure a custom groovy parser that doesn't set the filename.
            So Warnings NG doesn't know the file name and can't do anything about it (this also avoids NoSuchFileException issues).

            romanz Roman Zwi added a comment - Quick workaround that worked for me: Configure a custom groovy parser that doesn't set the filename. So Warnings NG doesn't know the file name and can't do anything about it (this also avoids NoSuchFileException issues).
            drulli Ulli Hafner made changes -
            Labels help-wanted newbie-friendly help-wanted integration-test newbie-friendly testing-course
            drulli Ulli Hafner made changes -
            Rank Ranked lower
            drulli Ulli Hafner made changes -
            Labels help-wanted integration-test newbie-friendly testing-course help-wanted newbie-friendly
            drulli Ulli Hafner added a comment -

            I finally moved the source code rendering to https://github.com/jenkinsci/prism-api-plugin. In https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 I started the refactoring of the source code retention strategy in the code Coverage API. The what is missing still is an option in the steps that allows to select the strategy to use.

            drulli Ulli Hafner added a comment - I finally moved the source code rendering to https://github.com/jenkinsci/prism-api-plugin . In https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 I started the refactoring of the source code retention strategy in the code Coverage API. The what is missing still is an option in the steps that allows to select the strategy to use.
            drulli Ulli Hafner made changes -
            Epic Link JENKINS-62153 [ 206075 ]
            drulli Ulli Hafner made changes -
            Link This issue relates to JENKINS-61038 [ JENKINS-61038 ]
            drulli Ulli Hafner made changes -
            Component/s prism-api-plugin [ 28824 ]
            kon Kalle Niemitalo added a comment - - edited

            After the latest Code Coverage API release v2.0.4, https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 was closed, and https://github.com/jenkinsci/code-coverage-api-plugin/pull/310 was merged instead.

            https://www.jenkins.io/doc/pipeline/steps/code-coverage-api/ shows "NEVER_STORE, STORE_LAST_BUILD, STORE_ALL_BUILD" as levels of source file storage. After the PR 310 merge, these are defined in enum SourceFileResolverLevel. The level is passed in via the SourceFileResolver extension point but its sole implementation class DefaultSourceFileResolver is deprecated. I suppose that means Warnings NG should not imitate the SourceFileResolver extension point.

            For my purposes, it would be sufficient to support only NEVER_STORE and STORE_ALL_BUILD. I would then use NEVER_STORE, forward the warnings to the version-control server via Checks API, and view them from there. The level in Warnings NG could still be an enum, to allow STORE_LAST_BUILD to be implemented later if needed.

            For other users, it might make sense to offer a time-based preservation limit too, like in LogRotator. It would not be possible to use LogRotator itself, though.

            If Jenkins does not store the source files, but they are available from the web UI of the version-control server, then it would be nice to have each warning link there. RepositoryBrowser currently supports only linking to change sets, and not to lines of files, but I imagine that could be added pretty easily.

            kon Kalle Niemitalo added a comment - - edited After the latest Code Coverage API release v2.0.4, https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 was closed, and https://github.com/jenkinsci/code-coverage-api-plugin/pull/310 was merged instead. https://www.jenkins.io/doc/pipeline/steps/code-coverage-api/ shows "NEVER_STORE, STORE_LAST_BUILD, STORE_ALL_BUILD" as levels of source file storage. After the PR 310 merge, these are defined in enum SourceFileResolverLevel . The level is passed in via the SourceFileResolver extension point but its sole implementation class DefaultSourceFileResolver is deprecated. I suppose that means Warnings NG should not imitate the SourceFileResolver extension point. For my purposes, it would be sufficient to support only NEVER_STORE and STORE_ALL_BUILD. I would then use NEVER_STORE, forward the warnings to the version-control server via Checks API, and view them from there. The level in Warnings NG could still be an enum, to allow STORE_LAST_BUILD to be implemented later if needed. For other users, it might make sense to offer a time-based preservation limit too, like in LogRotator . It would not be possible to use LogRotator itself, though. If Jenkins does not store the source files, but they are available from the web UI of the version-control server, then it would be nice to have each warning link there. RepositoryBrowser currently supports only linking to change sets, and not to lines of files, but I imagine that could be added pretty easily.
            drulli Ulli Hafner added a comment - - edited

            I moved the new constants into the prism plugin because I want to use the same approach in the warnings plugin as well. But I simply did not find spare time yet to implement that in the warnings plugin. (My plan is to add automatic compression of these files as well as in the Coverage plugin).

            drulli Ulli Hafner added a comment - - edited I moved the new constants into the prism plugin because I want to use the same approach in the warnings plugin as well. But I simply did not find spare time yet to implement that in the warnings plugin. (My plan is to add automatic compression of these files as well as in the Coverage plugin).

            I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel.

            kon Kalle Niemitalo added a comment - I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel.
            kon Kalle Niemitalo added a comment - - edited

            Were you planning to add the SourceCodeRetention setting of Warnings NG as a parameter of the pipeline steps, or as a job property?

            Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue.

            kon Kalle Niemitalo added a comment - - edited Were you planning to add the SourceCodeRetention setting of Warnings NG as a parameter of the pipeline steps, or as a job property? Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue.
            drulli Ulli Hafner added a comment -

            I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel.

            Yes, this needs to be backward compatible so this looks somehow duplicated.

            drulli Ulli Hafner added a comment - I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel. Yes, this needs to be backward compatible so this looks somehow duplicated.
            drulli Ulli Hafner added a comment -

            Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue.

            I never thought about using it as a job property but that actually sounds like a good idea. Then all plugins can could use the same property. I need to look into Jenkins core to see what kind of extension point is required for this feature. (But I would assume that some people would like to use them differently for warnings and coverage).

            drulli Ulli Hafner added a comment - Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue. I never thought about using it as a job property but that actually sounds like a good idea. Then all plugins can could use the same property. I need to look into Jenkins core to see what kind of extension point is required for this feature. (But I would assume that some people would like to use them differently for warnings and coverage).
            drulli Ulli Hafner added a comment -

            As you mentioned above, it might be helpful as well, if we would not copy source code files at all and fetch them on demand directly from the SCM.

            drulli Ulli Hafner added a comment - As you mentioned above, it might be helpful as well, if we would not copy source code files at all and fetch them on demand directly from the SCM.

            I didn't mean Jenkins would fetch the source code files on demand. I meant, https://ci.jenkins.io/job/Plugins/job/prism-api-plugin/job/main/25/cpd/ currently displays warnings like this

            Details File Package Severity #Lines Duplicated In Age
              PrismConfigurationTest.java:50 io.jenkins.plugins.prism Low 5
            • PrismConfigurationTest.java:50
            • PrismConfigurationTest.java:84
            16

            If the project were configured not to preserve any files, then Jenkins could point the "PrismConfigurationTest.java:50" hyperlink to  https://github.com/jenkinsci/prism-api-plugin/blob/7ff237a95b02be1c58fbfe3ba1199b0321422d9f/src/test/java/io/jenkins/plugins/prism/PrismConfigurationTest.java#L50 rather than https://ci.jenkins.io/job/Plugins/job/prism-api-plugin/job/main/25/cpd/source.08941d3c-b38c-486c-b23f-8cb5a1cd3ddd/#50. The user could then click the link to view the source code to which the warning applies, although the warning itself might not be visible on that web page.

            This would require support in RepositoryBrowser or perhaps in SCMFile, and would not work right if the file is generated or modified during the build.

            kon Kalle Niemitalo added a comment - I didn't mean Jenkins would fetch the source code files on demand. I meant, https://ci.jenkins.io/job/Plugins/job/prism-api-plugin/job/main/25/cpd/ currently displays warnings like this Details File Package Severity #Lines Duplicated In Age   PrismConfigurationTest.java:50 io.jenkins.plugins.prism Low 5 PrismConfigurationTest.java:50 PrismConfigurationTest.java:84 16 If the project were configured not to preserve any files, then Jenkins could point the "PrismConfigurationTest.java:50" hyperlink to  https://github.com/jenkinsci/prism-api-plugin/blob/7ff237a95b02be1c58fbfe3ba1199b0321422d9f/src/test/java/io/jenkins/plugins/prism/PrismConfigurationTest.java#L50 rather than https://ci.jenkins.io/job/Plugins/job/prism-api-plugin/job/main/25/cpd/source.08941d3c-b38c-486c-b23f-8cb5a1cd3ddd/#50 . The user could then click the link to view the source code to which the warning applies, although the warning itself might not be visible on that web page. This would require support in RepositoryBrowser or perhaps in SCMFile, and would not work right if the file is generated or modified during the build.
            drulli Ulli Hafner added a comment -

            Ah ok, thanks for clarifying. That would be easier to implement.

            drulli Ulli Hafner added a comment - Ah ok, thanks for clarifying. That would be easier to implement.

            I feel the source code URLs should be generated by RepositoryBrowser (javadoc) implementations, because those already need to know about the URL format of the software that serves the files to Web browsers. Whether that is a version-control system like GitHub or separate code-review software.

            If Jenkins built a pull request merge, then it may be impossible to generate URLs for the merged files. Even if the code-review software supports such URLs, it might not merge the versions in exactly the same way. The RepositoryBrowser implementation should be told about the merge so that it can decide whether it wants to try constructing a URL or just give up.

            SCMRevision (javadoc) seems the obvious type to use for this, as you could then use ChangeRequestSCMRevision (javadoc) to represent the merge. However, these types are in scm-api-plugin while RepositoryBrowser is in core, so one cannot just add an Optional<URL> getFileLineLink(SCMRevision revision, String path, int startLine, int endLine) method to RepositoryBrowser. Instead, scm-api-plugin could define an interface with such a method, and individual RepositoryBrowser implementations could then implement that interface.

            kon Kalle Niemitalo added a comment - I feel the source code URLs should be generated by RepositoryBrowser ( javadoc ) implementations, because those already need to know about the URL format of the software that serves the files to Web browsers. Whether that is a version-control system like GitHub or separate code-review software. If Jenkins built a pull request merge, then it may be impossible to generate URLs for the merged files. Even if the code-review software supports such URLs, it might not merge the versions in exactly the same way. The RepositoryBrowser implementation should be told about the merge so that it can decide whether it wants to try constructing a URL or just give up. SCMRevision ( javadoc ) seems the obvious type to use for this, as you could then use ChangeRequestSCMRevision ( javadoc ) to represent the merge. However, these types are in scm-api-plugin while RepositoryBrowser is in core, so one cannot just add an Optional<URL> getFileLineLink(SCMRevision revision, String path, int startLine, int endLine) method to RepositoryBrowser. Instead, scm-api-plugin could define an interface with such a method, and individual RepositoryBrowser implementations could then implement that interface.
            drulli Ulli Hafner added a comment -

            The SCM API does not really provide a good API for such use cases. A couple of weeks ago I failed to add simple hyperlinks to Git commits in my plugin, so I needed to extend the Git plugin. The same needs to be done here...

            drulli Ulli Hafner added a comment - The SCM API does not really provide a good API for such use cases. A couple of weeks ago I failed to add simple hyperlinks to Git commits in my plugin, so I needed to extend the Git plugin. The same needs to be done here...

            I guess you mean https://github.com/jenkinsci/git-plugin/pull/1034 and https://github.com/jenkinsci/git-forensics-plugin/pull/241. For the source line links though, I don't think the API should be specific to Git. There are RepositoryBrowser implementations for other version-control systems, and at least some of them could implement the source line link API.

            For example, the RhodeCode software apparently supports https://try.rhodecode.com/ci-notify/files/3ad87bf0c7e571c685989e843936f16e945a899c/notify.py#L6 in a Mercurial repository. https://try.rhodecode.com/ci-notify/files/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md however seems problematic because it renders the Markdown to HTML and does not output anchors for line numbers. https://try.rhodecode.com/ci-notify/raw/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md outputs the original Markdown but that's text/plain and so has no anchors at all. Perhaps the hyperlink would have to use https://try.rhodecode.com/ci-notify/annotate/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md.

            kon Kalle Niemitalo added a comment - I guess you mean https://github.com/jenkinsci/git-plugin/pull/1034 and https://github.com/jenkinsci/git-forensics-plugin/pull/241 . For the source line links though, I don't think the API should be specific to Git. There are RepositoryBrowser implementations for other version-control systems, and at least some of them could implement the source line link API. For example, the RhodeCode software apparently supports https://try.rhodecode.com/ci-notify/files/3ad87bf0c7e571c685989e843936f16e945a899c/notify.py#L6 in a Mercurial repository. https://try.rhodecode.com/ci-notify/files/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md however seems problematic because it renders the Markdown to HTML and does not output anchors for line numbers. https://try.rhodecode.com/ci-notify/raw/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md outputs the original Markdown but that's text/plain and so has no anchors at all. Perhaps the hyperlink would have to use https://try.rhodecode.com/ci-notify/annotate/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md .

            My scenario is that we have binary files in version control (using Git LFS), and build steps sometimes output warnings about these files. In such cases, Warnings NG copies the files to the controller and displays them as text, which is not helpful at all. The Jenkinsfile in that project might be considered a bit incorrect, as it specifies sourceCodeEncoding: 'UTF-8' in the recordIssues step and that then applies to the binary files too; but well, there is no option to exclude file patterns from this encoding claim.

            kon Kalle Niemitalo added a comment - My scenario is that we have binary files in version control (using Git LFS), and build steps sometimes output warnings about these files. In such cases, Warnings NG copies the files to the controller and displays them as text, which is not helpful at all. The Jenkinsfile in that project might be considered a bit incorrect, as it specifies sourceCodeEncoding: 'UTF-8' in the recordIssues step and that then applies to the binary files too; but well, there is no option to exclude file patterns from this encoding claim.

            People

              Unassigned Unassigned
              pmr Philipp Moeller
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: