Maybe we should provide the same options as the coverage API plugin.

drulli I would like to get started with this issue, any pointers on getting started with this ?

One thing to start with is to have a look at the coverage-api plugin and see which options they provide for the user.

Then we can discuss if we are heading for the simple solution (boolean flag) or a more sophisticating approach. One question for the latter one is: should this be solved in my plugin? Or should this be part of a source code renderer plugin that handles such a thing for all Jenkins plugins that show source code in some form. I discussed this in the coverage API Gitter channel a couple of weeks ago. I think providing a plugin would definitely be the best way but also the most complex one. I think this would be a GSoC project on its own...

Yeah I seen the thread on gitter, well then I guess I should move on to picking another issue, since the scope for the implementation is pretty wide.

Quick workaround that worked for me:
Configure a custom groovy parser that doesn't set the filename.
So Warnings NG doesn't know the file name and can't do anything about it (this also avoids NoSuchFileException issues).

I finally moved the source code rendering to https://github.com/jenkinsci/prism-api-plugin. In https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 I started the refactoring of the source code retention strategy in the code Coverage API. The what is missing still is an option in the steps that allows to select the strategy to use.

After the latest Code Coverage API release v2.0.4, https://github.com/jenkinsci/code-coverage-api-plugin/pull/265 was closed, and https://github.com/jenkinsci/code-coverage-api-plugin/pull/310 was merged instead.

https://www.jenkins.io/doc/pipeline/steps/code-coverage-api/ shows "NEVER_STORE, STORE_LAST_BUILD, STORE_ALL_BUILD" as levels of source file storage. After the PR 310 merge, these are defined in enum SourceFileResolverLevel. The level is passed in via the SourceFileResolver extension point but its sole implementation class DefaultSourceFileResolver is deprecated. I suppose that means Warnings NG should not imitate the SourceFileResolver extension point.

For my purposes, it would be sufficient to support only NEVER_STORE and STORE_ALL_BUILD. I would then use NEVER_STORE, forward the warnings to the version-control server via Checks API, and view them from there. The level in Warnings NG could still be an enum, to allow STORE_LAST_BUILD to be implemented later if needed.

For other users, it might make sense to offer a time-based preservation limit too, like in LogRotator. It would not be possible to use LogRotator itself, though.

If Jenkins does not store the source files, but they are available from the web UI of the version-control server, then it would be nice to have each warning link there. RepositoryBrowser currently supports only linking to change sets, and not to lines of files, but I imagine that could be added pretty easily.

I moved the new constants into the prism plugin because I want to use the same approach in the warnings plugin as well. But I simply did not find spare time yet to implement that in the warnings plugin. (My plan is to add automatic compression of these files as well as in the Coverage plugin).

I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel.

Were you planning to add the SourceCodeRetention setting of Warnings NG as a parameter of the pipeline steps, or as a job property?

Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue.

I see, it's SourceCodeRetention in prism-api-plugin, and CoverageProcessor.mapSourceCodeRetention translates to that from SourceFileResolverLevel.

Yes, this needs to be backward compatible so this looks somehow duplicated.

Recently, I have been setting up a matrix build that compiles different binaries from the same source code. I'd like to scan the log files and publish the warnings in each matrix cell. If different matrix cells report warnings about the same files, then it would be best not to store multiple copies of the files in the Jenkins controller. However, if SourceCodeRetention is a parameter of pipeline steps, and different matrix cells specify different retention settings, then that would seem to require some kind of reference counting per file. Making SourceCodeRetention a property of the job would avoid that issue.

I never thought about using it as a job property but that actually sounds like a good idea. Then all plugins can could use the same property. I need to look into Jenkins core to see what kind of extension point is required for this feature. (But I would assume that some people would like to use them differently for warnings and coverage).

As you mentioned above, it might be helpful as well, if we would not copy source code files at all and fetch them on demand directly from the SCM.

Ah ok, thanks for clarifying. That would be easier to implement.

I feel the source code URLs should be generated by RepositoryBrowser (javadoc) implementations, because those already need to know about the URL format of the software that serves the files to Web browsers. Whether that is a version-control system like GitHub or separate code-review software.

If Jenkins built a pull request merge, then it may be impossible to generate URLs for the merged files. Even if the code-review software supports such URLs, it might not merge the versions in exactly the same way. The RepositoryBrowser implementation should be told about the merge so that it can decide whether it wants to try constructing a URL or just give up.

SCMRevision (javadoc) seems the obvious type to use for this, as you could then use ChangeRequestSCMRevision (javadoc) to represent the merge. However, these types are in scm-api-plugin while RepositoryBrowser is in core, so one cannot just add an Optional<URL> getFileLineLink(SCMRevision revision, String path, int startLine, int endLine) method to RepositoryBrowser. Instead, scm-api-plugin could define an interface with such a method, and individual RepositoryBrowser implementations could then implement that interface.

The SCM API does not really provide a good API for such use cases. A couple of weeks ago I failed to add simple hyperlinks to Git commits in my plugin, so I needed to extend the Git plugin. The same needs to be done here...

I guess you mean https://github.com/jenkinsci/git-plugin/pull/1034 and https://github.com/jenkinsci/git-forensics-plugin/pull/241. For the source line links though, I don't think the API should be specific to Git. There are RepositoryBrowser implementations for other version-control systems, and at least some of them could implement the source line link API.

For example, the RhodeCode software apparently supports https://try.rhodecode.com/ci-notify/files/3ad87bf0c7e571c685989e843936f16e945a899c/notify.py#L6 in a Mercurial repository. https://try.rhodecode.com/ci-notify/files/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md however seems problematic because it renders the Markdown to HTML and does not output anchors for line numbers. https://try.rhodecode.com/ci-notify/raw/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md outputs the original Markdown but that's text/plain and so has no anchors at all. Perhaps the hyperlink would have to use https://try.rhodecode.com/ci-notify/annotate/9d541eb7ff6e840ca2618ba63f4e0e17ff901808/readme.md.

My scenario is that we have binary files in version control (using Git LFS), and build steps sometimes output warnings about these files. In such cases, Warnings NG copies the files to the controller and displays them as text, which is not helpful at all. The Jenkinsfile in that project might be considered a bit incorrect, as it specifies sourceCodeEncoding: 'UTF-8' in the recordIssues step and that then applies to the binary files too; but well, there is no option to exclude file patterns from this encoding claim.