Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56613

MSBuild parser reports errors when message contains tool instead of file

      In some of our jobs we have seen messages like:

      13:19:52 [MSBuild] [ERROR] - '//INTRANET/Releases/SmartTranslator/ConsoleTranslator.exe', IO exception has been thrown: java.io.UncheckedIOException: java.nio.charset.UnmappableCharacterException: Input length = 1
      17:19:33 [MSBuild] [-ERROR-] - 'rs', IO exception has been thrown: java.nio.file.NoSuchFileException: rs
      17:19:33 [MSBuild] [-ERROR-] - 'NMAKE', IO exception has been thrown: java.nio.file.NoSuchFileException: NMAKE

      The source of them seem to be these lines in the output:

      16:43:02 \\INTRANET\Releases\SmartTranslator\ConsoleTranslator.exe: Warning ST: Could not translate string 'Expand f&olders larger than:'.
      17:19:23 NMAKE : fatal error U1077: '.\Tests.exe' : return code '0x1'
      17:19:23 NMAKE : fatal error U1077: '"C:\Program Files (x86)\Utils\NMAKE.EXE"' : return code '0x2'
      16:55:11 rs: error 002: Renaming the File  was not possible. Message: The process cannot access the file '...' because it is being used by another process.
      

      The problem for us is, that we also use the console parser plugin. When another tool reports a warning, the MSBuild parser reports an error because it has a problem with the tool and the job fails due to our configuration of the console parser plugin. As a workaround we currently execute the console parser plugin before the warnings-ng plugin.

      Based on https://blogs.msdn.microsoft.com/msbuild/2006/11/02/msbuild-visual-studio-aware-error-messages-and-message-formats/ tool names are valid.

          [JENKINS-56613] MSBuild parser reports errors when message contains tool instead of file

          Ulli Hafner added a comment -

          Can you please describe in more detail what the expected behavior should be?

          Ulli Hafner added a comment - Can you please describe in more detail what the expected behavior should be?

          We would expect that the MSBuild parser does not report any errors when a MSBuild warning contains a tool name instead of a file name.

          From what we have seen, the tool name seems to be always just the name without an extension or path (the ConsoleTranslator.exe in the example above is an inhouse project and we can correct it). Would it be a sufficient solution to decide on the format of the tool/file name what the behavior should be? If the name does not contain a path or extension it is a tool and the parser treats it like this. Meaning that it will not report any errors because the file was not found or cannot be parsed with the used charset. In all other cases the parser treats the name as a filename and continues with the current behavior. The only edge case of this solution we can currently think of would be that a file has no extension and is located in the current working directory (no path information).

          Artur Lipinski added a comment - We would expect that the MSBuild parser does not report any errors when a MSBuild warning contains a tool name instead of a file name. From what we have seen, the tool name seems to be always just the name without an extension or path (the ConsoleTranslator.exe in the example above is an inhouse project and we can correct it). Would it be a sufficient solution to decide on the format of the tool/file name what the behavior should be? If the name does not contain a path or extension it is a tool and the parser treats it like this. Meaning that it will not report any errors because the file was not found or cannot be parsed with the used charset. In all other cases the parser treats the name as a filename and continues with the current behavior. The only edge case of this solution we can currently think of would be that a file has no extension and is located in the current working directory (no path information).

          We get warnings like this (redacted a bit):

          EXEC : warning : Project file C:\J\ws\project@2\src\website\ cannot be found. [C:\J\ws\project@2\ci.msbuildproj]
          

          which then causes an error in Warnings NG:

          [MSBuild] [-ERROR-] Can't create fingerprints for some files:
          [MSBuild] [-ERROR-] - 'C:/J/ws/project@2/EXEC' file not found
          

          Here, EXEC is not a file name. Rather, it indicates MSBuild executed a program that output something to stderr. MSBuild captured that output and displayed it as a warning from the EXEC tool.

          Any of the following behaviors would be better:

          • For EXEC only, extract the embedded output "Project file C:\J\ws\project@2\src\website\ cannot be found." and then let other parsers try to figure it out. This seems complex to implement though, and there might not even be a parser that understands the output.
          • Assign the warning to the MSBuild project file, here "C:\J\ws\project@2\ci.msbuildproj". However, the MSBuild project file might reside outside the workspace, if it is from a NuGet package or from an SDK. In those cases, I think it is not useful to copy the content of the file to the Jenkins controller.
          • Just save the warning without assigning it to any file.

          Kalle Niemitalo added a comment - We get warnings like this (redacted a bit): EXEC : warning : Project file C:\J\ws\project@2\src\website\ cannot be found. [C:\J\ws\project@2\ci.msbuildproj] which then causes an error in Warnings NG: [MSBuild] [-ERROR-] Can't create fingerprints for some files: [MSBuild] [-ERROR-] - 'C:/J/ws/project@2/EXEC' file not found Here, EXEC is not a file name. Rather, it indicates MSBuild executed a program that output something to stderr. MSBuild captured that output and displayed it as a warning from the EXEC tool. Any of the following behaviors would be better: For EXEC only, extract the embedded output "Project file C:\J\ws\project@2\src\website\ cannot be found." and then let other parsers try to figure it out. This seems complex to implement though, and there might not even be a parser that understands the output. Assign the warning to the MSBuild project file, here "C:\J\ws\project@2\ci.msbuildproj". However, the MSBuild project file might reside outside the workspace, if it is from a NuGet package or from an SDK. In those cases, I think it is not useful to copy the content of the file to the Jenkins controller. Just save the warning without assigning it to any file.

          Ulli Hafner added a comment -

          jwalteri Are you still planning to provide a PR?

          Ulli Hafner added a comment - jwalteri Are you still planning to provide a PR?

          Warnings Next Generation 9.10.3 + Analysis Model API 10.9.1 seem to discard the tool warnings altogether, which affects the total number of warnings. Did someone want that behavior? I expected that the plugins would keep parsing the tool warnings, and just the I/O errors would be fixed.

          Kalle Niemitalo added a comment - Warnings Next Generation 9.10.3 + Analysis Model API 10.9.1 seem to discard the tool warnings altogether, which affects the total number of warnings. Did someone want that behavior? I expected that the plugins would keep parsing the tool warnings, and just the I/O errors would be fixed.

          Ulli Hafner added a comment -

          Seems that I did misunderstood the actual problem. So you do not want to want to ignore those warnings? You rather want that no "errors" will be reported by the plugin? Then I would rather suggest that the parser simply does not set the filename property.

          Ulli Hafner added a comment - Seems that I did misunderstood the actual problem. So you do not want to want to ignore those warnings? You rather want that no "errors" will be reported by the plugin? Then I would rather suggest that the parser simply does not set the filename property.

          Kalle Niemitalo added a comment - - edited

          Not setting the filename property would be fine. After all, if the EXEC tool had wanted to associate the warning with a specific file, it would have included the file name.

          Kalle Niemitalo added a comment - - edited Not setting the filename property would be fine. After all, if the EXEC tool had wanted to associate the warning with a specific file, it would have included the file name.

          Kalle Niemitalo added a comment - - edited

          Regarding this line:

          EXEC : warning : Project file C:\J\ws\project@2\src\website\ cannot be found. [C:\J\ws\project@2\ci.msbuildproj]
          

          It's originally output by nuget.exe restore in the following format, but captured and parsed by the Exec task and then reformatted by the MSBuild file logger:

          WARNING: Project file C:\J\ws\project@2\src\website\ cannot be found.
          

          If I wanted Warnings NG to associate this warning with "C:\J\ws\project@2\src\website\" (even though that is a directory rather than a file), I could implement a custom MSBuild task based on Exec and override the LogEventsFromTextOutput method. This way, the directory name would also be available to MSBuild loggers, so this approach would be better than making any Jenkins plugin parse the directory name from within the message.

          Kalle Niemitalo added a comment - - edited Regarding this line: EXEC : warning : Project file C:\J\ws\project@2\src\website\ cannot be found. [C:\J\ws\project@2\ci.msbuildproj] It's originally output by nuget.exe restore in the following format, but captured and parsed by the Exec task and then reformatted by the MSBuild file logger: WARNING: Project file C:\J\ws\project@2\src\website\ cannot be found. If I wanted Warnings NG to associate this warning with "C:\J\ws\project@2\src\website\" (even though that is a directory rather than a file), I could implement a custom MSBuild task based on Exec and override the LogEventsFromTextOutput method. This way, the directory name would also be available to MSBuild loggers, so this approach would be better than making any Jenkins plugin parse the directory name from within the message.

          We also didn't expect the current behavior. We, too, expected that the plugin would keep parsing the tool warnings, and just the I/O errors would be fixed somehow.

          The current solution also seems wrong to us because:

          • When the filename ends with ".exe", it isn't necessarily the name of a tool. It could still be the name of a scanned file. For example, we have analyzers that work with binary files to make sure that the copyright is set correctly or that the files are obfuscated in case of assemblies. For some files (i.e. *.dll) the warnings are shown, but for others (i.e. *.exe) not. This inconsistent behavior isn't good.
          • Files without an extension could also be valid scan targets, e.g. "Dockerfile" or "Jenkinsfile". Therefore, it doesn't seem like a good idea to exclude them.

          I'm still not sure how a correct/valid solution should look like because:

          • Detecting if a file is binary and not text, is not 100% reliably possible, as far as I know. Also, showing the user a message that the encoding is wrong is meaningful information and should be kept.
          • Detecting automatically if the filename is a tool or a scanned file, is also not 100% possible.

          Since automatic detection doesn't seem possible, maybe the only solution is to provide the possible tool names via configuration. The plugin could then skip some parts of the processing for such warnings, such as calculating the fingerprint or parsing the file.

          Artur Lipinski added a comment - We also didn't expect the current behavior. We, too, expected that the plugin would keep parsing the tool warnings, and just the I/O errors would be fixed somehow. The current solution also seems wrong to us because: When the filename ends with ".exe", it isn't necessarily the name of a tool. It could still be the name of a scanned file. For example, we have analyzers that work with binary files to make sure that the copyright is set correctly or that the files are obfuscated in case of assemblies. For some files (i.e. *.dll) the warnings are shown, but for others (i.e. *.exe) not. This inconsistent behavior isn't good. Files without an extension could also be valid scan targets, e.g. "Dockerfile" or "Jenkinsfile". Therefore, it doesn't seem like a good idea to exclude them. I'm still not sure how a correct/valid solution should look like because: Detecting if a file is binary and not text, is not 100% reliably possible, as far as I know. Also, showing the user a message that the encoding is wrong is meaningful information and should be kept. Detecting automatically if the filename is a tool or a scanned file, is also not 100% possible. Since automatic detection doesn't seem possible, maybe the only solution is to provide the possible tool names via configuration. The plugin could then skip some parts of the processing for such warnings, such as calculating the fingerprint or parsing the file.

          Ulli Hafner added a comment -

          Ulli Hafner added a comment - Will be partly fixed by https://github.com/jenkinsci/analysis-model/pull/1003

            Unassigned Unassigned
            x3ntrix Artur Lipinski
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: