Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10596

Improve handling of source code files with duplicate relative path names

      I get the following error when trying to view the source of a compiler warning:

      01 Copying the source file 'main.c' from the workspace to the build folder '/var/lib/jenkins/home/jobs/Job Clean Build/builds/2011-08-05_00-04-13/workspace-files/bf88816e.tmp' on the Hudson master failed.
      02 Seems that the path is relative, however an absolute path is required when copying the sources.
      03 Is the file 'main.c' contained more than once in your workspace?
      04 Is the file 'main.c' a valid filename?
      05 If you are building on a slave: please check if the file is accessible under '$HUDSON_HOME/[job-name]/main.c'
      06 If you are building on the master: please check if the file is accessible under '$HUDSON_HOME/[job-name]/workspace/main.c'
      07 hudson.util.IOException2: remote file operation failed: main.c at hudson.remoting.LocalChannel@1eac6995
      08   at hudson.FilePath.act(FilePath.java:754)
      09   at hudson.FilePath.act(FilePath.java:740)
      10   at hudson.FilePath.copyTo(FilePath.java:1410)
      11   at hudson.plugins.analysis.core.HealthAwarePublisher.copyFilesWithAnnotationsToBuildFolder(HealthAwarePublisher.java:354)
      12   at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:316)
      13   at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
      14   at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
      15   at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
      16   at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
      17   at hudson.model.Build$RunnerImpl.post2(Build.java:161)
      18   at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
      19   at hudson.model.Run.run(Run.java:1400)
      20   at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      21   at hudson.model.ResourceController.execute(ResourceController.java:88)
      22   at hudson.model.Executor.run(Executor.java:175)
      23 Caused by: java.io.FileNotFoundException: main.c (No such file or directory)
      24   at java.io.FileInputStream.open(Native Method)
      25   at java.io.FileInputStream.<init>(FileInputStream.java:120)
      26   at hudson.FilePath$30.invoke(FilePath.java:1414)
      27   at hudson.FilePath$30.invoke(FilePath.java:1410)
      28   at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1980)
      29   at hudson.remoting.LocalChannel.call(LocalChannel.java:45)
      30   at hudson.FilePath.act(FilePath.java:747)
      31   ... 14 more
      

      This job uses a custom workspace dir so maybe the plugin doesn't support that?

          [JENKINS-10596] Improve handling of source code files with duplicate relative path names

          David Hallas added a comment -

          From examining my log files it looks like it parses the paths correctly, but if the code expects all file names to be unique in the workspace then it wont work.

          I think it would be a big help to clarify this detail to the user since I don't think it is uncommon to have duplicate filenames on a per workspace basis.

          David Hallas added a comment - From examining my log files it looks like it parses the paths correctly, but if the code expects all file names to be unique in the workspace then it wont work. I think it would be a big help to clarify this detail to the user since I don't think it is uncommon to have duplicate filenames on a per workspace basis.

          Ulli Hafner added a comment -

          Duplicates are no problem if specified by an absolute path. The problem only occurs with relative paths.

          E.g. you have the following structure:
          module-a/main.c
          module-b/main.c
          When a warning only contains the file 'main.c' its not decidable which one to choose.

          What do you think about providing a new "selection" page where you can choose between all possible files with that name? E.g., when the user selects the link a new page will be shown:

          Affected file 'main.c' is contained more then once in the workspace. Please choose from the following files:

          • module-a/main.c
          • module-b/main.c

          Maybe we can even introduce a never ask that question again checkbox.

          Would that be even better?

          Ulli Hafner added a comment - Duplicates are no problem if specified by an absolute path. The problem only occurs with relative paths. E.g. you have the following structure: module-a/main.c module-b/main.c When a warning only contains the file 'main.c' its not decidable which one to choose. What do you think about providing a new "selection" page where you can choose between all possible files with that name? E.g., when the user selects the link a new page will be shown: Affected file 'main.c' is contained more then once in the workspace. Please choose from the following files: module-a/main.c module-b/main.c Maybe we can even introduce a never ask that question again checkbox. Would that be even better?

          We have full paths in the log (see http://qa.coreboot.org/job/coreboot-gerrit/ws/coreboot-builds/amd_torpedo/make.log/*view*/), but still short paths in warnings (see http://qa.coreboot.org/job/coreboot-gerrit/656/warningsResult/?), so it's not just a problem of relative paths in the build (we have a non-recursive build system, thus full paths everywhere).

          Patrick Georgi added a comment - We have full paths in the log (see http://qa.coreboot.org/job/coreboot-gerrit/ws/coreboot-builds/amd_torpedo/make.log/*view*/ ), but still short paths in warnings (see http://qa.coreboot.org/job/coreboot-gerrit/656/warningsResult/? ), so it's not just a problem of relative paths in the build (we have a non-recursive build system, thus full paths everywhere).

          Ulli Hafner added a comment -

          I think this is fixed now when the new gcc parser with make support is used.

          Ulli Hafner added a comment - I think this is fixed now when the new gcc parser with make support is used.

          Colin Bennett added a comment - - edited

          I have encountered this issue again.

          • Jenkins ver. 1.539
          • Warnings plugin 4.38
          Content of file C_IcspAutoIPTransport.c
          01 Copying the source file '../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' from the workspace to the build folder 'C:\Jenkins\Home\jobs\Corvette_RX_Integration\builds\2013-12-10_16-19-48\workspace-files\2c97d6fe.tmp' on the Jenkins master failed.
          02 Seems that the path is relative, however an absolute path is required when copying the sources.
          03 Is the file 'C_IcspAutoIPTransport.c' contained more than once in your workspace?
          04 Is the file '../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' a valid filename?
          05 If you are building on a slave: please check if the file is accessible under '$JENKINS_HOME/[job-name]/../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c'
          06 If you are building on the master: please check if the file is accessible under '$JENKINS_HOME/[job-name]/workspace/../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c'
          07 java.io.FileNotFoundException: ..\..\..\..\common\comps\C_Icsp\common\C_IcspAutoIPTransport.c (The system cannot find the path specified)
          08   at java.io.FileInputStream.open(Native Method)
          09   at java.io.FileInputStream.<init>(Unknown Source)
          10   at hudson.FilePath$34.invoke(FilePath.java:1778)
          11   at hudson.FilePath$34.invoke(FilePath.java:1773)
          12   at hudson.FilePath.act(FilePath.java:917)
          13   at hudson.FilePath.act(FilePath.java:890)
          14   at hudson.FilePath.copyTo(FilePath.java:1773)
          15   at hudson.plugins.analysis.core.HealthAwareRecorder.copyFilesWithAnnotationsToBuildFolder(HealthAwareRecorder.java:398)
          16   at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:334)
          17   at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:333)
          18   at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32)
          19   at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:781)
          20   at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
          21   at hudson.model.Build$BuildExecution.post2(Build.java:183)
          22   at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
          23   at hudson.model.Run.execute(Run.java:1704)
          24   at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          25   at hudson.model.ResourceController.execute(ResourceController.java:88)
          26   at hudson.model.Executor.run(Executor.java:230)
          

          I get this error for all the source files because they are all referenced using ../../../foo/bar/baz.c type relative paths.

          For instance, the file in the above error trace is actually located:

          $ git ls-files */C_IcspAutoIPTransport.c
          products/hardware/common/comps/C_Icsp/common/C_IcspAutoIPTransport.c
          

          The make process driving the build was being performed with a current working directory of something like x/y/z/projname/build, basically all the ../../../.. just are a relative way of referring to the Git project tree root. (In particular, because Cygwin GNU make does not like C:\foo absolute paths – it chokes on the : colon and so I have to use tedious relative paths to avoid this.)

          Also, note that there's only one file with this "basename" so there is not ambiguity. We could maybe have a simple fallback case if the usual absolute resolution fails that could search for the file by basename only, if it's not ambiguous.

          The problem with completely fixing this issue is that it's hard for Jenkins to know universally what the current directory was when that warning was emitted, so it doesn't know that ../../../../.. refers to project root.

          But at least a workaround that would use a search for basename when not ambiguous would be a 90% solution.

          Colin Bennett added a comment - - edited I have encountered this issue again. Jenkins ver. 1.539 Warnings plugin 4.38 Content of file C_IcspAutoIPTransport.c 01 Copying the source file '../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' from the workspace to the build folder 'C:\Jenkins\Home\jobs\Corvette_RX_Integration\builds\2013-12-10_16-19-48\workspace-files\2c97d6fe.tmp' on the Jenkins master failed. 02 Seems that the path is relative, however an absolute path is required when copying the sources. 03 Is the file 'C_IcspAutoIPTransport.c' contained more than once in your workspace? 04 Is the file '../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' a valid filename? 05 If you are building on a slave: please check if the file is accessible under '$JENKINS_HOME/[job-name]/../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' 06 If you are building on the master: please check if the file is accessible under '$JENKINS_HOME/[job-name]/workspace/../../../../common/comps/C_Icsp/common/C_IcspAutoIPTransport.c' 07 java.io.FileNotFoundException: ..\..\..\..\common\comps\C_Icsp\common\C_IcspAutoIPTransport.c (The system cannot find the path specified) 08 at java.io.FileInputStream.open(Native Method) 09 at java.io.FileInputStream.<init>(Unknown Source) 10 at hudson.FilePath$34.invoke(FilePath.java:1778) 11 at hudson.FilePath$34.invoke(FilePath.java:1773) 12 at hudson.FilePath.act(FilePath.java:917) 13 at hudson.FilePath.act(FilePath.java:890) 14 at hudson.FilePath.copyTo(FilePath.java:1773) 15 at hudson.plugins.analysis.core.HealthAwareRecorder.copyFilesWithAnnotationsToBuildFolder(HealthAwareRecorder.java:398) 16 at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:334) 17 at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:333) 18 at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32) 19 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:781) 20 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753) 21 at hudson.model.Build$BuildExecution.post2(Build.java:183) 22 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706) 23 at hudson.model.Run.execute(Run.java:1704) 24 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) 25 at hudson.model.ResourceController.execute(ResourceController.java:88) 26 at hudson.model.Executor.run(Executor.java:230) I get this error for all the source files because they are all referenced using ../../../foo/bar/baz.c type relative paths. For instance, the file in the above error trace is actually located: $ git ls-files */C_IcspAutoIPTransport.c products/hardware/common/comps/C_Icsp/common/C_IcspAutoIPTransport.c The make process driving the build was being performed with a current working directory of something like x/y/z/projname/build , basically all the ../../../.. just are a relative way of referring to the Git project tree root. (In particular, because Cygwin GNU make does not like C:\foo absolute paths – it chokes on the : colon and so I have to use tedious relative paths to avoid this.) Also, note that there's only one file with this "basename" so there is not ambiguity. We could maybe have a simple fallback case if the usual absolute resolution fails that could search for the file by basename only, if it's not ambiguous. The problem with completely fixing this issue is that it's hard for Jenkins to know universally what the current directory was when that warning was emitted, so it doesn't know that ../../../../.. refers to project root. But at least a workaround that would use a search for basename when not ambiguous would be a 90% solution.

          Colin Bennett added a comment -

          What is that "new gcc parser with make support"? I am BTW using a Freescale/Metrowerks ColdFire compiler... ugh... much prefer gcc of course.

          Colin Bennett added a comment - What is that "new gcc parser with make support"? I am BTW using a Freescale/Metrowerks ColdFire compiler... ugh... much prefer gcc of course.

          Ulli Hafner added a comment - - edited

          Which parser are you using?

          Did you enable the scan workspace option?

          Ulli Hafner added a comment - - edited Which parser are you using? Did you enable the scan workspace option?

          Colin Bennett added a comment -

          What is the "scan workspace option"? I see there is a "Scan workspace files" section with an "Add" button that allows one to "Workspace files to scan for compiler warnings using a predefined parser", but that's not what I want, I am already scanning and correctly capturing the warnings on the console output – I don't have a log file to scan. I did try to enable the "Resolve relative paths" checkbox under "Advanced..." but that didn't fix the issue.

          I am using a custom parser for the Freescale (formerly Metrowerks) ColdFire compiler, with the "-msgstyle gcc" option.

          Here's the regex:

          ^\s*(?:\[exec\])\s*(.*):(\d+):\s*(warning|error): (\S.*)$
          

          and here's the mapping script:

          import hudson.plugins.warnings.parser.Warning
          
          String fileName = matcher.group(1)
          String lineNumber = matcher.group(2)
          String category = matcher.group(3)
          String message = matcher.group(4)
          
          return new Warning(fileName, Integer.parseInt(lineNumber), "Dynamic Parser", category, message);
          

          and here's an example message:

               [exec] ..\..\..\..\common\comps\mx\mqx\mxDefs.h:247: warning: florbity frobbity grokkity goo
          

          I see no problem with the parsing; it works fine and correctly identifies the file path and line number etc.

          Colin Bennett added a comment - What is the "scan workspace option"? I see there is a "Scan workspace files" section with an "Add" button that allows one to "Workspace files to scan for compiler warnings using a predefined parser", but that's not what I want, I am already scanning and correctly capturing the warnings on the console output – I don't have a log file to scan. I did try to enable the "Resolve relative paths" checkbox under "Advanced..." but that didn't fix the issue. I am using a custom parser for the Freescale (formerly Metrowerks) ColdFire compiler, with the "-msgstyle gcc" option. Here's the regex: ^\s*(?:\[exec\])\s*(.*):(\d+):\s*(warning|error): (\S.*)$ and here's the mapping script: import hudson.plugins.warnings.parser.Warning String fileName = matcher.group(1) String lineNumber = matcher.group(2) String category = matcher.group(3) String message = matcher.group(4) return new Warning(fileName, Integer.parseInt(lineNumber), "Dynamic Parser", category, message); and here's an example message: [exec] ..\..\..\..\common\comps\mx\mqx\mxDefs.h:247: warning: florbity frobbity grokkity goo I see no problem with the parsing; it works fine and correctly identifies the file path and line number etc.

          Ulli Hafner added a comment -

          Yes I thought the 'Resolve relative paths' option will help. Can you please activate the logging for hudson.plugins.analysis.core.ParserResult (level FINE). There should be some messages why the resolve relative path failed.

          Ulli Hafner added a comment - Yes I thought the 'Resolve relative paths' option will help. Can you please activate the logging for hudson.plugins.analysis.core.ParserResult (level FINE). There should be some messages why the resolve relative path failed.

          Ulli Hafner added a comment -

          Ulli Hafner added a comment - See JENKINS-32150 .

            drulli Ulli Hafner
            hallas David Hallas
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: