• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • Hudson 1.354 (?)

      This started over in JENKINS-6254, but I think it's not limited to Perforce, so I'm submitting this in Core.
      Today, after upgrading to 1.354, I'm seeing cases where builds move from their job into other jobs instead. In JENKINS-6254 I note one case where two builds (412+413) from one job move to another job, and possibly cause the Perforce plugin to get very confused about where to build.
      Earlier I had two jobs where 3 builds from each had switched sides and now were listed as belonging to the other job.
      The switching was complete in the sense that the build directories on the Hudson server were in the wrong places. I stopped the service and moved the directories around, and they looked fine again.
      One of the those jobs consistently failed like this:

      FATAL: null
      java.lang.NullPointerException
      	at hudson.tasks.ArtifactArchiver.prebuild(ArtifactArchiver.java:147)
      	at hudson.model.AbstractBuild$AbstractRunner.preBuild(AbstractBuild.java:595)
      	at hudson.model.AbstractBuild$AbstractRunner.preBuild(AbstractBuild.java:590)
      	at hudson.model.AbstractBuild$AbstractRunner.preBuild(AbstractBuild.java:586)
      	at hudson.model.Build$RunnerImpl.doRun(Build.java:114)
      	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
      	at hudson.model.Run.run(Run.java:1244)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:122)
      

      I'm not really keen on going through all our jobs to scan for switched builds, so I'll likely have to downgrade to 1.353.
      A curious thing, though, is that all of the switched jobs (so far) took place April 7th (a week ago). We were not running 1.354 then, but I didn't see the problems that I do now, so I'm mostly inclined to believe that it's something to do with 1.354.
      After downgrading, I cannot really be of much help as we don't have a "testing" server (but I'm beginning to think we should).

          [JENKINS-6256] Builds are listed under the wrong job

          ralfhergert added a comment -

          We have had this issue exactly after we installed the DownstreamBuildViewPlugin on our Hudson (1.358). After installing the plugin about 50 of 550 projects have been rebuild, till we noticed that in 8 projects builds were displayed which had a higher build number, then last regular build. We shutted down our server and inspected hudson's filesystem. It showed out, that in all affected builds each build.xml (.hudson/jobs/<project>/builds/<buildnumber>/build.xml) was overwritten by a build.xml of another project. All other files of the affected build were correct.

          I guess torbent is right suspecting the DownstreamBuildView plugin.

          ralfhergert added a comment - We have had this issue exactly after we installed the DownstreamBuildViewPlugin on our Hudson (1.358). After installing the plugin about 50 of 550 projects have been rebuild, till we noticed that in 8 projects builds were displayed which had a higher build number, then last regular build. We shutted down our server and inspected hudson's filesystem. It showed out, that in all affected builds each build.xml (.hudson/jobs/<project>/builds/<buildnumber>/build.xml) was overwritten by a build.xml of another project. All other files of the affected build were correct. I guess torbent is right suspecting the DownstreamBuildView plugin.

          torbent added a comment -

          When upgrading to 1.359 the other day (without DownstreamBuildView) I noticed no ill effects (had written a script to detect it!). Will check again when I, according to my plan, upgrade to 1.360 tomorrow. It's looking a bit more like DownstreamBuildView is to blame, so I'll move the issue to that component.
          ralfhergert, you shouldn't only look for higher numbers - I had several cases where the bad number was much lower. And I think there was one where the numbers sort of were ok; that was a bit harder to spot.

          torbent added a comment - When upgrading to 1.359 the other day (without DownstreamBuildView) I noticed no ill effects (had written a script to detect it!). Will check again when I, according to my plan, upgrade to 1.360 tomorrow. It's looking a bit more like DownstreamBuildView is to blame, so I'll move the issue to that component. ralfhergert, you shouldn't only look for higher numbers - I had several cases where the bad number was much lower. And I think there was one where the numbers sort of were ok; that was a bit harder to spot.

          torbent added a comment -

          Move to downstream-buildview component, as that's the prime suspect.

          torbent added a comment - Move to downstream-buildview component, as that's the prime suspect.

          ralfhergert added a comment -

          hello torbent, we did not rely only on the buildnumbers shown on hudson's gui. We wrote a shell script to find all build.xml files which are located in a workspace in which they are not belonging to. Our script uses the <workspace>-tag inside the build.xml for analysis.

          #!/bin/sh
          # this script will list all build.xml files which are located in a wrong build directory.
          # execute this script in the directory where hudson stores all jobs. (.hudson)
          for i in `find jobs/*/builds -type d | awk 'BEGIN {FS="/"} { print $2; }'  | uniq`
          do
                  echo testing project: $i
          
                  cd jobs/$i/builds
                  # check whether the projectname in <workspace> equals the directory project name
                  find . -name "build.xml" -exec grep -in "workspace" {} /dev/null \; | grep -v $i
          
                  cd ../../..
          done
          

          ralfhergert added a comment - hello torbent, we did not rely only on the buildnumbers shown on hudson's gui. We wrote a shell script to find all build.xml files which are located in a workspace in which they are not belonging to. Our script uses the <workspace>-tag inside the build.xml for analysis. #!/bin/sh # this script will list all build.xml files which are located in a wrong build directory. # execute this script in the directory where hudson stores all jobs. (.hudson) for i in `find jobs/*/builds -type d | awk 'BEGIN {FS= "/" } { print $2; }' | uniq` do echo testing project: $i cd jobs/$i/builds # check whether the projectname in <workspace> equals the directory project name find . -name "build.xml" -exec grep -in "workspace" {} /dev/ null \; | grep -v $i cd ../../.. done

          torbent added a comment -

          Looks very much like my script - same method

          torbent added a comment - Looks very much like my script - same method

          Is this still an issue?
          Was this problem only seen when upgrading the Hudson server or also when restarting the server?

          peter_schuetze added a comment - Is this still an issue? Was this problem only seen when upgrading the Hudson server or also when restarting the server?

          torbent added a comment -

          As I wrote above, I usually only restart when upgrading, so I cannot tell for sure whether it only happened when upgrading.
          I also do not know whether the issue is still there. I haven't run with DownStreamBuildView since I disabled it in May – but I haven't detected the issue since.

          torbent added a comment - As I wrote above, I usually only restart when upgrading, so I cannot tell for sure whether it only happened when upgrading. I also do not know whether the issue is still there. I haven't run with DownStreamBuildView since I disabled it in May – but I haven't detected the issue since.

          duality72 added a comment -

          I experienced the issue within the last week.

          duality72 added a comment - I experienced the issue within the last week.

          We have experienced the same issue and this has been corrupting our build history for quite some time...

          We also have experienced very strange symptoms because of this when the build number in a wrongly filed build folder was higher than the highest build number of the job; the workspace would either appear empty or it would show the workspace of the other job... Then CVS polling would go haywire and decide the job needed rebuilding as the workspace would be empty or contain the wrong code...

          Only just found this bug report. Can it please be assigned to someone and fixed with high priority?

          We are still on Hudson 1.395 and hope to move to Jenkins 1.447.1 LTS soon.

          Patrick Lemmens added a comment - We have experienced the same issue and this has been corrupting our build history for quite some time... We also have experienced very strange symptoms because of this when the build number in a wrongly filed build folder was higher than the highest build number of the job; the workspace would either appear empty or it would show the workspace of the other job... Then CVS polling would go haywire and decide the job needed rebuilding as the workspace would be empty or contain the wrong code... Only just found this bug report. Can it please be assigned to someone and fixed with high priority? We are still on Hudson 1.395 and hope to move to Jenkins 1.447.1 LTS soon.

          Joshua Ellis added a comment - - edited

          I'm experiencing a similar problem but with build.xmls being placed in the wrong jobs.

          I've figured out a pattern. It looks like everything is shifted over. The build.xmls are ending up in the folder of the job last finished running.
          Examples are always nice so what I'm saying is if my finish order is a,b,c a has b's build.xml and b has c's.

          Is there a known work around? Would a Jenkins clean install help?

          Joshua Ellis added a comment - - edited I'm experiencing a similar problem but with build.xmls being placed in the wrong jobs. I've figured out a pattern. It looks like everything is shifted over. The build.xmls are ending up in the folder of the job last finished running. Examples are always nice so what I'm saying is if my finish order is a,b,c a has b's build.xml and b has c's. Is there a known work around? Would a Jenkins clean install help?

            Unassigned Unassigned
            torbent torbent
            Votes:
            4 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: