Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30873

Jenkins' Git plugin reparses all previous build.xml on restart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • git-plugin
    • None

      edit: I originally thought this was related to renaming a project, but this seems to be a general issue when restarting Jenkins. When we upgraded to 1.634 we simply restart Jenkins with the new .war file and even after 12h it wasn't done parsing the XML files from our tens of thousands of archived builds. We ended up moving all the old build directories out of the way and this immediately resolved the problem for us (we just can't access these old builds anymore, but that's less problematic than Jenkins itself being stuck on certain projects).

      I renamed a job with the following steps:

      1. Shut down Jenkins
      2. mv ~jenkins/job/prevname ~jenkins/job/newname
      3. Adjust any other references to the job name
      4. Start Jenkins

      (It's not the first time I do this and I've never had any issues in the past)

      This time I renamed a couple jobs with "lots" of previous builds (26k and 5k, respectively). Upon launching the first build for either of these jobs after restarting Jenkins, it was apparent that the build wasn't starting, couldn't be stopped, and was seemingly hung. Checking /threadDump and doing a bit of strace'ing showed that Jenkins was loading every single prior build.xml for these jobs.

      Common part of the stack trace:

      	at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:158)
	at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:135)
	at hudson.model.RunMap.retrieve(RunMap.java:224)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)
	-  locked hudson.model.RunMap@2cedf74
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
	at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:357)
	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:199)
	at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:107)
	at hudson.plugins.git.GitSCM.getBuildData(GitSCM.java:1553)
	at hudson.plugins.git.GitSCM.buildEnvVars(GitSCM.java:1161)
	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:940)
	at hudson.model.AbstractBuild$AbstractBuildExecution.decideWorkspace(AbstractBuild.java:481)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
	at hudson.model.Run.execute(Run.java:1741)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:408)

      strace output:

      ~/jobs/foo-master @sw-pm1.sjc> strace -fp 1964 -e trace='!futex' |& fgrep arista                                                                                                                                                           
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/6461/build.xml", O_RDONLY) = 506
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/6459/build.xml", O_RDONLY) = 500
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/6458/build.xml", O_RDONLY) = 506
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/6457/build.xml", O_RDONLY) = 500
[... several minutes later...]
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1089/build.xml", O_RDONLY) = 506
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1088/build.xml", O_RDONLY) = 506
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1087/build.xml", O_RDONLY) = 506
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1086/build.xml", O_RDONLY) = 502
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1085/build.xml", O_RDONLY) = 502
[pid  2216] open("/home/jenkins/jobs/foo-master/builds/1084/build.xml", O_RDONLY) = 502

      I've also seen it load other files, e.g.:

      [pid 14094] open("/home/jenkins/jobs/foo/builds/5266/archive", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14094] open("/home/jenkins/jobs/foo/builds/5268/log", O_RDONLY) = 500
[pid 14094] open("/home/jenkins/jobs/foo/builds/5268/log", O_RDONLY) = 500
[pid 18327] open("/home/jenkins/jobs/foo/builds/5177/build.xml", O_RDONLY) = 492
[pid 18328] open("/home/jenkins/jobs/foo/builds/5178/build.xml", O_RDONLY) = 492

            Unassigned Unassigned
            tsuna Benoit Sigoure
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: