Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7535

Rebuilding dependency graph slow on large installations

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • Apache Software Foundation, Hudson 1.376, Ubuntu server, Sun 1.6 JDK, 4Gb RAM

      In the ASF Hudson installation, rebuilding the dependency graph is quite slow (minutes). As rebuilding seems to happen twice on saving project configuration (AbstractProject.doConfigSubmit()), working with the project configuration is very slow and will frequently time out.

      We have a large number of jobs (>300) and from a quick look at the code, it seems like building the dependency graph is done even if there are no changes to the dependencies. Also, building the graph seems to be done by iterating over all jobs which will in turn iterator over all other jobs. This doesn't seem to scale to our size

      Here's a typical thread dump from saving a project configuration:

      "Handling POST /hudson/job/ftpserver-trunk-jdk1.6-osx/configSubmit : http-8090-27" daemon prio=10 tid=0x000000004340a000 nid=0x4576 runnable [0x00007f2fa8e48000]
      java.lang.Thread.State: RUNNABLE
      at java.lang.String.intern(Native Method)
      at hudson.maven.ModuleDependency.<init>(ModuleDependency.java:48)
      at hudson.maven.ModuleDependency.<init>(ModuleDependency.java:54)
      at hudson.maven.MavenModule.asDependency(MavenModule.java:296)
      at hudson.maven.MavenModule.buildDependencyGraph(MavenModule.java:385)
      at hudson.maven.MavenModuleSet.buildDependencyGraph(MavenModuleSet.java:487)
      at hudson.model.DependencyGraph.<init>(DependencyGraph.java:100)
      at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3346)
      at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:588)
      at sun.reflect.GeneratedMethodAccessor1225.invoke(Unknown Source)
      ...

          [JENKINS-7535] Rebuilding dependency graph slow on large installations

          protocol7b created issue -

          tdunning added a comment -

          It would be lovely if the dependency computation were to use a more clever algorithm so that it is very fast, but there are some quick wins available:

          a) don't do it if nothing changed

          b) don't do it if another thread is already doing a global recomputation. At the least, just have a single thread that does this dependency computation and have save mark the graph is needing an update. That way nobody is delayed and the worst that happens is one thread does the updates back-to-back forever.

          c) only change as much as is needed by the recently changed dependencies. This is an extension of (a) to make the entire computation incremental. Basically, you can keep an update time on each node and on the graph in general. In doing the dependency computation, you can skip part of the computation as soon as you note that the node under consideration hasn't changed more recently than the last graph update.

          tdunning added a comment - It would be lovely if the dependency computation were to use a more clever algorithm so that it is very fast, but there are some quick wins available: a) don't do it if nothing changed b) don't do it if another thread is already doing a global recomputation. At the least, just have a single thread that does this dependency computation and have save mark the graph is needing an update. That way nobody is delayed and the worst that happens is one thread does the updates back-to-back forever. c) only change as much as is needed by the recently changed dependencies. This is an extension of (a) to make the entire computation incremental. Basically, you can keep an update time on each node and on the graph in general. In doing the dependency computation, you can skip part of the computation as soon as you note that the node under consideration hasn't changed more recently than the last graph update.

          protocol7b added a comment -

          This problem has now grown to become even more serious in our environment. Since the dependency graph is also rebuilt after parsing the POM when building, builds are taking a pretty long time (10-15 minutes in the "Parsing POMs" state).

          Any chance of this problem being fixed in near time?

          protocol7b added a comment - This problem has now grown to become even more serious in our environment. Since the dependency graph is also rebuilt after parsing the POM when building, builds are taking a pretty long time (10-15 minutes in the "Parsing POMs" state). Any chance of this problem being fixed in near time?

          Can someone please look at this issue? We have large installation with cca 400 jobs (and increasing) and "parsing pom" phase takes cca 20-30minutes for each job. It begun to be a big issue for us, because we are able to build only small amounts of build per day.

          Jakub Štiller added a comment - Can someone please look at this issue? We have large installation with cca 400 jobs (and increasing) and "parsing pom" phase takes cca 20-30minutes for each job. It begun to be a big issue for us, because we are able to build only small amounts of build per day.

          evernat added a comment -

          The stack-trace in the issue is about the calls to the native method java.lang.String.intern() in the ModuleDependency constructor.

          To optimize the dependency graph, I have written a patch (JENKINS-7535-maven-plugin_evernat.patch attached) which divides the number of calls to intern() by 3 or a bit more.
          I couldn't test the patch and I certainly can't test with such a large installation, but I am quite sure that it won't break anything. So if possible, can someone test the patch and eventually make a pull request in github for it.

          The patch certainly needs review: in particular, some committers may not think that the assumption in the added private constructor of ModuleDependency is worth this part of the optimisation, even for this issue (if it is the case the 3 arguments constructor can be used instead).

          evernat added a comment - The stack-trace in the issue is about the calls to the native method java.lang.String.intern() in the ModuleDependency constructor. To optimize the dependency graph, I have written a patch ( JENKINS-7535 -maven-plugin_evernat.patch attached) which divides the number of calls to intern() by 3 or a bit more. I couldn't test the patch and I certainly can't test with such a large installation, but I am quite sure that it won't break anything. So if possible, can someone test the patch and eventually make a pull request in github for it. The patch certainly needs review: in particular, some committers may not think that the assumption in the added private constructor of ModuleDependency is worth this part of the optimisation, even for this issue (if it is the case the 3 arguments constructor can be used instead).
          evernat made changes -
          Attachment New: JENKINS-7535-maven-plugin_evernat.patch [ 20285 ]

          kutzi added a comment -

          Has anyone already confirmed that String.intern is really the part which is taking most of the time?
          Having a single stack trace which shows that is IMO not enough evidence to point out intern() as the culprit.

          kutzi added a comment - Has anyone already confirmed that String.intern is really the part which is taking most of the time? Having a single stack trace which shows that is IMO not enough evidence to point out intern() as the culprit.

          protocol7b added a comment -

          I've done quite a few thread dumps on our installation at ASF. All those has been pointing to String.intern. I can do additional debugging in our environment if needed.

          protocol7b added a comment - I've done quite a few thread dumps on our installation at ASF. All those has been pointing to String.intern. I can do additional debugging in our environment if needed.

          kutzi added a comment -

          Okay, that's good enough for me. Just wanted to make sure that no invalid assumptions are made based on a single dump.

          kutzi added a comment - Okay, that's good enough for me. Just wanted to make sure that no invalid assumptions are made based on a single dump.

          Olivier Lamy added a comment -

          Applying evernat patch looks good. I will do it.

          But I wonder simply removing those intern() call too ?
          IMHO not sure it's really usefull.
          Someone else ?

          Olivier Lamy added a comment - Applying evernat patch looks good. I will do it. But I wonder simply removing those intern() call too ? IMHO not sure it's really usefull. Someone else ?

            kutzi kutzi
            protocol7b protocol7b
            Votes:
            14 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: