Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-75711

Significant Declarative Pipeline Performance Issues

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins 2.504.1 LTS JDK21 installed in Kubernetes
      Installed 5.8.49 jenkins helm chart
      Plugins installed via jenkins-plugin-cli in base container extending jenkins 2.504.1-lts-jdk21 container image

      Attached an old list of plugins (old-plugins-to-install.yaml) and a new list of plugins (new-plugins-to-install.yaml) set of files. Both can be installed into 2.504.1 LTS jenkins, but the new list of plugins runs significantly slower.

      The performance problem is most easily demonstrated demonstrating running with a simple pipeline:

      pipeline {
          agent any
      
          stages {
              stage('Stage1') {
                  steps {
                      echo 'Hello World'
                  }
              }
              stage('Stage2') {
                  steps {
                      echo 'Hello World'
                  }
              }
              stage('Stage3') {
                  steps {
                      echo 'Hello World'
                  }
              }
          }
      } 

      With a newer list of plugins (see attachment), a Jenkins job defined with the code directly in the job takes roughly 10 seconds to execute, where the pipeline view shows each stage taking 3-4 seconds to run.

      With the older list of plugins (see attachment), a Jenkins job setup the same way takes well under 1 second to execute, where the pipeline view shows roughly 100 ms per stage to execute.

      First noticed this issue while trying to standard pipeline code from SCM using a shared global library. After much troubleshooting and disabling many extraneous plugins in a fresh install, was able to come up with the list above. After digging and bumping log levels and comparing release notes on various items, I was unable to come up with the root cause of this problem.

      In environments where others are using it, I will be holding off on upgrade until we can get a fix because the performance impact at scale is impactful. At this time, have not truly isolated the specific plugins impacted.

      Some other notes:

      • When using pipeline from SCM (used in tandem with cloudbees-bitbucket-branch-source and an organization job) , the first build for a branch is significantly faster than subsequent builds
        • Example: First build can be 17 seconds whereas 2nd, 3rd, 4th, etc. builds take closer to 2 minutes to complete
      • Freestyle jobs continue to execute just fine
      • Imperative jobs execute much faster than declarative, but still likely slower than ideal
        • Saw roughly 1 second per stage instead of 3-4 seconds per stage, but much higher than the ~100ms per stage given above example code
      • Changing pipeline durability (performance or otherwise) does not have a major impact the overall performance of the job

        1. image-2025-05-27-22-39-56-988.png
          image-2025-05-27-22-39-56-988.png
          69 kB
        2. image-2025-05-27-22-41-31-371.png
          image-2025-05-27-22-41-31-371.png
          196 kB
        3. image-2025-05-28-08-33-22-589.png
          image-2025-05-28-08-33-22-589.png
          19 kB
        4. image-2025-05-28-08-42-01-019.png
          image-2025-05-28-08-42-01-019.png
          86 kB
        5. new-plugins-performance.png
          new-plugins-performance.png
          124 kB
        6. new-plugins-RHEL-8.10.png
          new-plugins-RHEL-8.10.png
          131 kB
        7. new-plugins-to-install.yaml
          10 kB
        8. old-plugins-performance.png
          old-plugins-performance.png
          145 kB
        9. old-plugins-RHEL-8.10.png
          old-plugins-RHEL-8.10.png
          138 kB
        10. old-plugins-to-install.yaml
          11 kB
        11. run-jenkins.sh
          0.9 kB
        12. test-job-ldap-enabled.png
          test-job-ldap-enabled.png
          69 kB

            Unassigned Unassigned
            sweecm Colby
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: