Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63655

Performance degradation when running multiple stages in parallel

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Simple steps are taking too long to be finished when we are running multiple stages (10+) in parallel (assuming that each stage creates a pod).

      A step like a simple `which` command that under normal conditions should take less than 1s is taking 40s. I attached 2 pictures, one when we are running only 4 stages in parallel and the other when there are 30 stages. Note in the other image attached (conflicting_times) that according to the logs a task runs in ~6s however Jenkins takes 27s to recognize it as done.

      The same pipeline with 70 parallel stages running on static pods (without using the plugin) on Kubernetes with the same Docker image takes ~7 min to be completed. Whereas this pipeline with the same 70 stages, is taking ~25 minutes under elastic agents (Kubernetes Plugin).

      I've attached the JNLP container logs as well although it does not have much information.

        Attachments

        1. 4_parallel_stages.png
          4_parallel_stages.png
          44 kB
        2. 50_parallel_stages.png
          50_parallel_stages.png
          46 kB
        3. conflicting times.png
          conflicting times.png
          70 kB
        4. jnlp_logs.txt
          2 kB

          Activity

          Hide
          vlatombe Vincent Latombe added a comment - - edited

          https://github.com/jenkinsci/kubernetes-plugin/pull/826 should have helped with that. Please try with recent versions (1.27.x or later)

          Also consider raising the maximum number of connections in the Cloud configuration. It defaults to 64, and each call inside a container block holds a websocket connection while it is running, which can cause additional calls to wait or event time out.

          Show
          vlatombe Vincent Latombe added a comment - - edited https://github.com/jenkinsci/kubernetes-plugin/pull/826 should have helped with that. Please try with recent versions (1.27.x or later) Also consider raising the maximum number of connections in the Cloud configuration. It defaults to 64, and each call inside a container block holds a websocket connection while it is running, which can cause additional calls to wait or event time out.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            mateusvtt Mateus Tanure
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated: