Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68307

Multibranch pipeline with many jenkinsfiles runs out of ports on master to connect to perforce

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • p4-plugin
    • Jenkins 2.332.1
      Pipeline: Multibranch Version711.vdfef37cda_816
      P4 Plugin Version1.12.2
      Pipeline Version2.7

    Description

      We are setting up a lot of builds in a single multibranch pipeline.  When we first added them we hit an error where our jenkins master was running out of ports to connect to perforce as it scanned.  So we made a simple repro.  We generated 153 jenkinsfiles in folders (I've attached an example jenkinsfile we used, and a screenshot showing the folder structure).  Then we created a new multibranch pipeline pointing at those files (see branch sources.png).  When that pipeline is added it tries to scan the multibranch pipeline then generates the error in scan_multibranch.txt (attached).  Running netstat on the master shows that all ports from 49152 to 65535 are open (in TIME_WAIT) to our perforce box!

      We can delay the running out of ports error by adding the "Suppress automatic SCM triggering" property, however, if we then try to build about 10 of the builds at once we hit the run out of ports error, so this is not a solution for us.

      Looking at the tty when it builds one of the jobs - it appears to grab all the jenkins files, rather than just the jenkinsfile for that particular job - see build_abcdef.txt.  Why does it do this?  Have I misunderstood how to configure a relatively large number of pipelines, or is this a bug?  I assume this is why we have so many TCP connections open to perforce when this occurs.

      Expected behaviour:

      When building job abcdef it grabs <folder>/abcdef/JenkinsFile from perforce, then does the build

      Observed behaviour:

      When building job abcdef it grabs <folder>/abcdef/JenkinsFile from perforce, <folder>/abcdfe/JenkinsFile, ... all the 153 jenkins files, before trying the build.

      This leads to many many connections to the perforce server, and it then runs out of ports. 

      Thank you!

       

      Attachments

        1. Screenshot 2022-04-19 165605.png
          Screenshot 2022-04-19 165605.png
          51 kB
        2. scan_multibranch.txt
          62 kB
        3. Jenkinsfile
          0.5 kB
        4. build_abcdef.txt
          225 kB
        5. branch sources.png
          branch sources.png
          24 kB

        Activity

          amyphillips Amy Phillips added a comment -

          Also occurs on Jenkins 2.332.2

           

          amyphillips Amy Phillips added a comment - Also occurs on Jenkins 2.332.2  

          People

            Unassigned Unassigned
            amyphillips Amy Phillips
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: