Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72655

Decreasing Time disparity between Linux and Windows automated test run time

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core, git-client-plugin
    • None

      The Jenkins core tests takes 2 hours to complete on Linux and 4 hours to complete on windows. In the Jenkins git client plugin the tests take 8.5 minutes to complete on Linux and 21 minutes on windows. A successful solution would be to try to at least locate a way to decrease this time disparity between these two Operating Systems when Jenkin’s tests are run. If we find this solution faster than expected we may increase the scope of our performance investigation.

      https://ci.jenkins.io/job/Core/job/jenkins/job/master/lastSuccessfulBuild/

      https://ci.jenkins.io/job/Core/job/jenkins/job/master/lastSuccessfulBuild/

      https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/lastBuild/pipeline-graph/

      https://ci.jenkins.io/job/Plugins/job/git-client-plugin/job/master/lastBuild/pipeline-graph/

          [JENKINS-72655] Decreasing Time disparity between Linux and Windows automated test run time

          Joshua created issue -

          Daniel Beck added a comment -

          If this is about Jenkins project infra, this should be tracked elsewhere (and I'm pretty sure it already is).

          Daniel Beck added a comment - If this is about Jenkins project infra, this should be tracked elsewhere (and I'm pretty sure it already is).

          Joshua added a comment -

          Where would you suggest I track it else where? I got this issue when I reached out to the new contributors forum for my team and I work on an issue and Mark Waite responded https://matrix.to/#/!VGaaNzNprwyVwVwxCr:gitter.im/$kkP9xo9Y4pa-RnDsFA9PHP_fZUXeMqppNRaI51Ds3vU?via=gitter.im&via=matrix.org&via=mozilla.org

          Joshua added a comment - Where would you suggest I track it else where? I got this issue when I reached out to the new contributors forum for my team and I work on an issue and Mark Waite responded https://matrix.to/#/!VGaaNzNprwyVwVwxCr:gitter.im/$kkP9xo9Y4pa-RnDsFA9PHP_fZUXeMqppNRaI51Ds3vU?via=gitter.im&via=matrix.org&via=mozilla.org

          Daniel Beck added a comment -

          Jenkins infra issues are generally tracked in the helpdesk: https://github.com/jenkins-infra/helpdesk/

          The INFRA project in this Jira is no longer in use, and JENKINS is also wrong unless this is more general than Jenkins project infra. Depends on what your intended scope is.

          Daniel Beck added a comment - Jenkins infra issues are generally tracked in the helpdesk: https://github.com/jenkins-infra/helpdesk/ The INFRA project in this Jira is no longer in use, and JENKINS is also wrong unless this is more general than Jenkins project infra. Depends on what your intended scope is.

          Mark Waite added a comment -

          This isn't an issue with Jenkins infrastructure. It is an issue that highlights a measurable performance difference between Windows and Linux for the same tests, even when using the same types of processors and same types of disc drives. I consistently see slower performance on the newer Windows computers at my house than on older Linux computers. The examples from ci.jenkins.io are examples of something similiar.

          I don't know the root reason why Jenkins tests of the git client plugin and some other plugins are much faster on Linux than on Windows, but I invited them to consider that difference as a candidate for a performance investigation.

          Mark Waite added a comment - This isn't an issue with Jenkins infrastructure. It is an issue that highlights a measurable performance difference between Windows and Linux for the same tests, even when using the same types of processors and same types of disc drives. I consistently see slower performance on the newer Windows computers at my house than on older Linux computers. The examples from ci.jenkins.io are examples of something similiar. I don't know the root reason why Jenkins tests of the git client plugin and some other plugins are much faster on Linux than on Windows, but I invited them to consider that difference as a candidate for a performance investigation.
          Mark Waite made changes -
          Summary Original: Decreasing Time disparity between Linux and Windows New: Decreasing Time disparity between Linux and Windows automated test run time
          Mark Waite made changes -
          Component/s New: git-client-plugin [ 17423 ]
          Component/s Original: git-plugin [ 15543 ]

          Daniel Beck added a comment - - edited

          using the same types of processors and same types of disc drives. I I consistently see slower performance on the newer Windows computers at my house than on older Linux computers. The examples from ci.jenkins.io are examples of something similiar.

          Quick googling finds e.g. https://www.reddit.com/r/linux/comments/w7no0p/why_are_most_operations_in_windows_much_slower/ and https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 and while neither is a good answer for this issue as filed (one is mostly about WSL, while the other has no definitive solution), I would start by trying to determine whether IO isn't just slower in general on Windows as these references indicate, and what we're seeing here is downstream from that.

          Daniel Beck added a comment - - edited using the same types of processors and same types of disc drives. I I consistently see slower performance on the newer Windows computers at my house than on older Linux computers. The examples from ci.jenkins.io are examples of something similiar. Quick googling finds e.g. https://www.reddit.com/r/linux/comments/w7no0p/why_are_most_operations_in_windows_much_slower/ and https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829 and while neither is a good answer for this issue as filed (one is mostly about WSL, while the other has no definitive solution), I would start by trying to determine whether IO isn't just slower in general on Windows as these references indicate, and what we're seeing here is downstream from that.

          Chris added a comment -

          Hi Daniel,

          I am part of Joshua's team. I agree with the findings from your first reddit link, Windows IO is generally slower than Linux because Windows is typically stepping on its own toes in many different ways that all kind of buildup to slow IO. Briefly skimming the WSL issue, the /mnt/ drives are under Windows protection, crossing the Windows/Linux filesystem "membrane" causes a drastic slowdown as Windows files != Linux files and there is conversion and checks that take place behind the scenes when file operations are crossing that. I remember reading long ago one of the big slowdowns is WSL doing file conversions to unix style and then back to dos style every time a file is accessed across the "membrane".

           

          We will begin investigating if the test slowness is a property of WIndows and what the exact property/properties are that cause this.

           

          I work at Microsoft and develop on Windows/Linux in my day to day, and I witness the exact same issues occurring in our test and build environments. I will see if I can find a company contact that might be able to provide some more info into how we mitigate the discrepancy. One thing I am thinking off the top of my head is what if we use Tiny11 as the test OS? Should be able to remove some of the process bloat that way and bypass any anti-virus scans. This isn't ideal for local dev setups that are on full versions of Windows, but would reduce the pipeline time and improve the time it takes to check in with Windows tests enabled.

           

          For local setups if this is indeed a windows property, we can narrow it down and provide recommendations for devs on OS settings to improve the performance, such as exclusion folders for antimaware, etc.

           

          Let me know your thoughts on above.

           

          Chris added a comment - Hi Daniel, I am part of Joshua's team. I agree with the findings from your first reddit link, Windows IO is generally slower than Linux because Windows is typically stepping on its own toes in many different ways that all kind of buildup to slow IO. Briefly skimming the WSL issue, the /mnt/ drives are under Windows protection, crossing the Windows/Linux filesystem "membrane" causes a drastic slowdown as Windows files != Linux files and there is conversion and checks that take place behind the scenes when file operations are crossing that. I remember reading long ago one of the big slowdowns is WSL doing file conversions to unix style and then back to dos style every time a file is accessed across the "membrane".   We will begin investigating if the test slowness is a property of WIndows and what the exact property/properties are that cause this.   I work at Microsoft and develop on Windows/Linux in my day to day, and I witness the exact same issues occurring in our test and build environments. I will see if I can find a company contact that might be able to provide some more info into how we mitigate the discrepancy. One thing I am thinking off the top of my head is what if we use Tiny11 as the test OS? Should be able to remove some of the process bloat that way and bypass any anti-virus scans. This isn't ideal for local dev setups that are on full versions of Windows, but would reduce the pipeline time and improve the time it takes to check in with Windows tests enabled.   For local setups if this is indeed a windows property, we can narrow it down and provide recommendations for devs on OS settings to improve the performance, such as exclusion folders for antimaware, etc.   Let me know your thoughts on above.  

          Daniel Beck added a comment -

          cherczeg

          One thing I am thinking off the top of my head is what if we use Tiny11 as the test OS? Should be able to remove some of the process bloat that way and bypass any anti-virus scans. This … would reduce the pipeline time and improve the time it takes to check in with Windows tests enabled.

          Hence my confusion whether this is a general or Jenkins infra issue

          To clarify, I don't want to get in the way of people wanting to solve problems and improve the situation, whether for Jenkins infra or more generally. We just get many improperly filed issues, and the first version of this looked like it was in the wrong place (and therefore likely not seen by folks who would be able to assist with infra).

          Daniel Beck added a comment - cherczeg One thing I am thinking off the top of my head is what if we use Tiny11 as the test OS? Should be able to remove some of the process bloat that way and bypass any anti-virus scans. This … would reduce the pipeline time and improve the time it takes to check in with Windows tests enabled. Hence my confusion whether this is a general or Jenkins infra issue To clarify, I don't want to get in the way of people wanting to solve problems and improve the situation, whether for Jenkins infra or more generally. We just get many improperly filed issues, and the first version of this looked like it was in the wrong place (and therefore likely not seen by folks who would be able to assist with infra).

            jerondon Joshua
            jerondon Joshua
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: