Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52165

Use push rather than pull for durable task logging

    XMLWordPrintable

Details

    • Pipeline Nodes and Processes 2.22

    Description

      Retroactively filing an issue originally treated as part of JENKINS-38381, since it is independently mergeable and has some independent benefits even when using existing (per-step, master-based) log storage.

      Historically, when running durable tasks (sh and kin), running output was handled by having the master send a callable to the agent at intervals, initially short (¼s) but growing exponentially up to some maximum (15s) if the process seems to be idle, and resetting to short again if and when fresh output is detected.

      Not only is there up to a 15s delay in displaying new output, this is wasteful of master and network resources when the process is idle for a long time; and sending a UserRequest and corresponding response involves a fair amount of Java serialization.

      By contrast, non-durable processes (such as those created by Launcher in a freestyle build) use RemoteOutputStream to send content from the agent to master as soon as it is available, minimizing network traffic; Remoting is also able to optimize this kind of traffic by sending low-overhead Chunk packets of tailored sizes.

      Durable tasks should switch from the policy of pulling log output to having the agent push log output. The content cannot be detected immediately, since we are effectively tailing a log file, but it can be detected quickly after the log file is updated with minimal overhead.

      Attachments

        Issue Links

          Activity

            basil Basil Crow added a comment -

            The experimental (newer) push mode reliably hangs when running on Java 17 in the context of the workflow-durable-task-step-plugin test suite (see JENKINS-68080). The default (older) pull mode passes on Java 17 in the same test suite.

            basil Basil Crow added a comment - The experimental (newer) push mode reliably hangs when running on Java 17 in the context of the workflow-durable-task-step-plugin test suite (see JENKINS-68080 ). The default (older) pull mode passes on Java 17 in the same test suite.
            jglick Jesse Glick added a comment -

            Disabling this mode in workflow-durable-task-step 2.25 until some reported issues can be tracked down and fixed.

            jglick Jesse Glick added a comment - Disabling this mode in workflow-durable-task-step 2.25 until some reported issues can be tracked down and fixed.
            dnusbaum Devin Nusbaum added a comment -

            Released in Durable Task version 1.26 and Pipeline Nodes and Processes 2.22.

            dnusbaum Devin Nusbaum added a comment - Released in Durable Task version 1.26 and Pipeline Nodes and Processes 2.22.
            jglick Jesse Glick added a comment -

            This change supersedes the code affected by JENKINS-37575.

            jglick Jesse Glick added a comment - This change supersedes the code affected by JENKINS-37575 .

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: