Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34213

Under high throughput the Unexporter can fail to clean up fast enough

      The Unexporter does not try to drain the queue, rather it unexports at most one object every 200ms... which can create GC pressure when the remoting layer is under stress.

          [JENKINS-34213] Under high throughput the Unexporter can fail to clean up fast enough

          Stephen Connolly created issue -
          Stephen Connolly made changes -
          Assignee New: Stephen Connolly [ stephenconnolly ]

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/aee5a6f1756d6bcfba0123567e0195c4d584f8a2
          Log:
          [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/aee5a6f1756d6bcfba0123567e0195c4d584f8a2 Log: [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Stephen Connolly made changes -
          Labels New: lts-candidate

          Arnaud Héritier added a comment - stephenconnolly why is is it resolved ? The fix isn't yet merged / released ?? https://github.com/jenkinsci/remoting/commit/aee5a6f1756d6bcfba0123567e0195c4d584f8a2

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/98a7e3a837853ab1e8bf8b008862ed5738378bfc
          Log:
          JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability

          • We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much
          • When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO`
          • In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure
          • My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/98a7e3a837853ab1e8bf8b008862ed5738378bfc Log: JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO` In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/d9654829cade2bbedae787a1b5db5c2a66005b03
          Log:
          JENKINS-34213 Oops typo in the stats measurement

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/d9654829cade2bbedae787a1b5db5c2a66005b03 Log: JENKINS-34213 Oops typo in the stats measurement

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/4dbf0ea480defd9bf98fca6d8a50f499df4ff08d
          Log:
          JENKINS-34213 Copy and paste error

          The three most common programming errors are:

          • Off by one errors
          • Copy and paste errors

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/4dbf0ea480defd9bf98fca6d8a50f499df4ff08d Log: JENKINS-34213 Copy and paste error The three most common programming errors are: Off by one errors Copy and paste errors

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          http://jenkins-ci.org/commit/remoting/c9c077b7508fd715315a59a0423b7faec0b1a143
          Log:
          JENKINS-34213 Just when you were least expecting it, findbugs finds an actual real bug

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/RemoteInvocationHandler.java http://jenkins-ci.org/commit/remoting/c9c077b7508fd715315a59a0423b7faec0b1a143 Log: JENKINS-34213 Just when you were least expecting it, findbugs finds an actual real bug

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: