Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65624

Webhook failures after upgrading jetty to 9.4.39.v20210325 in 2.277.3

    • 2.290, 2.289.2

      After upgrading Jenkins from 2.249.2 to 2.277.3, webhooks being fired from github enterprise by way of the ghrpb started randomly failing with errors indicating that the payload was missing from the request. Messages were similar to below:

      SEVERE: Request doesn't contain payload. You are sending url encoded request, so you should pass github payload through 'payload' request parameter

      We saw similarly worded errors coming from the Github plugin as well.

      I believe I've traced the failure to this issue in Jetty which looks like it's been corrected with this PR

      The underlying error was exposed after enabling Jenkins logging on jetty:

       

      May 12, 2021 11:34:33 AM FINE org.eclipse.jetty.server.HttpConnection fillRequestBuffer
      javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded

      Which further down in the stack indicated it was bubbling up from ghprb:

      org.jenkinsci.plugins.ghprb.GhprbRootAction.extractRequestBody(GhprbRootAction.java:286)

       

          [JENKINS-65624] Webhook failures after upgrading jetty to 9.4.39.v20210325 in 2.277.3

          Oleg Nenashev added a comment -

          CC olamy

           

          Oleg Nenashev added a comment - CC olamy  

          Olivier Lamy added a comment -

          _wayne thanks for the report. this should be fixed with Jetty 9.4.40 which is coming with winstone 5.17 from jenkins-2.282 

          Olivier Lamy added a comment - _wayne  thanks for the report. this should be fixed with Jetty 9.4.40 which is coming with winstone 5.17 from  jenkins-2.282 

          Olivier Lamy added a comment -

          let us know if upgrading fix your issue

          Olivier Lamy added a comment - let us know if upgrading fix your issue

          wayne goyer added a comment -

          Appreciate the info olamy !

          From release list it looks like 9.4.40 is available starting in Jenkins release 2.290. I'll test it in that release when I get a chance and add the results here. If this isn't what you had in mind lemme know.

          wayne goyer added a comment - Appreciate the info  olamy  ! From release list it looks like 9.4.40 is available starting in Jenkins release 2.290. I'll test it in that release when I get a chance and add the results here. If this isn't what you had in mind lemme know.

          Evan Bluhm added a comment -

           We've been running into the same bug in 2.277.3 (as have some other users in https://issues.jenkins.io/browse/JENKINS-65581). I can confirm that downgrading to 2.277.2 or upgrading to 2.290 both successfully resolve the problem. This does mean we're currently blocked from receiving any LTS security updates until the Winstone updates are backported or included in a new LTS version (it looks like the next RC is 2.289, so we might be waiting a long time).

          Evan Bluhm added a comment -  We've been running into the same bug in 2.277.3 (as have some other users in https://issues.jenkins.io/browse/JENKINS-65581 ). I can confirm that downgrading to 2.277.2 or upgrading to 2.290 both successfully resolve the problem. This does mean we're currently blocked from receiving any LTS security updates until the Winstone updates are backported or included in a new LTS version (it looks like the next RC is 2.289, so we might be waiting a long time).

          timja how do you feel about getting this one backported to 2.289.2?

          Baptiste Mathus added a comment - timja  how do you feel about getting this one backported to 2.289.2?

          I realize the related PR was not linked here, it's https://github.com/jenkinsci/jenkins/pull/5437

          Baptiste Mathus added a comment - I realize the related PR was not linked here, it's  https://github.com/jenkinsci/jenkins/pull/5437

            olamy Olivier Lamy
            _wayne wayne goyer
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: