Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62200

MF Application Automation Tools plugin: violation of RFC7230

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The plugin is unable to authenticate inside ALM during "Execute tests using ALM Lab Management" step if ALM server is behind haproxy v2.0 and above.

      The root cause is that the plugin expects Case-Sensitive http headers and by doing that violates RFC7230:
      https://tools.ietf.org/html/rfc7230#section-3.2

      Each header field consists of a case-insensitive field name followed
      by a colon (":"), optional leading whitespace, the field value, and
      optional trailing whitespace.

      And as we see in code, there are many places that violate this:

      For Set-Cookie header:

      For WWW-Authenticate header:

      Maybe there are other places and headers as well.

      We've faced the issue because newer versions of haproxy (2.0+) now use the new http processing mechanism internally (h2) by default. And because of this, all the http headers are now lowercased by default. So haproxy outputs "www-authenticate" instead of the original "WWW-Authenticate" which is perfectly compliant with RFC. But not with the plugin.

      Workaround

      There is a haproxy config option to override this behavior for some headers: https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#3.1-h1-case-adjust. You should use 2 haproxy configuration options in conjunction: "h1-case-adjust" and "option h1-case-adjust-bogus-client".

      We used it and I can prove that this is a valid workaround.

        Attachments

          Activity

          Hide
          roy_lu Roy Lu added a comment - - edited

          This is not something we could fix at the plugin side. The header is in the response of ALM. Should rise a ticket to ALM.

          Show
          roy_lu Roy Lu added a comment - - edited This is not something we could fix at the plugin side. The header is in the response of ALM. Should rise a ticket to ALM.
          Hide
          fff Fedor Radzievskiy added a comment -

          Roy Lu, maybe the description is slightly misleading.

          Currently we use ALM in the following configuration:
          User (a person or a Jenkins) -> HAProxy (load balancer) -> ALM (application server)

          If we use it without HAProxy, everything works as expected, so I'm not sure if I should rise an ALM ticket.

          If we use it with HAProxy in the middle, the issue occurs. But it's not a HAProxy issue or ALM issue, because it's the plugin who does not understand lowercased HTTP headers.

          Not only such behavior violates the RFC for HTTP/1.x, but also it makes it impossible to use the plugin with HTTP/2.
          See https://www.rfc-editor.org/rfc/rfc7540.html#section-8.1.2:

          Just as in HTTP/1.x, header field names are strings of ASCII
          characters that are compared in a case-insensitive fashion. However,
          header field names MUST be converted to lowercase prior to their
          encoding in HTTP/2. A request or response containing uppercase
          header field names MUST be treated as malformed (Section 8.1.2.6).

          And

          This is not something we could fix at the plugin side

          Exactly the opposite. And I've even referenced some places where you should fix it at the plugin side.

          Show
          fff Fedor Radzievskiy added a comment - Roy Lu , maybe the description is slightly misleading. Currently we use ALM in the following configuration: User (a person or a Jenkins) -> HAProxy (load balancer) -> ALM (application server) If we use it without HAProxy, everything works as expected, so I'm not sure if I should rise an ALM ticket. If we use it with HAProxy in the middle, the issue occurs. But it's not a HAProxy issue or ALM issue, because it's the plugin who does not understand lowercased HTTP headers. Not only such behavior violates the RFC for HTTP/1.x, but also it makes it impossible to use the plugin with HTTP/2. See https://www.rfc-editor.org/rfc/rfc7540.html#section-8.1.2: Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2. A request or response containing uppercase header field names MUST be treated as malformed (Section 8.1.2.6). And This is not something we could fix at the plugin side Exactly the opposite. And I've even referenced some places where you should fix it at the plugin side.
          Hide
          roy_lu Roy Lu added a comment - - edited

          Hi Fedor,

          I understand what you mean. It's because HAProxy has a restrict policy. If you don't want to by pass it through workaround, you need to rise ticket to ALM. We would see if it could support HAProxy or only by workaround.

          The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin. It is in ALM's response. The plugin just reads the value from this header. So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products. We have to let someone decide whether we can and how to fix this.

          Show
          roy_lu Roy Lu added a comment - - edited Hi Fedor, I understand what you mean. It's because HAProxy has a restrict policy. If you don't want to by pass it through workaround, you need to rise ticket to ALM. We would see if it could support HAProxy or only by workaround. The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin. It is in ALM's response. The plugin just reads the value from this header. So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products. We have to let someone decide whether we can and how to fix this.
          Hide
          fff Fedor Radzievskiy added a comment -

          So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products.

          I agree that this might affect many things if it is changed on ALM side, but still I don't understand why this should be changed on ALM side.

          The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin.

          Yes, but also it is a standard HTTP header. It's not ALM-specific.

          The plugin just reads the value from this header

          Yes, but reads in a case-sensitive manner, i.e. exactly "WWW-Authenticate".

          Haproxy lowercases all the headers in both directions, so, e.g.:
          1. the plugin calls /qcbin/rest/is-authenticated endpoint via haproxy
          2. haproxy forwards this request to ALM server with all the headers lowercased
          3. ALM server responds with 401 and sends "WWW-Authenticate" header
          4. ...which gets lowercased by haproxy...
          5. ...so that the plugin sees "www-authenticate"

          So at step 5 the plugin is unable to find the "WWW-Authenticate" header.

          But if we implement the mentioned workaround, then this will affect only the client (i.e. Jenkins and the plugin), so this will work correctly:

          1. the plugin posts "Authentication: Basic ..." header to /qcbin/authentication-point/alm-authenticate
          2. haproxy replaces this header to "authentication: ..." and forwards it to ALM
          3. ALM understands it in lowercase and responds with a valid "Set-Cookie" header

          I think ALM will understand most of the headers in lowercase because it uses HTTP-compliant Jetty server. At least we don't have any issues with ALM itself when using lowercased-headers.

          and how to fix this

          I see two solutions:

          • A proper one: use a dedicated http library in the plugin (e.g. Apache HTTP Client, google-http-client, OkHttp, etc... there are plenty of them and none of them uses raw HashMap for dealing with headers)
          • A fast one: extend HashMap that is used for headers in the plugin so that it treats header keys in a case-insensitive way
          Show
          fff Fedor Radzievskiy added a comment - So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products. I agree that this might affect many things if it is changed on ALM side, but still I don't understand why this should be changed on ALM side. The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin. Yes, but also it is a standard HTTP header. It's not ALM-specific. The plugin just reads the value from this header Yes, but reads in a case-sensitive manner , i.e. exactly "WWW-Authenticate". Haproxy lowercases all the headers in both directions, so, e.g.: 1. the plugin calls /qcbin/rest/is-authenticated endpoint via haproxy 2. haproxy forwards this request to ALM server with all the headers lowercased 3. ALM server responds with 401 and sends "WWW-Authenticate" header 4. ...which gets lowercased by haproxy... 5. ...so that the plugin sees "www-authenticate" So at step 5 the plugin is unable to find the "WWW-Authenticate" header. But if we implement the mentioned workaround, then this will affect only the client (i.e. Jenkins and the plugin), so this will work correctly: 1. the plugin posts "Authentication: Basic ..." header to /qcbin/authentication-point/alm-authenticate 2. haproxy replaces this header to "authentication: ..." and forwards it to ALM 3. ALM understands it in lowercase and responds with a valid "Set-Cookie" header I think ALM will understand most of the headers in lowercase because it uses HTTP-compliant Jetty server. At least we don't have any issues with ALM itself when using lowercased-headers. and how to fix this I see two solutions: A proper one: use a dedicated http library in the plugin (e.g. Apache HTTP Client, google-http-client, OkHttp, etc... there are plenty of them and none of them uses raw HashMap for dealing with headers) A fast one: extend HashMap that is used for headers in the plugin so that it treats header keys in a case-insensitive way
          Hide
          roy_lu Roy Lu added a comment - - edited

          Hi Fedor Radzievskiy,

          I got your point. Will fix it.

          Show
          roy_lu Roy Lu added a comment - - edited Hi Fedor Radzievskiy , I got your point. Will fix it.
          Hide
          roy_lu Roy Lu added a comment - - edited

          Hi Fedor Radzievskiy,

          After my investigation I have some points:

          1. No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'. I tested 'www-authenticate', 'WWW-AUTHENTICATE', 'wWw-auTHenticaTE'. For client, no matter client app or browser, the header would always be 'WWW-Authenticate'. I think this is the meaning of RFC-7230.
          2. Same behavior for all other headers. You can check that.
          3. Even I tried to use Apache http client, nothing different. Using http client can not solve this issue. Besides, we have some special URL format that would be consider illegal by these client api. And in Jenkins plugin, these clients can not use Jenkins Proxy configuration.
          4. I believe this is a special behavior of haproxy. Because it seems that other apps follow camel case rule. No matter how you set the name of header, it would always be in camel case at client side.
          5. I will make a patch for only your environment on condition that it would not produce regression on other environments.
          6. I can't test this at my environment. Could you help to test my build at your side?
          Show
          roy_lu Roy Lu added a comment - - edited Hi Fedor Radzievskiy , After my investigation I have some points: No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'. I tested 'www-authenticate', 'WWW-AUTHENTICATE', 'wWw-auTHenticaTE'. For client, no matter client app or browser, the header would always be 'WWW-Authenticate'. I think this is the meaning of RFC-7230. Same behavior for all other headers. You can check that. Even I tried to use Apache http client, nothing different. Using http client can not solve this issue. Besides, we have some special URL format that would be consider illegal by these client api. And in Jenkins plugin, these clients can not use Jenkins Proxy configuration. I believe this is a special behavior of haproxy. Because it seems that other apps follow camel case rule. No matter how you set the name of header, it would always be in camel case at client side. I will make a patch for only your environment on condition that it would not produce regression on other environments. I can't test this at my environment. Could you help to test my build at your side?
          Hide
          fff Fedor Radzievskiy added a comment - - edited

          Hi Roy Lu,

          No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'.

          The main question is: How did you manage to change it on ALM's side? I thought it is hardcoded there...

          I think this is the meaning of RFC-7230

          Well I treat it like this: "any client should be capable of parsing headers regardless of the case of their keys". From what you described, it seems that headers are still parsed case-sensitively, but for some reason their case is always the same (i.e. exactly WWW-Authenticate) and I don't understand how did you manage to achieve that.

          I can't test this at my environment. Could you help to test my build at your side?

          Sure. And I can also make a simple haproxy config for you so that you are able to test it on your environment because this should be way faster. The haproxy installation and configuration is pretty straightforward.

          Show
          fff Fedor Radzievskiy added a comment - - edited Hi Roy Lu , No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'. The main question is: How did you manage to change it on ALM's side? I thought it is hardcoded there... I think this is the meaning of RFC-7230 Well I treat it like this: "any client should be capable of parsing headers regardless of the case of their keys". From what you described, it seems that headers are still parsed case-sensitively, but for some reason their case is always the same (i.e. exactly WWW-Authenticate) and I don't understand how did you manage to achieve that. I can't test this at my environment. Could you help to test my build at your side? Sure. And I can also make a simple haproxy config for you so that you are able to test it on your environment because this should be way faster. The haproxy installation and configuration is pretty straightforward.
          Hide
          roy_lu Roy Lu added a comment -

          Hi Fedor Radzievskiy

          You're right. I found camel case is a behavior of ALM server. And for getting headers it always needs ignore case.

          // Get header in Apache http client
          public Header[] getHeaders(String name) {
              List<Header> headersFound = null;
          
              for(int i = 0; i < this.headers.size(); ++i) {
                  Header header = (Header)this.headers.get(i);
                  if (header.getName().equalsIgnoreCase(name)) {
                      if (headersFound == null) {
                          headersFound = new ArrayList();
                      }
          
                      headersFound.add(header);
                  }
              }
          
              return headersFound != null ? (Header[])headersFound.toArray(new Header[headersFound.size()]) : this.EMPTY;
          }
          

          Actually I wanted to replace our client with 3rd http client long ago. But because some URLs would be considered illegal by it. So I haven't apply it yet.

          I sent you the pb by email. Please check.

          Thanks.

           

           

          Show
          roy_lu Roy Lu added a comment - Hi Fedor Radzievskiy You're right. I found camel case is a behavior of ALM server. And for getting headers it always needs ignore case. // Get header in Apache http client public Header[] getHeaders( String name) { List<Header> headersFound = null ; for ( int i = 0; i < this .headers.size(); ++i) { Header header = (Header) this .headers.get(i); if (header.getName().equalsIgnoreCase(name)) { if (headersFound == null ) { headersFound = new ArrayList(); } headersFound.add(header); } } return headersFound != null ? (Header[])headersFound.toArray( new Header[headersFound.size()]) : this .EMPTY; } Actually I wanted to replace our client with 3rd http client long ago. But because some URLs would be considered illegal by it. So I haven't apply it yet. I sent you the pb by email. Please check. Thanks.    
          Hide
          fff Fedor Radzievskiy added a comment -

          Hi Roy Lu,
          Unfortunately I don't see anything on email. Maybe it's a Jira issue...
          Anyway I'll ask our Support to reach you to send it via FTP.

          some URLs would be considered illegal by it

          Can you please specify these URLs? We use our own java wrappers for ALM REST internally and I haven't encountered such issues so far. But our implementation does not cover all the endpoints.

          Show
          fff Fedor Radzievskiy added a comment - Hi Roy Lu , Unfortunately I don't see anything on email. Maybe it's a Jira issue... Anyway I'll ask our Support to reach you to send it via FTP. some URLs would be considered illegal by it Can you please specify these URLs? We use our own java wrappers for ALM REST internally and I haven't encountered such issues so far. But our implementation does not cover all the endpoints.
          Hide
          roy_lu Roy Lu added a comment - - edited

          Hi Fedor Radzievskiy,

          I resolved the illegal issue. Replacing with 3rd client lib is proceeding.

          An event log query api still got problem such as:

          http://host:port/qcbin/rest/domains/DEFAULT/projects/p1/event-log-reads?query=

          {context['*Timeslot: 1010*']}

          &fields=id

          Show
          roy_lu Roy Lu added a comment - - edited Hi Fedor Radzievskiy , I resolved the illegal issue. Replacing with 3rd client lib is proceeding. An event log query api still got problem such as: http://host:port/qcbin/rest/domains/DEFAULT/projects/p1/event-log-reads?query= {context['*Timeslot: 1010*']} &fields=id

            People

            Assignee:
            roy_lu Roy Lu
            Reporter:
            fff Fedor Radzievskiy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: