This is not something we could fix at the plugin side. The header is in the response of ALM. Should rise a ticket to ALM.

roy_lu, maybe the description is slightly misleading.

Currently we use ALM in the following configuration:
User (a person or a Jenkins) -> HAProxy (load balancer) -> ALM (application server)

If we use it without HAProxy, everything works as expected, so I'm not sure if I should rise an ALM ticket.

If we use it with HAProxy in the middle, the issue occurs. But it's not a HAProxy issue or ALM issue, because it's the plugin who does not understand lowercased HTTP headers.

Not only such behavior violates the RFC for HTTP/1.x, but also it makes it impossible to use the plugin with HTTP/2.
See https://www.rfc-editor.org/rfc/rfc7540.html#section-8.1.2:

Just as in HTTP/1.x, header field names are strings of ASCII
characters that are compared in a case-insensitive fashion. However,
header field names MUST be converted to lowercase prior to their
encoding in HTTP/2. A request or response containing uppercase
header field names MUST be treated as malformed (Section 8.1.2.6).

And

This is not something we could fix at the plugin side

Exactly the opposite. And I've even referenced some places where you should fix it at the plugin side.

Hi Fedor,

I understand what you mean. It's because HAProxy has a restrict policy. If you don't want to by pass it through workaround, you need to rise ticket to ALM. We would see if it could support HAProxy or only by workaround.

The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin. It is in ALM's response. The plugin just reads the value from this header. So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products. We have to let someone decide whether we can and how to fix this.

So in order to fix this violation, we have to rise a ticket to ALM and it will go through the process. Because changing this header would affect many of our products.

I agree that this might affect many things if it is changed on ALM side, but still I don't understand why this should be changed on ALM side.

The "WWW-Authenticate" header you see in the plugin code, is no decided by the plugin.

Yes, but also it is a standard HTTP header. It's not ALM-specific.

The plugin just reads the value from this header

Yes, but reads in a case-sensitive manner, i.e. exactly "WWW-Authenticate".

Haproxy lowercases all the headers in both directions, so, e.g.:
1. the plugin calls /qcbin/rest/is-authenticated endpoint via haproxy
2. haproxy forwards this request to ALM server with all the headers lowercased
3. ALM server responds with 401 and sends "WWW-Authenticate" header
4. ...which gets lowercased by haproxy...
5. ...so that the plugin sees "www-authenticate"

So at step 5 the plugin is unable to find the "WWW-Authenticate" header.

But if we implement the mentioned workaround, then this will affect only the client (i.e. Jenkins and the plugin), so this will work correctly:

1. the plugin posts "Authentication: Basic ..." header to /qcbin/authentication-point/alm-authenticate
2. haproxy replaces this header to "authentication: ..." and forwards it to ALM
3. ALM understands it in lowercase and responds with a valid "Set-Cookie" header

I think ALM will understand most of the headers in lowercase because it uses HTTP-compliant Jetty server. At least we don't have any issues with ALM itself when using lowercased-headers.

and how to fix this

I see two solutions:

• A proper one: use a dedicated http library in the plugin (e.g. Apache HTTP Client, google-http-client, OkHttp, etc... there are plenty of them and none of them uses raw HashMap for dealing with headers)
• A fast one: extend HashMap that is used for headers in the plugin so that it treats header keys in a case-insensitive way

Hi fff,

I got your point. Will fix it.

Hi fff,

After my investigation I have some points:

1. No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'. I tested 'www-authenticate', 'WWW-AUTHENTICATE', 'wWw-auTHenticaTE'. For client, no matter client app or browser, the header would always be 'WWW-Authenticate'. I think this is the meaning of RFC-7230.
2. Same behavior for all other headers. You can check that.
3. Even I tried to use Apache http client, nothing different. Using http client can not solve this issue. Besides, we have some special URL format that would be consider illegal by these client api. And in Jenkins plugin, these clients can not use Jenkins Proxy configuration.
4. I believe this is a special behavior of haproxy. Because it seems that other apps follow camel case rule. No matter how you set the name of header, it would always be in camel case at client side.
5. I will make a patch for only your environment on condition that it would not produce regression on other environments.
6. I can't test this at my environment. Could you help to test my build at your side?

Hi roy_lu,

No matter what header I set in ALM's response, client will always get 'WWW-Authenticate'.

The main question is: How did you manage to change it on ALM's side? I thought it is hardcoded there...

I think this is the meaning of RFC-7230

Well I treat it like this: "any client should be capable of parsing headers regardless of the case of their keys". From what you described, it seems that headers are still parsed case-sensitively, but for some reason their case is always the same (i.e. exactly WWW-Authenticate) and I don't understand how did you manage to achieve that.

I can't test this at my environment. Could you help to test my build at your side?

Sure. And I can also make a simple haproxy config for you so that you are able to test it on your environment because this should be way faster. The haproxy installation and configuration is pretty straightforward.

Hi fff

You're right. I found camel case is a behavior of ALM server. And for getting headers it always needs ignore case.

// Get header in Apache http client
public Header[] getHeaders(String name) {
    List<Header> headersFound = null;

    for(int i = 0; i < this.headers.size(); ++i) {
        Header header = (Header)this.headers.get(i);
        if (header.getName().equalsIgnoreCase(name)) {
            if (headersFound == null) {
                headersFound = new ArrayList();
            }

            headersFound.add(header);
        }
    }

    return headersFound != null ? (Header[])headersFound.toArray(new Header[headersFound.size()]) : this.EMPTY;
}

Actually I wanted to replace our client with 3rd http client long ago. But because some URLs would be considered illegal by it. So I haven't apply it yet.

I sent you the pb by email. Please check.

Thanks.

Hi roy_lu,
Unfortunately I don't see anything on email. Maybe it's a Jira issue...
Anyway I'll ask our Support to reach you to send it via FTP.

some URLs would be considered illegal by it

Can you please specify these URLs? We use our own java wrappers for ALM REST internally and I haven't encountered such issues so far. But our implementation does not cover all the endpoints.

Hi fff,

I resolved the illegal issue. Replacing with 3rd client lib is proceeding.

An event log query api still got problem such as:

http://host:port/qcbin/rest/domains/DEFAULT/projects/p1/event-log-reads?query=

&fields=id