there is no need to set proxy config if splunk is directly reachable.

You probably set a proxy server which requires authentication, and got 403 error when password is absent

-Dhttp.proxyUser=<proxy user>
-Dhttp.proxyPassword=<proxy password>
 

Thank you for the quick response...

I removed every java option for proxy and it works. Gives us a way forward appreciated.

We need to look at any impact given we had originally (with 1.8.2) http.proxyHost http.proxyPort https.proxyHost

(Then added http.nonProxyHosts in attempt to fix for 1.9.4)

Should it be needed, will the splunk plugin use  nonProxyHosts (or no_proxy) for the Splunk server url should it be needed?

I think it dependent on your corp's network access control. 

If your jenkins server can reach splunk directly, there is no need to user a proxy server.

I know some users who host jenkins on a on-perm network which has no access to splunk server, and a proxy server is required.

you can verify proxy connectivity via curl 

curl -k -x "http://username:password@proxy.server:port" "https://splunk-host:${hec_port}/services/collector/event" -H "Authorization: Splunk ${hec_token}" -d \
'{"host":"test-host","index":"jenkins_console","sourcetype":"json:jenkins","source":"logger://dummy","event":{"level":"INFO","log_source":"cmdline","message":"Test HEC"}}' 

The proxy cmd should work as you suggest using user/pwd . But as a java passed variable we wouldn't use the pwd details in Jenkins config.

Our org uses the proxy settings at the OS level to facilitate cloud provider APIs and internet resources (other examples, the plugin updates that access without the proxy and yum installs etc All without need of pwd details).

If the latest plugin detects and explicitly uses the variable then we will require an "out" i.e. as advised, we don't require the proxy for Splunk so we will have to experiment with no_proxy type settings to override.

My hope is that the no_proxy or equivalent is also explicitly detected by the plugin

it should respect http.nonProxyHosts settings

https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

http.nonProxyHosts:a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by '|'. The patterns may start or end with a '*' for wildcards. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.

As promised, it does respect the settings, thank you again for all your timely help and advise.