Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63665

Splunk plugin fails test connection when upgraded past 1.8.2 rollback restores

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: splunk-devops-plugin
    • Labels:
    • Environment:
      Aws ec2/Redhat: 3.10.0-1127.19.1.el7.x86_64
      Jenkins: Jenkins 2.249.1
      Plugins: splunk-devops-plugin (1.8.2 and 1.9.4)
      Java: openjdk version "1.8.0_262"
    • Similar Issues:

      Description

      Upgrading the working splunk-devops-plugin from (1.8.2) to (1.9.4) breaks the connection, rollback to 1.8.2 restores it.

      Test:

      Upgrade plugin press "Test Connection" (splunk for Jenkins configuration) get error

      Token:XXXXXX response:Forbidden

      Rollback to version 1.8.2

      Test connection success.

      In a an adhoc Jenkins instance created to debug this we updated all plugins to the latest version with the latest Jenkins.

      We also set the the variables http.proxyHost http.proxyPort https.proxyHost http.nonProxyHosts (per recent release notes)

       

      In the Jenkins log we see the following.

      com.splunk.splunkjenkins.utils.LogConsumer$SplunkClientError: Forbidden, http event collector token is invalid, status code:403

       

       

        Attachments

          Activity

          Hide
          fengxx Ted Xiao added a comment -

          there is no need to set proxy config if splunk is directly reachable.

          You probably set a proxy server which requires authentication, and got 403 error when password is absent

          -Dhttp.proxyUser=<proxy user>
          -Dhttp.proxyPassword=<proxy password>
           
          Show
          fengxx Ted Xiao added a comment - there is no need to set proxy config if splunk is directly reachable. You probably set a proxy server which requires authentication, and got 403 error when password is absent -Dhttp.proxyUser=<proxy user> -Dhttp.proxyPassword=<proxy password>
          Hide
          jim_zarakis Jim Zarakis added a comment -

          Thank you for the quick response...

          I removed every java option for proxy and it works. Gives us a way forward appreciated.

          We need to look at any impact given we had originally (with 1.8.2) http.proxyHost http.proxyPort https.proxyHost

          (Then added http.nonProxyHosts in attempt to fix for 1.9.4)

          Should it be needed, will the splunk plugin use  nonProxyHosts (or no_proxy) for the Splunk server url should it be needed?

           

           

          Show
          jim_zarakis Jim Zarakis added a comment - Thank you for the quick response... I removed every java option for proxy  and it works. Gives us a way forward appreciated. We need to look at any impact given we had originally (with 1.8.2) http.proxyHost http.proxyPort https.proxyHost (Then added http.nonProxyHosts in attempt to fix for 1.9.4) Should it be needed, will the splunk plugin use  nonProxyHosts (or no_proxy) for the Splunk server url should it be needed?    
          Hide
          fengxx Ted Xiao added a comment - - edited

          I think it dependent on your corp's network access control. 

          If your jenkins server can reach splunk directly, there is no need to user a proxy server.

          I know some users who host jenkins on a on-perm network which has no access to splunk server, and a proxy server is required.

           

          you can verify proxy connectivity via curl 

          curl -k -x "http://username:password@proxy.server:port" "https://splunk-host:${hec_port}/services/collector/event" -H "Authorization: Splunk ${hec_token}" -d \
          '{"host":"test-host","index":"jenkins_console","sourcetype":"json:jenkins","source":"logger://dummy","event":{"level":"INFO","log_source":"cmdline","message":"Test HEC"}}' 
          Show
          fengxx Ted Xiao added a comment - - edited I think it dependent on your corp's network access control.  If your jenkins server can reach splunk directly, there is no need to user a proxy server. I know some users who host jenkins on a on-perm network which has no access to splunk server, and a proxy server is required.   you can verify proxy connectivity via curl  curl -k -x "http: //username:password@proxy.server:port" "https://splunk-host:${hec_port}/services/collector/event" -H "Authorization: Splunk ${hec_token}" -d \ '{ "host" : "test-host" , "index" : "jenkins_console" , "sourcetype" : "json:jenkins" , "source" : "logger: //dummy" , "event" :{ "level" : "INFO" , "log_source" : "cmdline" , "message" : "Test HEC" }}'
          Hide
          jim_zarakis Jim Zarakis added a comment -

          The proxy cmd should work as you suggest using user/pwd . But as a java passed variable we wouldn't use the pwd details in Jenkins config.

          Our org uses the proxy settings at the OS level to facilitate cloud provider APIs and internet resources (other examples, the plugin updates that access without the proxy and yum installs etc All without need of pwd details).

          If the latest plugin detects and explicitly uses the variable then we will require an "out" i.e. as advised, we don't require the proxy for Splunk so we will have to experiment with no_proxy type settings to override.

          My hope is that the no_proxy or equivalent is also explicitly detected by the plugin

          Show
          jim_zarakis Jim Zarakis added a comment - The proxy cmd should work as you suggest using user/pwd . But as a java passed variable we wouldn't use the pwd details in Jenkins config. Our org uses the proxy settings at the OS level to facilitate cloud provider APIs and internet resources (other examples, the plugin updates that access without the proxy and yum installs etc All without need of pwd details). If the latest plugin detects and explicitly uses the variable then we will require an "out" i.e. as advised, we don't require the proxy for Splunk so we will have to experiment with no_proxy type settings to override. My hope is that the no_proxy or equivalent is also explicitly detected by the plugin
          Hide
          fengxx Ted Xiao added a comment -

          it should respect http.nonProxyHosts settings

          https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

          http.nonProxyHosts:a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by '|'. The patterns may start or end with a '*' for wildcards. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.

          Show
          fengxx Ted Xiao added a comment - it should respect  http.nonProxyHosts settings https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html http.nonProxyHosts :a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by '|'. The patterns may start or end with a '*' for wildcards. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
          Hide
          jim_zarakis Jim Zarakis added a comment -

          As promised, it does respect the settings, thank you again for all your timely help and advise.

          Show
          jim_zarakis Jim Zarakis added a comment - As promised, it does respect the settings, thank you again for all your timely help and advise.

            People

            Assignee:
            fengxx Ted Xiao
            Reporter:
            jim_zarakis Jim Zarakis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: