Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30834

ReverseProxySetupMonitor seems not to be working with mod_jk and no reverse proxy

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Not A Defect
    • Component/s: core
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Some commenters in
      https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken
      complain that when using mod_jk and no reverse proxy, the reverse proxy check does not work correctly. I am experiencing this too.

      I ran the test suggested on that page and got these results

      % curl -iL -e http://jenkins.example.com/manage http://jenkins.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test
      HTTP/1.1 302 Found
      Date: Thu, 08 Oct 2015 00:32:40 GMT
      Server: Apache
      Location: http://jenkins.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http%3A%2F%2Fjenkins.example.com%2Fmanage/
      Content-Length: 0
      
      HTTP/1.1 404 http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage
      Date: Thu, 08 Oct 2015 00:32:40 GMT
      Server: Apache
      Cache-Control: must-revalidate,no-cache,no-store
      Content-Length: 1646
      Content-Type: text/html;charset=ISO-8859-1
      
      <html>
      <head>
      <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
      <title>Error 404 http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage</title>
      </head>
      <body><h2>HTTP ERROR 404</h2>
      <p>Problem accessing /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http:%252F%252Fjenkins.example.com%252Fmanage/. Reason:
      <pre>    http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      <br/>                                                
      
      </body>
      </html>
      
      With this configuration
      
      

      The config is

      /etc/libapache2-mod-jk/workers.properties:
      
      workers.tomcat_home=/var/lib/tomcat7
      workers.java_home=/usr/lib/jvm/java-1.7.0-openjdk-amd64
      ps=/
      worker.list=jenkins_worker
      worker.jenkins_worker.port=9009
      worker.jenkins_worker.host=jenkins.example.com
      worker.jenkins_worker.type=ajp13
      
      /etc/apache2/sites-enabled/jenkins.example.com:
      
      <VirtualHost *:80>
              ServerName      jenkins.example.com
              ServerAlias     jenkins-int.example.com
              ServerAdmin     web@example.com
              ServerSignature Off
              AllowEncodedSlashes NoDecode
              JkMount       /*   jenkins_worker
              JkLogLevel          info
              JkRequestLogFormat  "%w %V %T"
              JkLogStampFormat    "[%a %b %d %H:%M:%S %Y] "
              JkLogFile           /var/log/apache2/vhosts/jenkins.example.com/mod_jk.log
      
              LogLevel warn
              CustomLog    /var/log/apache2/vhosts/jenkins.example.com/access.log  backend_vhost
              ErrorLog     /var/log/apache2/vhosts/jenkins.example.com/error.log
      </VirtualHost>
      
      /etc/apache2/mods-enabled/jk.conf:
      
      <IfModule jk_module>
          JkWorkersFile /etc/libapache2-mod-jk/workers.properties
          JkLogFile /var/log/apache2/mod_jk.log
          JkLogLevel info
          JkShmFile /var/log/apache2/jk-runtime-status
          JkWatchdogInterval 60
          <Location /jk-status>
              JkMount jk-status
              Order deny,allow
              Deny from all
              Allow from 127.0.0.1
          </Location>
          <Location /jk-manager>
              JkMount jk-manager
              Order deny,allow
              Deny from all
              Allow from 127.0.0.1
          </Location>
      </IfModule>
      
      /etc/default/jenkins:
      
      NAME=jenkins
      JAVA=/usr/bin/java
      JAVA_ARGS="-Djava.awt.headless=true"  # Allow graphs etc. to work even when an X server is present
      JAVA_ARGS="$JAVA_ARGS -Dhudson.DNSMultiCast.disabled=true"
      PIDFILE=/var/run/jenkins/jenkins.pid
      JENKINS_USER=jenkins
      JENKINS_GROUP=jenkins
      JENKINS_WAR=/usr/share/jenkins/jenkins.war
      JENKINS_HOME=/var/lib/jenkins
      RUN_STANDALONE=true
      JENKINS_LOG=/var/log/jenkins/$NAME.log
      MAXOPENFILES=8192
      HTTP_PORT=-1
      AJP_PORT=9009
      PREFIX=/jenkins
      HTTP_ADDR=127.0.0.1
      JENKINS_ARGS="--webroot=/var/cache/jenkins/war -httpListenAddress=$HTTP_ADDR --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"
      
      

      In the jenkins /configure page, the URL is http://jenkins.example.com/

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Would be interesting to see what gets logged to org.kohsuke.stapler.Stapler on level FINER. https://wiki.jenkins-ci.org/display/JENKINS/Logging

          Also, enable Stapler request tracing by setting the System property stapler.trace to true and check the response headers.

          Show
          danielbeck Daniel Beck added a comment - Would be interesting to see what gets logged to org.kohsuke.stapler.Stapler on level FINER. https://wiki.jenkins-ci.org/display/JENKINS/Logging Also, enable Stapler request tracing by setting the System property stapler.trace to true and check the response headers.
          Hide
          xipmox Vince Murphy added a comment -

          Took a while to get back to this

          • I configured a logger for org.kohsuke.stapler.Stapler at FINER.
          • restarted jenkins with '-Dhudson.stapler.trace=true' in the JAVA_ARGS of the defaults file so when the process starts it calls java like this:
                /usr/bin/java -Djava.awt.headless=true \
                                    -Dhudson.DNSMultiCast.disabled=true\
                                    -Dhudson.stapler.trace=true \
                                    -jar /usr/share/jenkins/jenkins.war \
                                    -webroot=/var/cache/jenkins/war \
                                   -httpListenAddress=127.0.0.1 --httpPort=-1 --ajp13Port=9009
            

          When I make the curl request as above, I get this in the jenkins.log

          Dec 15, 2015 3:04:02 PM hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup
          WARNING: http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage
          

          The stapler log in the web gui showed this:

          Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler
          Handled by hudson.logging.LogRecorder.doClear(...) for url=/clear/...
          Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler
          Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/...
          Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler
          Handled by jenkins.model.Jenkins.getLog() for url=/log/...
          Dec 15, 2015 3:03:59 PM FINE org.kohsuke.stapler.Stapler
          Processing request for /log/stapler/
          Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler
          Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/...
          Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler
          Handled by jenkins.model.Jenkins.getLog() for url=/log/...
          Dec 15, 2015 3:04:00 PM FINER org.kohsuke.stapler.Stapler
          Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/...
          Dec 15, 2015 3:04:00 PM FINER org.kohsuke.stapler.Stapler
          Handled by jenkins.model.Jenkins.getLog() for url=/log/...
          Dec 15, 2015 3:04:02 PM FINE org.kohsuke.stapler.Stapler
          Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test
          Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler
          Handled by hudson.diagnosis.ReverseProxySetupMonitor.doTest(...) for url=/test/...
          Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler
          Handled by jenkins.model.Jenkins.getAdministrativeMonitor(String) for url=/administrativeMonitor/TOKEN/...
          Dec 15, 2015 3:04:02 PM FINE org.kohsuke.stapler.Stapler
          Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http:%252F%252Fjenkins.example.com%252Fmanage/
          Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler
          Handled by jenkins.model.Jenkins.getAdministrativeMonitor(String) for url=/administrativeMonitor/TOKEN/...
          Dec 15, 2015 3:04:10 PM FINE org.kohsuke.stapler.Stapler
          Processing request for /log/stapler/
          

          The response headers I got were

          HTTP/1.1 302 Found
          Date: Tue, 15 Dec 2015 04:04:02 GMT
          Server: Apache
          X-Content-Type-Options: nosniff
          Location: http://jenkins.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http%3A%2F%2Fjenkins.example.com%2Fmanage/
          Content-Length: 0
          
          HTTP/1.1 404 http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage
          Date: Tue, 15 Dec 2015 04:04:02 GMT
          Server: Apache
          X-Content-Type-Options: nosniff
          Cache-Control: must-revalidate,no-cache,no-store
          Content-Length: 1646
          Content-Type: text/html;charset=ISO-8859-1
          
          Show
          xipmox Vince Murphy added a comment - Took a while to get back to this I configured a logger for org.kohsuke.stapler.Stapler at FINER. restarted jenkins with '-Dhudson.stapler.trace=true' in the JAVA_ARGS of the defaults file so when the process starts it calls java like this: /usr/bin/java -Djava.awt.headless=true \ -Dhudson.DNSMultiCast.disabled=true\ -Dhudson.stapler.trace=true \ -jar /usr/share/jenkins/jenkins.war \ -webroot=/var/cache/jenkins/war \ -httpListenAddress=127.0.0.1 --httpPort=-1 --ajp13Port=9009 When I make the curl request as above, I get this in the jenkins.log Dec 15, 2015 3:04:02 PM hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup WARNING: http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage The stapler log in the web gui showed this: Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler Handled by hudson.logging.LogRecorder.doClear(...) for url=/clear/... Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/... Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler Handled by jenkins.model.Jenkins.getLog() for url=/log/... Dec 15, 2015 3:03:59 PM FINE org.kohsuke.stapler.Stapler Processing request for /log/stapler/ Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/... Dec 15, 2015 3:03:59 PM FINER org.kohsuke.stapler.Stapler Handled by jenkins.model.Jenkins.getLog() for url=/log/... Dec 15, 2015 3:04:00 PM FINER org.kohsuke.stapler.Stapler Handled by hudson.logging.LogRecorderManager.getDynamic(String,StaplerRequest,StaplerResponse) for url=/TOKEN/... Dec 15, 2015 3:04:00 PM FINER org.kohsuke.stapler.Stapler Handled by jenkins.model.Jenkins.getLog() for url=/log/... Dec 15, 2015 3:04:02 PM FINE org.kohsuke.stapler.Stapler Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler Handled by hudson.diagnosis.ReverseProxySetupMonitor.doTest(...) for url=/test/... Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler Handled by jenkins.model.Jenkins.getAdministrativeMonitor(String) for url=/administrativeMonitor/TOKEN/... Dec 15, 2015 3:04:02 PM FINE org.kohsuke.stapler.Stapler Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http:%252F%252Fjenkins.example.com%252Fmanage/ Dec 15, 2015 3:04:02 PM FINER org.kohsuke.stapler.Stapler Handled by jenkins.model.Jenkins.getAdministrativeMonitor(String) for url=/administrativeMonitor/TOKEN/... Dec 15, 2015 3:04:10 PM FINE org.kohsuke.stapler.Stapler Processing request for /log/stapler/ The response headers I got were HTTP/1.1 302 Found Date: Tue, 15 Dec 2015 04:04:02 GMT Server: Apache X-Content-Type-Options: nosniff Location: http://jenkins.example.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http%3A%2F%2Fjenkins.example.com%2Fmanage/ Content-Length: 0 HTTP/1.1 404 http://jenkins.example.com/manage vs. http:%2F%2Fjenkins.example.com%2Fmanage Date: Tue, 15 Dec 2015 04:04:02 GMT Server: Apache X-Content-Type-Options: nosniff Cache-Control: must-revalidate,no-cache,no-store Content-Length: 1646 Content-Type: text/html;charset=ISO-8859-1
          Hide
          danielbeck Daniel Beck added a comment -
          Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http:%252F%252Fjenkins.example.com%252Fmanage/

          As you can see, something in mod_jk appears to escape already escaped percent-encoded sequences (%25 is a percent-encoded % character).

          This is not a bug in Jenkins.

          My suggestion is to play around with these options:
          https://tomcat.apache.org/connectors-doc/reference/apache.html#Forwarding

          Show
          danielbeck Daniel Beck added a comment - Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http:%252F%252Fjenkins.example.com%252Fmanage/ As you can see, something in mod_jk appears to escape already escaped percent-encoded sequences (%25 is a percent-encoded % character). This is not a bug in Jenkins. My suggestion is to play around with these options: https://tomcat.apache.org/connectors-doc/reference/apache.html#Forwarding
          Hide
          xipmox Vince Murphy added a comment -

          Thanks for your analysis Daniel. Happy that it's not a bug in jenkins, but I would like to improve the documentation. I guess I should open a new bug for that.

          For completeness:
          I tried changing the JkOptions (in the jenkins vhost, rather than globally) but anything besides the default i.e.

                                                                                
          JkOptions +ForwardURIProxy                                                      
          

          results in a 500 error from apache.

          When I try to run the curl test with one of the other ForwardURI options I get this in mod_jk.log:

                                                                                
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] find_match::jk_uri_worker_map.c (941): Attempting to map context URI '/*=jenkins_worker' source 'JkMount'
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] find_match::jk_uri_worker_map.c (954): Found a wildchar match '/*=jenkins_worker'
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] jk_handler::mod_jk.c (2635): Into handler jakarta-servlet worker=jenkins_worker r->proxyreq=0
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] wc_get_worker_for_name::jk_worker.c (115): found a worker jenkins_worker
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] wc_get_name_for_type::jk_worker.c (292): Found worker type 'ajp13'
          [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [error] jk_handler::mod_jk.c (2747): Could not init service for worker=jenkins_worker
          

          I suspect those errors are yet another bug related to option-merging in mod_jk (I'm using 1.2.37-1), though I admit I don't see how. The configuration works fine if I change only the JkOption (back to ForwardURIProxy). I suspect the unexpected escaping is a bug there too.

          Show
          xipmox Vince Murphy added a comment - Thanks for your analysis Daniel. Happy that it's not a bug in jenkins, but I would like to improve the documentation. I guess I should open a new bug for that. For completeness: I tried changing the JkOptions (in the jenkins vhost, rather than globally) but anything besides the default i.e. JkOptions +ForwardURIProxy results in a 500 error from apache. When I try to run the curl test with one of the other ForwardURI options I get this in mod_jk.log: [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] find_match::jk_uri_worker_map.c (941): Attempting to map context URI '/*=jenkins_worker' source 'JkMount' [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] find_match::jk_uri_worker_map.c (954): Found a wildchar match '/*=jenkins_worker' [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] jk_handler::mod_jk.c (2635): Into handler jakarta-servlet worker=jenkins_worker r->proxyreq=0 [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] wc_get_worker_for_name::jk_worker.c (115): found a worker jenkins_worker [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [debug] wc_get_name_for_type::jk_worker.c (292): Found worker type 'ajp13' [Wed Dec 16 17:26:08 2015] [13650:140312934102848] [error] jk_handler::mod_jk.c (2747): Could not init service for worker=jenkins_worker I suspect those errors are yet another bug related to option-merging in mod_jk (I'm using 1.2.37-1), though I admit I don't see how. The configuration works fine if I change only the JkOption (back to ForwardURIProxy). I suspect the unexpected escaping is a bug there too.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            xipmox Vince Murphy
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: