Versions 2.117 to 2.119 of Jenkins hangs when the root context is requested on Solaris 10. The UI hangs indefinitely and never returns. Nothing is reported in the logs that there is an issue. If upgrading a previous installation jobs will continue to run in the back ground normally, however this issue happens on a clean install.

      To reproduce download the generic WAR file of versions 2.117, 2.118, or 2.119, which is the latest as of this time. Run the jenkins.war file on Solaris 10. Once it is reported that Jenkins is fully up and running try and get to the home page. The page should hang with no response.

      This prevents Jenkins from being usable on the Solaris. Please let me know if I can provide anything else.

       

       

        1. JenkinsSuccess-2.139-SNAPSHOT.jpg
          JenkinsSuccess-2.139-SNAPSHOT.jpg
          25 kB
        2. jenkins-2.128_2018-08-17.log
          5.23 MB
        3. jenkins-2.121.3_2018-08-17.log
          4.94 MB
        4. Jenkins-2.128.startup.log
          8 kB
        5. jenkins-thread-dump.log
          87 kB
        6. jenkins.log
          4 kB

          [JENKINS-51136] Jenkins UI hangs 2.117 - 2.119

          Oleg Nenashev added a comment -

          laughingdev Running Jenkins on Solaris may be a problem for core maintainers. Are there any system logs? Please also enable stapler tracing and provide the report for newly opened pages. https://wiki.jenkins.io/display/JENKINS/Figuring+out+URL+binding+of+Stapler

          Oleg Nenashev added a comment - laughingdev Running Jenkins on Solaris may be a problem for core maintainers. Are there any system logs? Please also enable stapler tracing and provide the report for newly opened pages. https://wiki.jenkins.io/display/JENKINS/Figuring+out+URL+binding+of+Stapler

          Oleg Nenashev added a comment -

          BTW, I would bet that it is caused by Jetty upgrade in 2.117. https://jenkins.io/changelog/#v2.117
          FYI olamy

          Oleg Nenashev added a comment - BTW, I would bet that it is caused by Jetty upgrade in 2.117. https://jenkins.io/changelog/#v2.117 FYI olamy

          Oleg Nenashev added a comment -

          Just to make sure, does it happen on versions before 2.117?

          Oleg Nenashev added a comment - Just to make sure, does it happen on versions before 2.117?

          Olivier Lamy added a comment -

          could help to have a threaddump collected or logs when the server hangs.

          Olivier Lamy added a comment - could help to have a threaddump collected or logs when the server hangs.

          The Jenkins log and thread dump are attached. It does not happen on versions before 2.117. 2.116 is working fine for right now. I could not get the response headers for the stapler tracing as no response is ever returned to the browser.

          I appreciate the help.

          Justin Carpenter added a comment - The Jenkins log and thread dump are attached. It does not happen on versions before 2.117. 2.116 is working fine for right now. I could not get the response headers for the stapler tracing as no response is ever returned to the browser. I appreciate the help.

          Mike Scholze added a comment - - edited

          Like I wrote here: https://issues.jenkins-ci.org/browse/JENKINS-33412

          Our docker hosts have 80 cores, with version 2.117 we could not reach jenkins via http, 2.116 works fine. On my local test system with 4 cores everything works fine.

          Maybe thats the reason:
          --> Update Winstone from 4.1.2 to 4.2 to update Jetty from 9.4.5 to 9.4.8 for various bugfixes and improvements. (full changelogJetty 9.4.6 changelogJetty 9.4.7 changelogJetty 9.4.8 changelog)

          Mike Scholze added a comment - - edited Like I wrote here: https://issues.jenkins-ci.org/browse/JENKINS-33412 Our docker hosts have 80 cores, with version 2.117 we could not reach jenkins via http, 2.116 works fine. On my local test system with 4 cores everything works fine. Maybe thats the reason: --> Update Winstone from 4.1.2 to 4.2 to update Jetty from 9.4.5 to 9.4.8 for various bugfixes and improvements. ( full changelog ,  Jetty 9.4.6 changelog ,  Jetty 9.4.7 changelog ,  Jetty 9.4.8 changelog )

          Mike Scholze added a comment - - edited

          comment removed, sorry olamy

          Mike Scholze added a comment - - edited comment removed, sorry olamy

          Olivier Lamy added a comment -

          can you try with 2.128?

          Please mikescholze do not hijack issues. here the issue might be related to solaris which is a different case from yours.

          Olivier Lamy added a comment - can you try with 2.128? Please mikescholze do not hijack issues. here the issue might be related to solaris which is a different case from yours.

          Marco Jacob added a comment -

          I have the same problem on Solaris 11 when switching from 2.107.3 to 2.121.3 today.
          I will give 2.128 a try.

           

          Marco Jacob added a comment - I have the same problem on Solaris 11 when switching from 2.107.3 to 2.121.3 today. I will give 2.128 a try.  

          Marco Jacob added a comment -

          Not working wih 2.128

          Jenkins-2.128.startup.log

          Marco Jacob added a comment - Not working wih 2.128 Jenkins-2.128.startup.log

          Marco Jacob added a comment - - edited

          Is there a chance to change winston to version from 2.107.3 in ear? I just put the old winston.jar in the new EAR but that doesn't work.

          Marco Jacob added a comment - - edited Is there a chance to change winston to version from 2.107.3 in ear? I just put the old winston.jar in the new EAR but that doesn't work.

          Olivier Lamy added a comment -

          marco_jacob thanks for the logs but this doesn't really help as there is no failure neither block in this logs.

          I need a threaddump when Jenkins UI hangs. Please use jstack for that.

          What OS are you using? I see you're using jdk 1.8.0_121-b13. Can you upgrade to recent one?

          Furthermore I need some jetty debug logs.

          You can have those using the following procedures:

          Create a file called jul.properties with the following content:

          handlers=java.util.logging.ConsoleHandler
          .level=INFO
          org.eclipse.jetty.level=ALL
          java.util.logging.ConsoleHandler.level=FINEST
           

          Then start jenkins with the sys prop:

          -Djava.util.logging.config.file=jul.properties 

          Olivier Lamy added a comment - marco_jacob thanks for the logs but this doesn't really help as there is no failure neither block in this logs. I need a threaddump when Jenkins UI hangs. Please use jstack for that. What OS are you using? I see you're using jdk 1.8.0_121-b13. Can you upgrade to recent one? Furthermore I need some jetty debug logs. You can have those using the following procedures: Create a file called jul.properties with the following content: handlers=java.util.logging.ConsoleHandler .level=INFO org.eclipse.jetty.level=ALL java.util.logging.ConsoleHandler.level=FINEST Then start jenkins with the sys prop: -Djava.util.logging.config.file=jul.properties

          Marco Jacob added a comment -

          Current OS is Solaris 11:

          > uname -a
          SunOS AP4S201 5.11 11.3 sun4v sparc sun4v

          Logfile with other log level and Thread Dump

          jenkins-2.121.3_2018-08-17.log

          Marco Jacob added a comment - Current OS is Solaris 11: > uname -a SunOS AP4S201 5.11 11.3 sun4v sparc sun4v Logfile with other log level and Thread Dump jenkins-2.121.3_2018-08-17.log

          Olivier Lamy added a comment - - edited

          marco_jacob  how do you start Jenkins? do you use special options to start your jvm?

          it looks you're using 1.8.0_121, can you try upgrading?

          80 GC threads is really high!

          are those logs generated with 2.128 or 2.138?

          because I can read

          Running from: /data/apps/jenkins/app/jenkins-2.121.3.war 

           

          Olivier Lamy added a comment - - edited marco_jacob   how do you start Jenkins? do you use special options to start your jvm? it looks you're using 1.8.0_121, can you try upgrading? 80 GC threads is really high! are those logs generated with 2.128 or 2.138? because I can read Running from: /data/apps/jenkins/app/jenkins-2.121.3.war  

          Marco Jacob added a comment - - edited

          We use LTS version and so I used 2.121.3.

          Start script:

          #!/bin/bash
          export LC_ALL="de_DE.ISO8859-15"
          export LANG="de_DE.ISO8859-15"
          export DTRACE_DOF_INIT_DISABLE=1
          export JAVA_HOME=/data/apps/java/jdk1.8
          # Parameter fuer Content Security Policy - siehe SEU-343
          export JENKINS_ADD_OPTION_1="-Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src 'none'; img-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self'; frame-src 'self';\""
          export JENKINS_JAVA_OPTIONS="-d64 -Djava.awt.headless=true -Xms4096m -Xmx4096m"
          export JENKINS_HOME=/data/apps/jenkins-home
          ulimit -n 4096
          nohup $JAVA_HOME/bin/java $JENKINS_JAVA_OPTIONS "$JENKINS_ADD_OPTION_1" -jar /data/apps/jenkins/app/jenkins-2.107.3.war --prefix=/ci > $JENKINS_HOME/jenkins.log 2>&1

          Marco Jacob added a comment - - edited We use LTS version and so I used 2.121.3. Start script: #!/bin/bash export LC_ALL="de_DE.ISO8859-15" export LANG="de_DE.ISO8859-15" export DTRACE_DOF_INIT_DISABLE=1 export JAVA_HOME=/data/apps/java/jdk1.8 # Parameter fuer Content Security Policy - siehe SEU-343 export JENKINS_ADD_OPTION_1="-Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src 'none'; img-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self'; frame-src 'self';\"" export JENKINS_JAVA_OPTIONS="-d64 -Djava.awt.headless=true -Xms4096m -Xmx4096m" export JENKINS_HOME=/data/apps/jenkins-home ulimit -n 4096 nohup $JAVA_HOME/bin/java $JENKINS_JAVA_OPTIONS "$JENKINS_ADD_OPTION_1" -jar /data/apps/jenkins/app/jenkins-2.107.3.war --prefix=/ci > $JENKINS_HOME/jenkins.log 2>&1

          Marco Jacob added a comment -

          Changed 2.107.3 to 2.121.3 with jetty parameter for the version not working.

          I downloaded last JDK version to give it a try.

          Marco Jacob added a comment - Changed 2.107.3 to 2.121.3 with jetty parameter for the version not working. I downloaded last JDK version to give it a try.

          Marco Jacob added a comment -

          Changed start-script to:

          #!/bin/bash
          export LC_ALL="de_DE.ISO8859-15"
          export LANG="de_DE.ISO8859-15"
          export DTRACE_DOF_INIT_DISABLE=1
          #export JAVA_HOME=/data/apps/java/jdk1.8
          export JAVA_HOME=/data/apps/java/jdk1.8.0_181
          # Parameter fuer Content Security Policy - siehe SEU-343
          export JENKINS_ADD_OPTION_1="-Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src 'none'; img-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self'; frame-src 'self';\""
          export JENKINS_JAVA_OPTIONS="-d64 -Djava.awt.headless=true -Xms4096m -Xmx4096m"
          export JENKINS_HOME=/data/apps/jenkins-home
          ulimit -n 4096
          nohup $JAVA_HOME/bin/java $JENKINS_JAVA_OPTIONS "$JENKINS_ADD_OPTION_1" -Djava.util.logging.config.file=/data/apps/scmadmin/jul.properties -jar /data/apps/jenkins/app/jenkins-2.128.war --prefix=/ci > $JENKINS_HOME/jenkins.log 2>&1

          New Logfile with 2.128 and JDK 1.8.0_181:

           jenkins-2.128_2018-08-17.log

          Marco Jacob added a comment - Changed start-script to: #!/bin/bash export LC_ALL="de_DE.ISO8859-15" export LANG="de_DE.ISO8859-15" export DTRACE_DOF_INIT_DISABLE=1 #export JAVA_HOME=/data/apps/java/jdk1.8 export JAVA_HOME=/data/apps/java/jdk1.8.0_181 # Parameter fuer Content Security Policy - siehe SEU-343 export JENKINS_ADD_OPTION_1="-Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src 'none'; img-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self'; frame-src 'self';\"" export JENKINS_JAVA_OPTIONS="-d64 -Djava.awt.headless=true -Xms4096m -Xmx4096m" export JENKINS_HOME=/data/apps/jenkins-home ulimit -n 4096 nohup $JAVA_HOME/bin/java $JENKINS_JAVA_OPTIONS "$JENKINS_ADD_OPTION_1" -Djava.util.logging.config.file=/data/apps/scmadmin/jul.properties -jar /data/apps/jenkins/app/jenkins-2.128.war --prefix=/ci > $JENKINS_HOME/jenkins.log 2>&1 New Logfile with 2.128 and JDK 1.8.0_181:   jenkins-2.128_2018-08-17.log

          Marco Jacob added a comment -

          I forgot to wrote that it is still not working with new JDK and Version 2.128.

          Please have a look at the last logfile.

          Marco Jacob added a comment - I forgot to wrote that it is still not working with new JDK and Version 2.128. Please have a look at the last logfile.

          Olivier Lamy added a comment -

          as 80 is a huge number. Can you try reducing the number of GC Threads?

          Maybe -XX:ParallelGCThreads=20?

          Olivier Lamy added a comment - as 80 is a huge number. Can you try reducing the number of GC Threads? Maybe -XX:ParallelGCThreads=20?

          Marco Jacob added a comment -

          I can do this, but that is normal for server jvm on solaris. 2.107.3 is also running with this settings without any problem.

          Any other suggestions?

          Marco Jacob added a comment - I can do this, but that is normal for server jvm on solaris. 2.107.3 is also running with this settings without any problem. Any other suggestions?

          Marco Jacob added a comment -

          No change with reduced amount of GC.

          Marco Jacob added a comment - No change with reduced amount of GC.

          Olivier Lamy added a comment - - edited

          marco_jacob

          or anyone having the issue can you please the war here  http://home.apache.org/~olamy/jenkins/

          use --help for other new jetty options 

            

          Olivier Lamy added a comment - - edited marco_jacob or anyone having the issue can you please the war here  http://home.apache.org/~olamy/jenkins/ use --help for other new jetty options    

          Marco Jacob added a comment -

          Back from weekend. I will try it out now.

          Marco Jacob added a comment - Back from weekend. I will try it out now.

          Marco Jacob added a comment - - edited

          Success! It works! Jenkins 2.139 up and running.

          Marco Jacob added a comment - - edited Success! It works! Jenkins 2.139 up and running.

          Olivier Lamy added a comment -

          Good to hear feel free to vote/comment for JENKINS-53239 or https://github.com/jenkinsci/winstone/pull/54

          Olivier Lamy added a comment - Good to hear feel free to vote/comment for  JENKINS-53239 or  https://github.com/jenkinsci/winstone/pull/54

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - https://github.com/jenkinsci/winstone/commit/74775cc02ef92feaf247e45a32b193e45800805a

            olamy Olivier Lamy
            laughingdev Justin Carpenter
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: