Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52804

Jenkins web GUI hangs when using a computer with more then 70 CPU's

      From version 2.117 jenkins web GUI hangs when using more then 70 cpus,  between 64 and 70 jenkins GUI is not updated correctly. Using wget instead of browser only to show what's failing, but using Chrome in real life

      Works fine (version 2.135 with 64 cpus)

      $docker run --cpuset-cpus=0-64 -p 8080:8080 -p 50000:50000 jenkins/jenkins:latest
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 22:52:42- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:8080... connected.
      HTTP request sent, awaiting response... 403 Forbidden
      2018-07-30 22:52:42 ERROR 403: Forbidden.

       

      Fails (Hangs forever in wget, version 2.135 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:latest
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 22:57:03- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::(1|:8080... connected.
      HTTP request sent, awaiting response...

      Works fine (version 2.116 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.116
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 23:01:12- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:(8080... connected.
      HTTP request sent, awaiting response... 403 Forbidden
      2018-07-30 23:01:12 ERROR 403: Forbidden.

       

      Fails (Hangs forever in wget) (version 2.117 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.117 
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 23:06:14- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:8080... connected.
      HTTP request sent, awaiting response...

       Most likely it's a race condition issue in the updated related to the

      "Update Winstone from 4.1.2 to 4.2 to update Jetty from 9.4.5 to 9.4.8"

        1. docker_test64.out
          2.88 MB
        2. docker_test70.out
          2.87 MB
        3. docker64.log
          12 kB
        4. docker70.log
          6 kB
        5. jenkins.log.save
          3.31 MB
        6. jenkins.threaddump
          77 kB
        7. jstack.out
          0.2 kB
        8. jstack.out.F
          65 kB

          [JENKINS-52804] Jenkins web GUI hangs when using a computer with more then 70 CPU's

          Olivier Lamy added a comment -

          well I cannot really see something wrong in the logs,

          jstack?

          what happen with env var or sys props 

          JETTY_AVAILABLE_PROCESSORS=32 (or smaller value)?

          Olivier Lamy added a comment - well I cannot really see something wrong in the logs, jstack? what happen with env var or sys props  JETTY_AVAILABLE_PROCESSORS=32 (or smaller value)?

          Anders Wallin added a comment -

          If I add JETTY_AVAILABLE_PROCESSORS=32 it works fine!

           

           

          $ cat docker_test
          #!/bin/bash
          set -x

          : ${TAG:=2.135}
          : ${MAXCPU:=64}

          cat data/jul.properties

          docker rm myjenkins
          docker run -name myjenkins --cpuset-cpus=0$MAXCPU -p 8080:8080 -p 50000:50000 \
          --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/jul.properties" \
          --env JETTY_AVAILABLE_PROCESSORS=32 \
          -v `pwd`/data:/var/jenkins_home jenkins/jenkins:$TAG

          $ MAXCPU=70 ./docker_test 2>&1 | tee docker_test70.out

           

          Anders Wallin added a comment - If I add JETTY_AVAILABLE_PROCESSORS=32 it works fine!     $ cat docker_test #!/bin/bash set -x : ${TAG:=2.135} : ${MAXCPU:=64} cat data/jul.properties docker rm myjenkins docker run - name myjenkins --cpuset-cpus=0 $MAXCPU -p 8080:8080 -p 50000:50000 \ --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/jul.properties" \ --env JETTY_AVAILABLE_PROCESSORS=32 \ -v `pwd`/data:/var/jenkins_home jenkins/jenkins:$TAG $ MAXCPU=70 ./docker_test 2>&1 | tee docker_test70.out  

          Olivier Lamy added a comment -

          is it possible to have a jstack when it hang?

          Olivier Lamy added a comment - is it possible to have a jstack when it hang?

          Eric Boehm added a comment -

          I tried adding JETTY_AVAILABLE_PROCESSORS=32 with 72 processors. It made no difference.

          I've attached my log. I am using the jul.properties example.

          Here's how Jenkins was started

          bash -x ./jenkins.sh start
          + DESC='Jenkins CI Server'
          + NAME=jenkins
          + JENKINS_USER=bcabuild
          + JENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime
          + JENKINS_PORT=8288
          + JENKINS_HOST=10.75.159.34
          + export JETTY_AVAILABLE_PROCESSORS=32
          + JETTY_AVAILABLE_PROCESSORS=32
          + LOGFILE=/opt/jenkins/mj-dashboard.new/jenkins.log
          + export JAVA_HOME=/usr/java/jdk1.8.0_144
          + JAVA_HOME=/usr/java/jdk1.8.0_144
          + JAVA=/usr/java/jdk1.8.0_144/bin/java
          + CMD='nohup /usr/java/jdk1.8.0_144/bin/java -Xss2048k -Xms4g -Xmx4g -XX:MaxPermSize=1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses -Djenkins.upstreamCulprits=true -Dhudson.upstreamCulprits=true -Dhudson.DNSMultiCast.disabled=true -Djava.util.logging.config.file=/opt/jenkins/mj-dashboard.new/jenkins_runtime/jul.properties -DJENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime/ -jar /opt/jenkins/mj-dashboard.new/jenkins_runtime/jenkins.war --httpPort=8288 --httpListenAddress=10.75.159.34 >> /opt/jenkins/mj-dashboard.new/jenkins.log 2>&1 &'
          + case $1 in
          + echo -n 'Starting Jenkins CI Server'
          Starting Jenkins CI Server+ d_start
          + ulimit -v 128000000
          + ulimit -c 200
          + ulimit -a
          core file size (blocks, -c) 200
          data seg size (kbytes, -d) unlimited
          scheduling priority (-e) 0
          file size (blocks, -f) unlimited
          pending signals (-i) 1549304
          max locked memory (kbytes, -l) 64
          max memory size (kbytes, -m) unlimited
          open files (-n) 1024
          pipe size (512 bytes, -p) 8
          POSIX message queues (bytes, -q) 819200
          real-time priority (-r) 0
          stack size (kbytes, -s) 10240
          cpu time (seconds, -t) unlimited
          max user processes (-u) 1549304
          virtual memory (kbytes, -v) 128000000
          file locks (-x) unlimited
          + /bin/su -p -c 'nohup /usr/java/jdk1.8.0_144/bin/java -Xss2048k -Xms4g -Xmx4g -XX:MaxPermSize=1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses -Djenkins.upstreamCulprits=true -Dhudson.upstreamCulprits=true -Dhudson.DNSMultiCast.disabled=true -Djava.util.logging.config.file=/opt/jenkins/mj-dashboard.new/jenkins_runtime/jul.properties -DJENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime/ -jar /opt/jenkins/mj-dashboard.new/jenkins_runtime/jenkins.war --httpPort=8288 --httpListenAddress=10.75.159.34 >> /opt/jenkins/mj-dashboard.new/jenkins.log 2>&1 &' bcabuild

          The log shows that I started running

          curl -I 'http://10.75.159.34:8288/login'

          at 04:41:24

          FINE: isSystemResource==false hudson.slaves.NodeProvisioner$2 jar:file:/opt/jenkins/mj-dashboard.new/jenkins_runtime/war/WEB-INF/lib/jenkins-core-2.117.jar!/hudson/slaves/NodeProvisioner$2.class
          Aug 24, 2018 4:39:59 AM org.eclipse.jetty.webapp.WebAppClassLoader loadClass
          FINE: WAP webapp loaded class hudson.slaves.NodeProvisioner$2
          Aug 24, 2018 4:41:24 AM org.eclipse.jetty.io.ManagedSelector submit
          FINE: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@3ce973aa on org.eclipse.jetty.io.ManagedSelector@4c2bb6e0 id=7 keys=0 selected=0 actions=0

          I finally killed it 5 minutes later

          I then tried to access it from a web browser (Chrome, http://10.75.159.34:8288/login). I let it run for 5 minutes, starting at 04:47:12

          I also saw the request for a jstack.

          I ran the following

          jstack 34908 > jstack.out 2>&1

          jstack -F 34908 > jstack.out.F 2>&1

          cp jenkins.log jenkins.log.save

          cat /dev/null > jenkins.log

          kill -3 34980

          cp jenkins.log jenkins.threaddump

          I've attached jenkins.log.save, jenkins.threaddump, jstack.out, jstack.out.F

           

          Eric Boehm added a comment - I tried adding JETTY_AVAILABLE_PROCESSORS=32 with 72 processors. It made no difference. I've attached my log. I am using the jul.properties example. Here's how Jenkins was started bash -x ./jenkins.sh start + DESC='Jenkins CI Server' + NAME=jenkins + JENKINS_USER=bcabuild + JENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime + JENKINS_PORT=8288 + JENKINS_HOST=10.75.159.34 + export JETTY_AVAILABLE_PROCESSORS=32 + JETTY_AVAILABLE_PROCESSORS=32 + LOGFILE=/opt/jenkins/mj-dashboard.new/jenkins.log + export JAVA_HOME=/usr/java/jdk1.8.0_144 + JAVA_HOME=/usr/java/jdk1.8.0_144 + JAVA=/usr/java/jdk1.8.0_144/bin/java + CMD='nohup /usr/java/jdk1.8.0_144/bin/java -Xss2048k -Xms4g -Xmx4g -XX:MaxPermSize=1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses -Djenkins.upstreamCulprits=true -Dhudson.upstreamCulprits=true -Dhudson.DNSMultiCast.disabled=true -Djava.util.logging.config.file=/opt/jenkins/mj-dashboard.new/jenkins_runtime/jul.properties -DJENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime/ -jar /opt/jenkins/mj-dashboard.new/jenkins_runtime/jenkins.war --httpPort=8288 --httpListenAddress=10.75.159.34 >> /opt/jenkins/mj-dashboard.new/jenkins.log 2>&1 &' + case $1 in + echo -n 'Starting Jenkins CI Server' Starting Jenkins CI Server+ d_start + ulimit -v 128000000 + ulimit -c 200 + ulimit -a core file size (blocks, -c) 200 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1549304 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1549304 virtual memory (kbytes, -v) 128000000 file locks (-x) unlimited + /bin/su -p -c 'nohup /usr/java/jdk1.8.0_144/bin/java -Xss2048k -Xms4g -Xmx4g -XX:MaxPermSize=1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses -Djenkins.upstreamCulprits=true -Dhudson.upstreamCulprits=true -Dhudson.DNSMultiCast.disabled=true -Djava.util.logging.config.file=/opt/jenkins/mj-dashboard.new/jenkins_runtime/jul.properties -DJENKINS_HOME=/opt/jenkins/mj-dashboard.new/jenkins_runtime/ -jar /opt/jenkins/mj-dashboard.new/jenkins_runtime/jenkins.war --httpPort=8288 --httpListenAddress=10.75.159.34 >> /opt/jenkins/mj-dashboard.new/jenkins.log 2>&1 &' bcabuild The log shows that I started running curl -I ' http://10.75.159.34:8288/login' at 04:41:24 FINE: isSystemResource==false hudson.slaves.NodeProvisioner$2 jar: file:/opt/jenkins/mj-dashboard.new/jenkins_runtime/war/WEB-INF/lib/jenkins-core-2.117.jar!/hudson/slaves/NodeProvisioner$2.class Aug 24, 2018 4:39:59 AM org.eclipse.jetty.webapp.WebAppClassLoader loadClass FINE: WAP webapp loaded class hudson.slaves.NodeProvisioner$2 Aug 24, 2018 4:41:24 AM org.eclipse.jetty.io.ManagedSelector submit FINE: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@3ce973aa on org.eclipse.jetty.io.ManagedSelector@4c2bb6e0 id=7 keys=0 selected=0 actions=0 I finally killed it 5 minutes later I then tried to access it from a web browser (Chrome, http://10.75.159.34:8288/login ). I let it run for 5 minutes, starting at 04:47:12 I also saw the request for a jstack. I ran the following jstack 34908 > jstack.out 2>&1 jstack -F 34908 > jstack.out.F 2>&1 cp jenkins.log jenkins.log.save cat /dev/null > jenkins.log kill -3 34980 cp jenkins.log jenkins.threaddump I've attached jenkins.log.save, jenkins.threaddump, jstack.out, jstack.out.F  

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - pr  https://github.com/jenkinsci/winstone/pull/54

          Olivier Lamy added a comment - - edited

          em_boehm

          or anyone having the issue can you please the war here  http://home.apache.org/~olamy/jenkins/

          use --help for other new jetty options

            

          Olivier Lamy added a comment - - edited em_boehm or anyone having the issue can you please the war here  http://home.apache.org/~olamy/jenkins/ use --help for other new jetty options   

          Eric Boehm added a comment -

          Olivier,

          Yes, that does seem to fix the problems. I am able to connect with curl and the browser with all 72 processors enabled using http://home.apache.org/~olamy/jenkins/jenkins_winstone_pr_54.war

           

           

          Eric Boehm added a comment - Olivier, Yes, that does seem to fix the problems. I am able to connect with curl and the browser with all 72 processors enabled using http://home.apache.org/~olamy/jenkins/jenkins_winstone_pr_54.war    

          Olivier Lamy added a comment -

          Good to hear feel free to vote/comment for JENKINS-53239 or https://github.com/jenkinsci/winstone/pull/54

          Olivier Lamy added a comment - Good to hear feel free to vote/comment for  JENKINS-53239 or  https://github.com/jenkinsci/winstone/pull/54

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - https://github.com/jenkinsci/winstone/commit/74775cc02ef92feaf247e45a32b193e45800805a

          Daniel Beck added a comment -

          Released on 2.141.

          Daniel Beck added a comment - Released on 2.141.

            olamy Olivier Lamy
            wallinux Anders Wallin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: