Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52804

Jenkins web GUI hangs when using a computer with more then 70 CPU's

      From version 2.117 jenkins web GUI hangs when using more then 70 cpus,  between 64 and 70 jenkins GUI is not updated correctly. Using wget instead of browser only to show what's failing, but using Chrome in real life

      Works fine (version 2.135 with 64 cpus)

      $docker run --cpuset-cpus=0-64 -p 8080:8080 -p 50000:50000 jenkins/jenkins:latest
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 22:52:42- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:8080... connected.
      HTTP request sent, awaiting response... 403 Forbidden
      2018-07-30 22:52:42 ERROR 403: Forbidden.

       

      Fails (Hangs forever in wget, version 2.135 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:latest
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 22:57:03- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::(1|:8080... connected.
      HTTP request sent, awaiting response...

      Works fine (version 2.116 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.116
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 23:01:12- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:(8080... connected.
      HTTP request sent, awaiting response... 403 Forbidden
      2018-07-30 23:01:12 ERROR 403: Forbidden.

       

      Fails (Hangs forever in wget) (version 2.117 with 70 cpus)

      $docker run --cpuset-cpus=0-70 -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.117 
      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

      ....

      INFO: Jenkins is fully up and running
      --> setting agent port for jnlp
      --> setting agent port for jnlp... done

      $ wget localhost:8080
      -2018-07-30 23:06:14- http://localhost:8080/
      Resolving localhost (localhost)... ::1, 127.0.0.1
      Connecting to localhost (localhost)|::1|:8080... connected.
      HTTP request sent, awaiting response...

       Most likely it's a race condition issue in the updated related to the

      "Update Winstone from 4.1.2 to 4.2 to update Jetty from 9.4.5 to 9.4.8"

        1. jenkins.log.save
          3.31 MB
        2. jenkins.threaddump
          77 kB
        3. jstack.out
          0.2 kB
        4. jstack.out.F
          65 kB
        5. docker_test64.out
          2.88 MB
        6. docker_test70.out
          2.87 MB
        7. docker70.log
          6 kB
        8. docker64.log
          12 kB

          [JENKINS-52804] Jenkins web GUI hangs when using a computer with more then 70 CPU's

          Anders Wallin created issue -

          Oleg Nenashev added a comment -

          CC olamy

           

          Oleg Nenashev added a comment - CC olamy  

          Olivier Lamy added a comment -

          any logs? because without it's impossible to find the trouble...
          Furthermore I'm interested to know the result of

          Runtime.getRuntime().availableProcessors()

          when you use

          --cpuset-cpus=0-64 or --cpuset-cpus=0-70

          .

          Olivier Lamy added a comment - any logs? because without it's impossible to find the trouble... Furthermore I'm interested to know the result of Runtime .getRuntime().availableProcessors() when you use --cpuset-cpus=0-64 or --cpuset-cpus=0-70 .

          Anders Wallin added a comment -

          Runtime.getRuntime().availableProcessors()

           

          On native host;

          $ java MyTest
          Number of cores available : 88

           

          Docker with --cpuset-cpus=0-70

          $ java MyTest
          Number of cores available : 71

           

          Docker with --cpuset-cpus=0-64

          $ java MyTest
          Number of cores available : 65

           

          Anders Wallin added a comment - Runtime.getRuntime().availableProcessors()   On native host; $ java MyTest Number of cores available : 88   Docker with --cpuset-cpus=0-70 $ java MyTest Number of cores available : 71   Docker with --cpuset-cpus=0-64 $ java MyTest Number of cores available : 65  
          Anders Wallin made changes -
          Attachment New: docker64.log [ 43556 ]
          Anders Wallin made changes -
          Attachment New: docker70.log [ 43557 ]

          Anders Wallin added a comment -

          Log files appended

          Anders Wallin added a comment - Log files appended

          Olivier Lamy added a comment -

          Sorry I didn't give you much details on what do I mean with logs...
          I need more jetty logs
          create a jul.properties file containing:

          handlers=java.util.logging.ConsoleHandler
          .level=INFO
          org.eclipse.jetty.level=ALL
          java.util.logging.ConsoleHandler.level=FINEST 
          

          Then start jenkins adding option:

          -Djava.util.logging.config.file=jul.properties

          Olivier Lamy added a comment - Sorry I didn't give you much details on what do I mean with logs... I need more jetty logs create a jul.properties file containing: handlers=java.util.logging.ConsoleHandler .level=INFO org.eclipse.jetty.level=ALL java.util.logging.ConsoleHandler.level=FINEST Then start jenkins adding option: -Djava.util.logging.config.file=jul.properties

          Anders Wallin added a comment -

          With better logs;

          I run the following script;

          $cat docker_test

          #!/bin/bash
          set -x

          : ${TAG:=2.135}
          : ${MAXCPU:=64}

          cat data/jul.properties

          docker rm myjenkins
          docker run -name myjenkins --cpuset-cpus=0$MAXCPU -p 8080:8080 -p 50000:50000 --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/jul.properties" -v `pwd`/data:/var/jenkins_home jenkins/jenkins:$TAG

          $MAXCPU=70 ./docker_test 2>&1 | tee docker_test70.out

          and when it's started I run "wget localhost:8080" from another shell

           

           

           

          Anders Wallin added a comment - With better logs; I run the following script; $cat docker_test #!/bin/bash set -x : ${TAG:=2.135} : ${MAXCPU:=64} cat data/jul.properties docker rm myjenkins docker run - name myjenkins --cpuset-cpus=0 $MAXCPU -p 8080:8080 -p 50000:50000 --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/jul.properties" -v `pwd`/data:/var/jenkins_home jenkins/jenkins:$TAG $MAXCPU=70 ./docker_test 2>&1 | tee docker_test70.out and when it's started I run "wget localhost:8080" from another shell      
          Anders Wallin made changes -
          Attachment New: docker_test70.out [ 43569 ]

            olamy Olivier Lamy
            wallinux Anders Wallin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: