Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63913

Jenkins controller starts slowly on 1 GB Pi 3 with SD card

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The startup time on a almost vanilla install is incredible slow. 

      I am running the jenkins server on a raspberry pi 3 on a SD card. Yes, the flash is slow. Yes, the pi is not strong. But 30 minutes for a basic setup without work being done is not acceptable.

      jenkins version: 2.222.3

      with the following installed plugins: 

       

      plugins:
        - artifactId: configuration-as-code
           source:
             version: 1.39
         - artifactId: ssh-slaves
          source:
            version: 1.31.2
        - artifactId: jdk-tool
          source:
            version: 1.4
        - artifactId: git
          source:
            version: 4.2.2
        - artifactId: job-dsl
          source:
            version: 1.77
        - artifactId: blueocean
          source:
            version: 1.23.1
        - artifactId: ws-cleanup
          source:
            version: 0.38
        - artifactId: docker
          source:
            version: 1.2.0
        - artifactId: credentials-binding
          source:
            version: 1.23
        - artifactId: token-macro
          source:
            version: 2.12
        - artifactId: matrix-project
          source:
            version: 1.16

       

       

      Here is my startup log (sorry there is no option to disable line breaks...): It takes time from 20:45 to 21:08 for stuff related to "jenkins.InitReactorRunner$1#onAttained". 

      jenkins | 2020-10-11 20:45:16.578+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @104040ms to org.eclipse.jetty.util.log.JavaUtilLog jenkins | 2020-10-11 20:45:18.299+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file jenkins | 2020-10-11 20:45:19.640+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 1.8.0_212-b04 jenkins | 2020-10-11 20:45:31.179+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet jenkins | 2020-10-11 20:45:32.575+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0 jenkins | 2020-10-11 20:45:32.577+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults jenkins | 2020-10-11 20:45:32.609+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms jenkins | 2020-10-11 20:45:56.463+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME") jenkins | 2020-10-11 20:45:59.002+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@b91815{Jenkins v2.222.3,/jenkins,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war} jenkins | 2020-10-11 20:45:59.207+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@115b42e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} jenkins | 2020-10-11 20:45:59.210+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @146676ms jenkins | 2020-10-11 20:45:59.213+0000 [id=18] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled jenkins | 2020-10-11 20:58:16.457+0000 [id=34] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization jenkins | 2020-10-11 20:58:37.969+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins jenkins | 2020-10-11 21:07:26.515+0000 [id=39] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins jenkins | 2020-10-11 21:07:26.734+0000 [id=40] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins jenkins | 2020-10-11 21:08:42.902+0000 [id=34] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions jenkins | 2020-10-11 21:08:50.733+0000 [id=33] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect buildmachine jenkins | 2020-10-11 21:08:50.996+0000 [id=38] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded jenkins | [10/11/20 21:11:12] SSH Launch of ...

       

      ===================

      Here is an update. The startup time is now acceptable 2 minutes. On better hardware it should be even better.

      Changes made on the same hardware:

      • Updated jenkins from 2.222 to 2.249.
      • Updated java on master and agent from java 8 to java 11.
      • Updated docker base image for jenkins from alpine to debian buster slim.

       

      jenkins             | Running from: /usr/share/jenkins/jenkins.warjenkins             | Running from: /usr/share/jenkins/jenkins.warjenkins             | webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")jenkins             | 2020-10-13 20:40:11.583+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @4738ms to org.eclipse.jetty.util.log.JavaUtilLogjenkins             | 2020-10-13 20:40:12.196+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war filejenkins             | 2020-10-13 20:40:24.664+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 11.0.8+10-post-Debian-1deb10u1jenkins             | 2020-10-13 20:40:27.766+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServletjenkins             | 2020-10-13 20:40:28.930+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0jenkins             | 2020-10-13 20:40:28.931+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaultsjenkins             | 2020-10-13 20:40:28.955+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000msjenkins             | WARNING: An illegal reflective access operation has occurredjenkins             | WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/var/jenkins_home/war/WEB-INF/lib/xstream-1.4.7-jenkins-1.jar) to field java.util.TreeMap.comparatorjenkins             | WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fieldsjenkins             | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsjenkins             | WARNING: All illegal access operations will be denied in a future releasejenkins             | 2020-10-13 20:40:31.615+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")jenkins             | 2020-10-13 20:40:33.750+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@1592946{Jenkins v2.249.2,/jenkins,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}jenkins             | 2020-10-13 20:40:33.911+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@17ff809{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}jenkins             | 2020-10-13 20:40:33.914+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @27087msjenkins             | 2020-10-13 20:40:33.948+0000 [id=22] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabledjenkins             | 2020-10-13 20:40:36.256+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Started initializationjenkins             | 2020-10-13 20:40:55.723+0000 [id=27] INFO jenkins.InitReactorRunner$1#onAttained: Listed all pluginsjenkins             | 2020-10-13 20:41:46.936+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all pluginsjenkins             | 2020-10-13 20:41:46.985+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started all pluginsjenkins             | 2020-10-13 20:41:55.928+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensionsjenkins             | 2020-10-13 20:41:57.060+0000 [id=32] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect buildmachinejenkins             | 2020-10-13 20:41:57.101+0000 [id=34] INFO jenkins.InitReactorRunner$1#onAttained: System config loadedjenkins             | 2020-10-13 20:42:11.731+0000 [id=28] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@14c33f7: display name [Root WebApplicationContext]; startup date [Tue Oct 13 20:42:11 UTC 2020]; root of context hierarchyjenkins             | 2020-10-13 20:42:11.733+0000 [id=28] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@14c33f7]: org.springframework.beans.factory.support.DefaultListableBeanFactory@cc97d6jenkins             | 2020-10-13 20:42:11.816+0000 [id=28] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@cc97d6: defining beans [authenticationManager]; root of factory hierarchyjenkins             | 2020-10-13 20:42:13.089+0000 [id=28] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@15eef08: display name [Root WebApplicationContext]; startup date [Tue Oct 13 20:42:13 UTC 2020]; root of context hierarchyjenkins             | 2020-10-13 20:42:13.091+0000 [id=28] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@15eef08]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1029f25jenkins             | 2020-10-13 20:42:13.095+0000 [id=28] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1029f25: defining beans [filter,legacy]; root of factory hierarchyjenkins             | [10/13/20 20:42:15] SSH Launch of buildmachine on 192.168.2.220 completed in 17,262 ms

       

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment - - edited

          Jenkins does not target 1 GB computers with slow file systems. That is not an area where the project has focused. I don't expect the project will ever focus there.

          I love my Raspberry Pi agents because they remind me that we can remain platform portable. However, they are agents connected to a larger machine as the controller.

          I agree that a 30 minute startup time is not acceptable. The solution is to not use a Raspberry Pi 3 with 1 GB of memory and an SD card file system to run a Jenkins controller.

          I'm able to start a minimal Jenkins 2.249.2 on my Raspberry Pi 2B from 2015 in about 4 minutes. That is to reach the installation wizard only and is not loading any plugins.

          processor       : 0
          model name      : ARMv7 Processor rev 5 (v7l)
          BogoMIPS        : 38.40
          Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
          CPU implementer : 0x41
          CPU architecture: 7
          CPU variant     : 0x0
          CPU part        : 0xc07
          CPU revision    : 5
          

          My Raspberry Pi 4 with 4 GB of RAM reaches that same point in less than 80 seconds. Still not a computer that we're actively targeting to run a Jenkins controller, since the SD card file system is known to be slow and Jenkins depends on a high performance file system.

          I was running JDK 11.0.8 in both those tests as provided by Raspbian 10.

          We saw similar slow performance when running OpenJDK 8 on an IBM s390x mainframe. It includes "Zero VM". Thankfully, OpenJDK 11 on IBM s390x and on ARM32 both use hotspot just in time compiler.

          Show
          markewaite Mark Waite added a comment - - edited Jenkins does not target 1 GB computers with slow file systems. That is not an area where the project has focused. I don't expect the project will ever focus there. I love my Raspberry Pi agents because they remind me that we can remain platform portable. However, they are agents connected to a larger machine as the controller. I agree that a 30 minute startup time is not acceptable. The solution is to not use a Raspberry Pi 3 with 1 GB of memory and an SD card file system to run a Jenkins controller. I'm able to start a minimal Jenkins 2.249.2 on my Raspberry Pi 2B from 2015 in about 4 minutes. That is to reach the installation wizard only and is not loading any plugins. processor : 0 model name : ARMv7 Processor rev 5 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc07 CPU revision : 5 My Raspberry Pi 4 with 4 GB of RAM reaches that same point in less than 80 seconds. Still not a computer that we're actively targeting to run a Jenkins controller, since the SD card file system is known to be slow and Jenkins depends on a high performance file system. I was running JDK 11.0.8 in both those tests as provided by Raspbian 10. We saw similar slow performance when running OpenJDK 8 on an IBM s390x mainframe. It includes "Zero VM". Thankfully, OpenJDK 11 on IBM s390x and on ARM32 both use hotspot just in time compiler.
          Hide
          rusche Thomas Rusche added a comment -

          I've made an update. With my changes I reduced the time from 30 minutes to 2 minutes, on the same hardware.

          I assume there was an issue with the java version...

          Show
          rusche Thomas Rusche added a comment - I've made an update. With my changes I reduced the time from 30 minutes to 2 minutes, on the same hardware. I assume there was an issue with the java version...
          Hide
          markewaite Mark Waite added a comment -

          I wonder if your first attempt was inadvertently made with a Java Virtual Machine using the Zero VM rather than the hotspot VM. I made the mistake of using Java 8 for arm32 that I downloaded from AdoptOpenJDK. The java -version output for that version reports:

          $ bin/java -version
          openjdk version "1.8.0_265"
          OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_265-b01)
          OpenJDK Zero VM (AdoptOpenJDK)(build 25.265-b01, interpreted mode)
          

          When I use the JVM included with Raspbian 10, it reports:

          $ /usr/lib/jvm/java-8-openjdk-armhf/bin/java -version
          openjdk version "1.8.0_212"
          OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1+rpi1-b01)
          OpenJDK Client VM (build 25.212-b01, mixed mode)
          

          The "Zero VM" has no just in time compiler. The "Client VM" uses the hotspot just in time compiler.

          Show
          markewaite Mark Waite added a comment - I wonder if your first attempt was inadvertently made with a Java Virtual Machine using the Zero VM rather than the hotspot VM. I made the mistake of using Java 8 for arm32 that I downloaded from AdoptOpenJDK . The java -version output for that version reports: $ bin/java -version openjdk version "1.8.0_265" OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_265-b01) OpenJDK Zero VM (AdoptOpenJDK)(build 25.265-b01, interpreted mode) When I use the JVM included with Raspbian 10, it reports: $ /usr/lib/jvm/java-8-openjdk-armhf/bin/java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1+rpi1-b01) OpenJDK Client VM (build 25.212-b01, mixed mode) The "Zero VM" has no just in time compiler. The "Client VM" uses the hotspot just in time compiler.
          Hide
          markewaite Mark Waite added a comment -

          I propose that we close this issue. I don't think that the Jenkins project intends to focus on reducing the startup time of Jenkins on computers with small amounts of memory and slow file systems.

          Show
          markewaite Mark Waite added a comment - I propose that we close this issue. I don't think that the Jenkins project intends to focus on reducing the startup time of Jenkins on computers with small amounts of memory and slow file systems.
          Hide
          rusche Thomas Rusche added a comment -

          Feel free to close it. Thanks anyway

          Show
          rusche Thomas Rusche added a comment - Feel free to close it. Thanks anyway
          Hide
          markewaite Mark Waite added a comment -

          Thomas Rusche resolved the issue by using a different Java VM.

          Show
          markewaite Mark Waite added a comment - Thomas Rusche resolved the issue by using a different Java VM.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            rusche Thomas Rusche
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: