Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69478

After 2.346.3 upgrade: Native memory allocation (malloc) failed to allocate 536870928 bytes for Chunk::new

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • Jenkins 2.346.3 running from jenkins/jenkins:2.346.3-almalinux Docker image
      OpenJDK Runtime Environment Temurin-11.0.16+8 (11.0.16+8)
      All plugins updated

      After updating master and the plugins, running under very light load (finishing one job running on Kubernetes agent) Java VM running Jenkins controller panicked:

      From jenkins.log (attachment contains complete log since Jenkins start):

      jenkins    | 2022-08-23 10:18:03.280+0000 [id=808]      INFO    o.c.j.p.k.p.ContainerExecDecorator$1#doLaunch: Created process inside pod: [fus-c67b406c-54a2-488a-8198-265e0e50fa5f-rg5dr-qmtgz], container: [fus-base][573 ms]
      jenkins    | 2022-08-23 10:18:06.237+0000 [id=792]      INFO    o.j.p.workflow.job.WorkflowRun#finish: fss/5 #11454 completed: SUCCESS
      jenkins    | #
      jenkins    | # There is insufficient memory for the Java Runtime Environment to continue.
      jenkins    | # Native memory allocation (malloc) failed to allocate 536870928 bytes for Chunk::new
      jenkins    | # An error report file with more information is saved as:
      jenkins    | # /tmp/hs_err_pid5.log
      jenkins    | [thread 1095 also had an error][thread 907 also had an error]
      jenkins    | 
      jenkins    | [thread 1028 also had an error]
      jenkins    | [thread 928 also had an error]
      jenkins    | 
      jenkins    | [timeout occurred during error reporting in step ""] after 30 s.
      jenkins    | Running from: /usr/share/jenkins/jenkins.war
      jenkins    | webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
      jenkins    | 2022-08-23 10:18:44.758+0000 [id=1]        INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @995ms to org.eclipse.jetty.util.log.JavaUtilLog
      jenkins    | 2022-08-23 10:18:44.893+0000 [id=1]        INFO    winstone.Logger#logInternal: Beginning extraction from war file
      jenkins    | 2022-08-23 10:18:44.954+0000 [id=1]        WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
      

      hs_err_pid5.log

      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (malloc) failed to allocate 536870928 bytes for Chunk::new
      # Possible reasons:
      #   The system is out of physical RAM or swap space
      #   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
      # Possible solutions:
      #   Reduce memory load on the system
      #   Increase physical memory or swap space
      #   Check if swap backing store is full
      #   Decrease Java heap size (-Xmx/-Xms)
      #   Decrease number of Java threads
      #   Decrease Java thread stack sizes (-Xss)
      #   Set larger code cache with -XX:ReservedCodeCacheSize=
      #   JVM is running with Zero Based Compressed Oops mode in which the Java heap is
      #     placed in the first 32GB address space. The Java Heap base address is the
      #     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
      #     to set the Java Heap base and to place the Java Heap above 32GB virtual address.
      # This output file may be truncated or incomplete.
      #
      #  Out of Memory Error (arena.cpp:197), pid=5, tid=28
      #
      # JRE version: OpenJDK Runtime Environment Temurin-11.0.16+8 (11.0.16+8) (build 11.0.16+8)
      # Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.16+8 (11.0.16+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
      # Core dump will be written. Default location: //core.5
      #
      
      

       

          [JENKINS-69478] After 2.346.3 upgrade: Native memory allocation (malloc) failed to allocate 536870928 bytes for Chunk::new

          Basil Crow added a comment -

          You do not appear to have read the 2.346.3 upgrade guide prior to filing this ticket. This issue is addressed by the first paragraph in that upgrade guide:

          OpenJDK 11.0.16/17.0.4 and the C2 JIT compiler

          OpenJDK 11.0.16 and 17.0.4 are susceptible to JDK-8292260, a regression in the C2 JIT compiler which may cause the JVM to crash unpredictably. The OpenJDK project has released fixes in 11.0.16.1 and 17.0.4.1. As of August 26, 2022, the official Jenkins Docker images have been updated with OpenJDK 11.0.16.1 and 17.0.4.1 in version 2.346.3-2.

          The evidence that you are hitting this issue is that the stack trace for the "C2 CompilerThread0" daemon in your hs_err_pid5.log file matches the one in JDK-8292260.

          Upgrade to version 2.346.3-2 of the official Jenkins Docker image with OpenJDK 11.0.16.1 or 17.0.4.1.

          Basil Crow added a comment - You do not appear to have read the 2.346.3 upgrade guide prior to filing this ticket. This issue is addressed by the first paragraph in that upgrade guide: OpenJDK 11.0.16/17.0.4 and the C2 JIT compiler OpenJDK 11.0.16 and 17.0.4 are susceptible to JDK-8292260 , a regression in the C2 JIT compiler which may cause the JVM to crash unpredictably . The OpenJDK project has released fixes in 11.0.16.1 and 17.0.4.1. As of August 26, 2022, the official Jenkins Docker images have been updated with OpenJDK 11.0.16.1 and 17.0.4.1 in version 2.346.3-2. The evidence that you are hitting this issue is that the stack trace for the "C2 CompilerThread0 " daemon in your hs_err_pid5.log file matches the one in JDK-8292260 . Upgrade to version 2.346.3-2 of the official Jenkins Docker image with OpenJDK 11.0.16.1 or 17.0.4.1.

          Thank you. I have read this page (and actually copied) this page when I was working on the upgrade on 23rd August. This information wasn't there at the time - I guess it wasn't yet known.

          Thank you for your excellent support!

          Marcin Cieślak added a comment - Thank you. I have read this page (and actually copied) this page when I was working on the upgrade on 23rd August. This information wasn't there at the time - I guess it wasn't yet known. Thank you for your excellent support!

            Unassigned Unassigned
            saper Marcin Cieślak
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: