Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61292

Malloc/Memory fail on windows

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
      Jenkins: 2.204.2
      docker-workflow-plugin: 1.21
      Windows 2019 DC Edition (ECS Optimized) - Feb 11 patch
    • Similar Issues:

      Description

      About every 4-5 runs on our windows-docker nodes we getAbout every 4-5 runs on our windows-docker nodes we get
      [2020-03-02T10:03:18.171Z] C:\Jenkins\workspace\administrative\resourcepoll>docker inspect -f . mcr.microsoft.com/powershell:preview 
      [2020-03-02T10:03:18.171Z] *fatal error: out of memory allocating heap arena map*
      ```
      [2020-03-02T10:03:17.324Z] $ docker run -t -d -u 500:500 -w /home/ec2-user/workspace/administrative/resourcepoll -v /home/ec2-user/workspace/administrative/resourcepoll:/home/ec2-user/workspace/administrative/resourcepoll:rw,z -v /home/ec2-user/workspace/administrative/resourcepoll@tmp:/home/ec2-user/workspace/administrative/resourcepoll@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** appropriate/curl cat
      [2020-03-02T10:03:18.070Z] $ docker top 2c0e0f1a802af847fe5457ae1754ff9b3f97d395eba043de176a6195199a8892 -eo pid,comm
      [2020-03-02T10:03:18.153Z] 
      [2020-03-02T10:03:18.171Z] C:\Jenkins\workspace\administrative\resourcepoll>docker inspect -f . mcr.microsoft.com/powershell:preview 
      [2020-03-02T10:03:18.171Z] fatal error: out of memory allocating heap arena map
      [2020-03-02T10:03:18.171Z] 
      [2020-03-02T10:03:18.171Z] runtime stack:
      [2020-03-02T10:03:18.171Z] runtime.throw(0x1ecaab1, 0x27)
      [2020-03-02T10:03:18.171Z] C:/.GOROOT/src/runtime/panic.go:617 +0x79 fp=0x484fa90 sp=0x484fa60 pc=0x435739
      [2020-03-02T10:03:18.171Z] runtime.(*mheap).sysAlloc(0x3478360, 0x2000, 0x3478370, 0x1)[2020-03-02T10:03:18.171Z] C:/.GOROOT/src/runtime/malloc.go:643 +0x71c fp=0x484fb38 sp=0x484fa90 pc=0x41154c
      [2020-03-02T10:03:18.171Z] runtime.(*mheap).grow(0x3478360, 0x1, 0x0)
      ```

      I looked at the code for the [docker-pipeline plugin](https://github.com/jenkinsci/docker-workflow-plugin/blob/6d01b80a4199317b336fb67ad0693976e13c9681/src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy) and it's invoking the native go tool instead of talking directly with the daemon - so, it's bound by the memory constraints of a subprocess on windows. We are running plugin 1.21

      We are running this on a windows docker-optimized AMI (Windows DataCenter 2019 with Feb11 patch). We've already bumped the memory args for the agent process considerably and I am not seeing a definitive impact but I am open to suggestions. I mean, there is only so much memory I can allocate - these virts are already damn beefy compared to their linux brethren. 

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            chb0jenkins Christian Bongiorno
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: