Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43594

Job kills "npm install" that stuck in preinstall step

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Won't Fix
    • Component/s: pipeline
    • Labels:
    • Environment:
      Jenkins 2.46.1 in Docker on AWS EC2 t2.medium instance with Amazon Linux (amd64, 4.4.19-29.55.amzn1.x86_64)

      Node JS 6.10.2, npm 3.10.10
      nodejs plugin 1.2.0

      The only master is used to run jobs
    • Similar Issues:

      Description

      Jenkins docker-compose:

      version: '2'
      services:
        jenkins:
          cpu_shares: 512
          image: jenkins:2.46.1
          mem_limit: 2g
          ports:
          - 80:8080
          - 50000:50000
          privileged: true
          user: root
          volumes:
          - /home/ec2-user/jenkins_home:/var/jenkins_home
          - /var/run/docker.sock:/var/run/docker.sock
          - /usr/bin/docker:/usr/local/bin/docker
          - /lib64/libdevmapper.so.1.02:/usr/lib/libdevmapper.so.1.02
          - /lib64/libudev.so.0:/usr/lib/libudev.so.0
          - /home/ec2-user/.m2:/root/.m2
      

      Job configuration:

      #!groovy
      
      node {
          timeout(20 /* minutes */) {
              // Need to replace the '%2F' used by Jenkins to deal with / in the path (e.g. story/...)
              // because tests that do getResource will escape the % again, and the test files can't be found.
              // See https://issues.jenkins-ci.org/browse/JENKINS-34564 for more.
              ws("workspace/${env.JOB_NAME}") {
                  stage('Checkout') {
                      checkout scm
                  }
      
                  stage('Build') {
                      parallel api: {
                          mvn 'clean install -DskipTests -Pprod'
                          archiveToS3 'api/target/api*-standalone.jar'
                      }, web: {
                          dir('frontend') {
                              npm 'install -d'
      
                              npm 'run build-web'
                              dir('web-react/build') {
                                  sh "tar zcf ../../web.tar.gz ./*"
                              }
      
                              archiveToS3 'web.tar.gz'
                          }
                      }, failFast: true
                  }
      ...
      
      def npm(args) {
          nodejs(nodeJSInstallationName: 'node6LTS') {
              sh "npm $args"
          }
      }

      Job stuck on the step 

      npm 'install -d'

      After some time (less than 20 minutes) the job fails with the log like this

      ...
      [web] npm info attempt registry request try #1 at 5:43:32 AM
      [web] npm http request GET https://registry.npmjs.org/http-errors
      [web] npm http 304 https://registry.npmjs.org/http-errors
      [web] npm info lifecycle <my-project-name>@1.2.1~preinstall: <my-project-name>@1.2.1
      [web] Killed
      [Pipeline] [web] }
      [Pipeline] [web] // wrap
      [Pipeline] [web] }
      [Pipeline] [web] // dir
      [Pipeline] [web] }
      [web] Failed in branch web
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // ws
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code 137
      Finished: FAILURE
      

      This behavior I encountered with Jenkins 2.19.4 after some plugins updates (about 7 April). Even after migration to Jenkins 2.46.1 I have the same issue.

      When I run the same command manually locally or on the same Jenkins job workspace it works well. I don't know if it is related to NodeJs plugin, or any other plugins, or Jenkins itself.

      Sometimes it works but it is a very rare case about 1 from 30 executions.

      P.S.:

      It is a blocker for us. Any hint or workaround is appreciated.

        Attachments

          Activity

          Hide
          abeletsky Andrey Beletsky added a comment - - edited

          I've noticed Mem 100% during

          npm install

          See 

          Let you know if I find something else.

          Show
          abeletsky Andrey Beletsky added a comment - - edited I've noticed Mem 100% during npm install See  Let you know if I find something else.
          Hide
          nfalco Nikolas Falco added a comment -

          The issue seems not related to the NodeJS plugin because I do not see anywhere you are using it.
          No tools or nodejs DSL steps are present in you Jenkinsfile. I do not know who provide implementation of the npm command you use.
          In case of NodeJs plugin I expect a sort of

          sh "npm install; npm run build-web"
          

          Please refer to the Wiki for the NodeJS plugin DSL step commands availables.

          Show
          nfalco Nikolas Falco added a comment - The issue seems not related to the NodeJS plugin because I do not see anywhere you are using it. No tools or nodejs DSL steps are present in you Jenkinsfile. I do not know who provide implementation of the npm command you use. In case of NodeJs plugin I expect a sort of sh "npm install; npm run build-web" Please refer to the Wiki for the NodeJS plugin DSL step commands availables.
          Hide
          abeletsky Andrey Beletsky added a comment -

          Actually, we use nodejs plugin. Updated Description.

          Show
          abeletsky Andrey Beletsky added a comment - Actually, we use nodejs plugin. Updated Description .
          Hide
          nfalco Nikolas Falco added a comment -

          Did you try outside docker? becuase the plugin only make available the node and npm command to the PATH environment, nothing more

          Show
          nfalco Nikolas Falco added a comment - Did you try outside docker? becuase the plugin only make available the node and npm command to the PATH environment, nothing more
          Hide
          abeletsky Andrey Beletsky added a comment -

          As I mentioned in the description there is no any evidence it is nodejs plugin issue.

          No, I didn't try it yet outside of docker. It doesn't make any sense to me due to we use docker approach. Currently, I try to figure out whether it is RAM 100% usage related issue.

          Show
          abeletsky Andrey Beletsky added a comment - As I mentioned in the description there is no any evidence it is nodejs plugin issue. No, I didn't try it yet outside of docker. It doesn't make any sense to me due to we use docker approach. Currently, I try to figure out whether it is RAM 100% usage related issue.
          Hide
          nfalco Nikolas Falco added a comment -

          Trying the DSL outside docker help you to reduce entropy

          Show
          nfalco Nikolas Falco added a comment - Trying the DSL outside docker help you to reduce entropy
          Hide
          abeletsky Andrey Beletsky added a comment -

          Thank you, I will try.

          Do you have any ideas who kills the job? The log "Killed" is not very descriptive. Probably there is any log to see detailed information about what happens or there is a way to turn it on.

          Show
          abeletsky Andrey Beletsky added a comment - Thank you, I will try. Do you have any ideas who kills the job? The log "Killed" is not very descriptive. Probably there is any log to see detailed information about what happens or there is a way to turn it on.
          Hide
          abeletsky Andrey Beletsky added a comment -

          The root cause of npm install failure is a shortage of memory (2G is not enough).
          I couldn't figure out how to tune node v8 options for npm
          --max_semi_space_size=512 --max_old_space_size=512 --max_executable_size=512 --gc_global
          I've just increased Jenkins memory usage limit from 2G to 3.5G (4G is a max for t2.medium EC2).

          Show
          abeletsky Andrey Beletsky added a comment - The root cause of npm install failure is a shortage of memory (2G is not enough). I couldn't figure out how to tune node v8 options for npm --max_semi_space_size=512 --max_old_space_size=512 --max_executable_size=512 --gc_global I've just increased Jenkins memory usage limit from 2G to 3.5G (4G is a max for t2.medium EC2).

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            abeletsky Andrey Beletsky
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: