Blocker Jenkins docker-compose:
version: '2' services: jenkins: cpu_shares: 512 image: jenkins:2.46.1 mem_limit: 2g ports: - 80:8080 - 50000:50000 privileged: true user: root volumes: - /home/ec2-user/jenkins_home:/var/jenkins_home - /var/run/docker.sock:/var/run/docker.sock - /usr/bin/docker:/usr/local/bin/docker - /lib64/libdevmapper.so.1.02:/usr/lib/libdevmapper.so.1.02 - /lib64/libudev.so.0:/usr/lib/libudev.so.0 - /home/ec2-user/.m2:/root/.m2
Job configuration:
#!groovy
node {
timeout(20 /* minutes */) {
// Need to replace the '%2F' used by Jenkins to deal with / in the path (e.g. story/...)
// because tests that do getResource will escape the % again, and the test files can't be found.
// See https://issues.jenkins-ci.org/browse/JENKINS-34564 for more.
ws("workspace/${env.JOB_NAME}") {
stage('Checkout') {
checkout scm
}
stage('Build') {
parallel api: {
mvn 'clean install -DskipTests -Pprod'
archiveToS3 'api/target/api*-standalone.jar'
}, web: {
dir('frontend') {
npm 'install -d'
npm 'run build-web'
dir('web-react/build') {
sh "tar zcf ../../web.tar.gz ./*"
}
archiveToS3 'web.tar.gz'
}
}, failFast: true
}
...
def npm(args) {
nodejs(nodeJSInstallationName: 'node6LTS') {
sh "npm $args"
}
}
Job stuck on the step
npm 'install -d'
After some time (less than 20 minutes) the job fails with the log like this
... [web] npm info attempt registry request try #1 at 5:43:32 AM [web] npm http request GET https://registry.npmjs.org/http-errors [web] npm http 304 https://registry.npmjs.org/http-errors [web] npm info lifecycle <my-project-name>@1.2.1~preinstall: <my-project-name>@1.2.1 [web] Killed [Pipeline] [web] } [Pipeline] [web] // wrap [Pipeline] [web] } [Pipeline] [web] // dir [Pipeline] [web] } [web] Failed in branch web [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // ws [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code 137 Finished: FAILURE
This behavior I encountered with Jenkins 2.19.4 after some plugins updates (about 7 April). Even after migration to Jenkins 2.46.1 I have the same issue.
When I run the same command manually locally or on the same Jenkins job workspace it works well. I don't know if it is related to NodeJs plugin, or any other plugins, or Jenkins itself.
Sometimes it works but it is a very rare case about 1 from 30 executions.
P.S.:
It is a blocker for us. Any hint or workaround is appreciated.
[JENKINS-43594] Job kills "npm install" that stuck in preinstall step
The issue seems not related to the NodeJS plugin because I do not see anywhere you are using it.
No tools or nodejs DSL steps are present in you Jenkinsfile. I do not know who provide implementation of the npm command you use.
In case of NodeJs plugin I expect a sort of
sh "npm install; npm run build-web"
Please refer to the Wiki for the NodeJS plugin DSL step commands availables.
Did you try outside docker? becuase the plugin only make available the node and npm command to the PATH environment, nothing more
As I mentioned in the description there is no any evidence it is nodejs plugin issue.
No, I didn't try it yet outside of docker. It doesn't make any sense to me due to we use docker approach. Currently, I try to figure out whether it is RAM 100% usage related issue.
Thank you, I will try.
Do you have any ideas who kills the job? The log "Killed" is not very descriptive. Probably there is any log to see detailed information about what happens or there is a way to turn it on.
The root cause of npm install failure is a shortage of memory (2G is not enough).
I couldn't figure out how to tune node v8 options for npm
--max_semi_space_size=512 --max_old_space_size=512 --max_executable_size=512 --gc_global
I've just increased Jenkins memory usage limit from 2G to 3.5G (4G is a max for t2.medium EC2).
I've noticed Mem 100% during
See
Let you know if I find something else.