Loading...

XML

Word

Printable

Type: Bug
Resolution: Not A Defect
Priority: Minor
Component/s: bitbucket-plugin, git-plugin, kubernetes-plugin
Labels:
None
Environment:
Dev

Similar Issues:

Show

We clone three repos in our pipeline from BitBucket:

Shared Libraries
Jenkinsfile
Source code

The build fails intermittently with logs shown below:

Started by an SCM change
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://bitbucket-server/scm/xxxx/xxxx.git
 > git config remote.origin.url https://bitbucket-server/scm/xxxx/xxxx.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/develop
Seen branch in repository origin/master
Seen 2 remote branches
Obtained Jenkinsfile from c310abe88924c711a29be22b659dd719ac680942
Running in Durability level: MAX_SURVIVABILITY
Loading library utilities@xxxx
Attempting to resolve xxxx from remote references...
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git ls-remote ssh://git@xxxx/xxxx/jenkins-libraries.git # timeout=10
ERROR: Timeout after 10 minutes
hudson.plugins.git.GitException: Command "git ls-remote ssh://git@xxxx:xxxx/xxxx/jenkins-libraries.git" returned status code 128:
stdout: 
stderr: Terminated
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1666)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1657)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2877)
    at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:806)
    at jenkins.scm.api.SCMSource.fetch(SCMSource.java:617)
    at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:83)
    at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:98)
    at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:83)
    at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
    at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
    at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

On triggering the same job manually after some time, it succeeds. I do not mean to say here that the poll scm is the issue. Sometimes it succeeds. We are using Jenkins in master - slave configuration. For the master, jenkins/jenkins:2.168 image is used and jenkins/jnlp-slave:3.10-1 for slave. This entire setup is running on Azure Kubernetes Service and deployed through Helm. For git, we use Git plugin version 3.9.3

Edit 1:

Please find below an excerpt of the logs:

Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.711748] Killed process 30955 (git) total-vm:15612kB, anon-rss:144kB, file-rss:900kB, shmem-rss:0kB
Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.725732] Memory cgroup out of memory: Kill process 26054 (java) score 1886 or sacrifice child
Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.733299] Killed process 30969 (git) total-vm:15612kB, anon-rss:144kB, file-rss:876kB, shmem-rss:0kB
Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.760196] Memory cgroup out of memory: Kill process 26054 (java) score 1887 or sacrifice child
Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.770192] Killed process 26054 (java) total-vm:13656520kB, anon-rss:3995608kB, file-rss:19420kB, shmem-rss:0kB
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.728959] IPv6: ADDRCONF(NETDEV_UP): azv67869479f64: link is not ready
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729980] azure0: port 14(azv67869479f64) entered blocking state
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729982] azure0: port 14(azv67869479f64) entered disabled state
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.730045] device azv67869479f64 entered promiscuous mode
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760317] IPv6: ADDRCONF(NETDEV_CHANGE): azv67869479f64: link becomes ready
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760382] azure0: port 14(azv67869479f64) entered blocking state
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760384] azure0: port 14(azv67869479f64) entered forwarding state
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760894] eth0: renamed from azv67869479f642
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780184] azure0: port 14(azv67869479f64) entered blocking state
Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780186] azure0: port 14(azv67869479f64) entered forwarding state
Mar 26 12:35:08 aks-prodagtpool-18673252-1 kernel: [2626416.564062] device azv67869479f64 left promiscuous mode
---
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102751] java invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=928
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102752] java cpuset=efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe mems_allowed=0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102758] CPU: 5 PID: 11193 Comm: java Not tainted 4.15.0-1037-azure #39~16.04.1-Ubuntu
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102759] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102760] Call Trace:
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102766]  dump_stack+0x63/0x82
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102770]  dump_header+0x77/0x285
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102772]  oom_kill_process+0x22e/0x450
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102774]  out_of_memory+0x11d/0x4c0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102778]  mem_cgroup_out_of_memory+0x4b/0x80
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102780]  mem_cgroup_oom_synchronize+0x32a/0x350
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102782]  ? mem_cgroup_css_reset+0xe0/0xe0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102784]  pagefault_out_of_memory+0x36/0x7b
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102788]  mm_fault_error+0x8f/0x190
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102792]  ? handle_mm_fault+0xcc/0x1c0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102794]  __do_page_fault+0x4cd/0x500
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102797]  ? SyS_futex+0x7f/0x180
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102799]  do_page_fault+0x2e/0xf0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102803]  ? page_fault+0x2f/0x50
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102805]  page_fault+0x45/0x50
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102807] RIP: 0033:0x7f00fc8cced2
---
podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe killed as a result of limit of /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212950] memory: usage 4194304kB, limit 4194304kB, failcnt 359672
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212951] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] kmem: usage 72724kB, limit 9007199254740988kB, failcnt 0
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212961] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/16c243b50272a881b6786a8b0ef20ffe0e27d7876e8d1399f97227db2bdf5919: cache:0KB rss:44KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212970] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/a5c5b66f6ef785fd6a110226a041960ba7d4f6c1d8927a769d2c3b79d5973ea1: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212978] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/3ae2c3e5cac91453836f9df37ea67add55d9a04ad79eeed74a82d962a3632968: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212986] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe: cache:136KB rss:4121400KB rss_huge:2103296KB shmem:0KB mapped_file:4KB dirty:4KB writeback:0KB inactive_anon:0KB active_anon:4121400KB inactive_file:80KB active_file:56KB unevictable:0KB

Jenkins is the only application running in the pod. You can take a look at screenshots for details about java versions.

Edit 2:

The Jenkins uses polling mechanism instead to get status of all the repositories

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

javamemory.png
2019-03-28 07:04
31 kB
Milind Sonawane
javaversion.JPG
2019-03-28 07:04
21 kB
Milind Sonawane

Assignee:: Unassigned

Reporter:: Milind Sonawane

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2019-03-18 09:31

Updated:: 2019-07-10 06:36

Resolved:: 2019-05-20 01:22

Details

Description

Attachments

Attachments

Activity

People

Dates