-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
Jenkins ver. 1.586
System: Ubuntu 12.04.4 LTS 32bit
Plugins:
Amazon EC2 plugin
Ant Plugin
Async Http Client
Backup plugin
Build Monitor View
Build Pipeline Plugin
build timeout plugin
buildgraph-view
CloudBees Build Flow plugin
Cobertura Plugin
conditional-buildstep
Config File Provider Plugin
Copy Artifact Plugin
Credentials Plugin
CVS Plug-in
Deployed On Column Plugin
Dynamic Parameter Plug-in
Edgewall Trac Plugin
Email-ext plugin
embeddable-build-status
Environment Injector Plugin
Fingerprint Plugin
Git Client Plugin
Git Parameter Plug-In
Git Plugin
Git Server Plugin
Gitlab Hook Plugin
Gitlab Merge Request Builder
Green Balls
HipChat Plugin
Hudson Post build task
Javadoc Plugin
Javascript Widgets Plugin
jQuery plugin
JUnit Plugin
LDAP Plugin
Locale plugin
Log Parser Plugin
Mailer
Managed Scripts
MapDB API Plugin
Matrix Authorization Strategy Plugin
Matrix Project Plugin
Maven Project Plugin
Metadata plugin
Monitoring
Monitoring external jobs
Multijob Plugin
Node Iterator API Plugin
OWASP Markup Formatter Plugin
PAM Authentication plugin
Parameterized Trigger plugin
Post-Build Script Plug-in
promoted builds plugin
Rebuilder
Reverse Proxy Auth Plugin
ruby-runtime
Run Condition Plugin
S3 publisher plugin
SCM API Plugin
SCM Sync Configuration Plugin
Scriptler
SSH Agent Plugin
SSH Credentials Plugin
SSH Slaves plugin
Subversion Plugin
Timestamper
Token Macro Plugin
Translation Assistance Plugin
Violations plugin
Windows Slaves PluginJenkins ver. 1.586 System: Ubuntu 12.04.4 LTS 32bit Plugins: Amazon EC2 plugin Ant Plugin Async Http Client Backup plugin Build Monitor View Build Pipeline Plugin build timeout plugin buildgraph-view CloudBees Build Flow plugin Cobertura Plugin conditional-buildstep Config File Provider Plugin Copy Artifact Plugin Credentials Plugin CVS Plug-in Deployed On Column Plugin Dynamic Parameter Plug-in Edgewall Trac Plugin Email-ext plugin embeddable-build-status Environment Injector Plugin Fingerprint Plugin Git Client Plugin Git Parameter Plug-In Git Plugin Git Server Plugin Gitlab Hook Plugin Gitlab Merge Request Builder Green Balls HipChat Plugin Hudson Post build task Javadoc Plugin Javascript Widgets Plugin jQuery plugin JUnit Plugin LDAP Plugin Locale plugin Log Parser Plugin Mailer Managed Scripts MapDB API Plugin Matrix Authorization Strategy Plugin Matrix Project Plugin Maven Project Plugin Metadata plugin Monitoring Monitoring external jobs Multijob Plugin Node Iterator API Plugin OWASP Markup Formatter Plugin PAM Authentication plugin Parameterized Trigger plugin Post-Build Script Plug-in promoted builds plugin Rebuilder Reverse Proxy Auth Plugin ruby-runtime Run Condition Plugin S3 publisher plugin SCM API Plugin SCM Sync Configuration Plugin Scriptler SSH Agent Plugin SSH Credentials Plugin SSH Slaves plugin Subversion Plugin Timestamper Token Macro Plugin Translation Assistance Plugin Violations plugin Windows Slaves Plugin
We've been experiencing problems with out builds for a while now and they've been pretty random.
Our builds create a sandbox within the workspace directory and use it to run tests and the sandbox is owned by root(as it's a whole system root). It's not a problem as long as all Jenkins does is fetch new changes from git to that workspace, but if the .git directory gets deleted when Jenkins tries doing a clone and it fails with:
Cloning the remote Git repository
Cloning repository git@gitlab.something.net:something/something.git
ERROR: Failed to clean the workspace
java.nio.file.AccessDeniedException: /srv/jenkins/workspace/some-build-job/ec2-precise32-development/sandbox/jail/sys
The .git directory(along with .gitignore) has been disappearing randomly from workspace directories randomly and I've configured auditd service in order to find out which process has been deleting them, because I expected it to be something part of our process(Makefile?) and what I've found is stuff like this:
time->Sat Nov 22 08:32:03 2014
type=PATH msg=audit(1416645123.091:196): item=1 name="/srv/jenkins/workspace/xxxxxxx/label/ec2-precise32-development/.gitignore" inode=827146 dev=ca:01 mode=0100664 ouid=1000 ogid=1000 rdev=00:00
type=PATH msg=audit(1416645123.091:196): item=0 name="/srv/jenkins/workspace/xxxxxxx/label/ec2-precise32-development/" inode=814260 dev=ca:01 mode=040755 ouid=1000 ogid=1000 rdev=00:00
type=CWD msg=audit(1416645123.091:196): cwd="/home/ubuntu"
type=SYSCALL msg=audit(1416645123.091:196): arch=40000003 syscall=10 success=yes exit=0 a0=9f8be2f0 a1=b774c000 a2=b7709ff4 a3=9f8be2f0 items=2 ppid=1011 pid=4298 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="java" exe="/usr/lib/jvm/jdk1.7.0_51/bin/java" key=(null)
A java process(Jenkins) executes syscall 10 (unlink/remove) on .gitignore(and I assume .git as well, I don't monitor it because the logs would be overflowing). I have no idea why it does that.
This particular deletion has caused a failure in a build four days later.
The interesting thing is that on the same slave around the same time (Sat Nov 22 08:32:03 2014) there was another build run, but it was completely different from the one that failed four days later. It used a different workspace directory, so it's hard to say if the two things are related.
This has been really messing up with out CI because the builds failing randomly are nightlies.
