Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
None
-
SDK Tools r11, Fedora Core 14 i686 (slave), number of executors per slave 4. Jenkins 1.414, Android Emulator 1.15
Description
When running multiple emulator instances on a single node each within their own Xvnc session, at the end of the job the plugin seems to end up killing the adb daemon. The other emulators lose connection to the daemon, and even though a background task may start adb again, emulators may not successfully reconnect to adb. This is particularly problematic when running large multi-configuration android-emulator jobs on a single node with a large degree of parallelism. 2 concurrent executors seems to occasionally cause problems but with 4 the results are not even worthwhile to review.
Reducing the number of executors on the node to 1 (or otherwise weighting the job to consume an entire system) does work-around the problem.
Attachments
Activity
Mike Elkin
added a comment - I am digging into this quite a bit more as a result of your comment. Right now it seems like it is sporadically, these are 32bit FC14 systems with Oracle JDK 1.6u24 + ant 1.8.2 + SDKr11 running multi-configuration on a variety of device configurations. I was able to instigate a crash but now after running some more tests it is not crashing. I will continue to investigate and hopefully come up with some useful logs for review.
Mike Elkin
added a comment - I am digging into this quite a bit more as a result of your comment. Right now it seems like it is sporadically, these are 32bit FC14 systems with Oracle JDK 1.6u24 + ant 1.8.2 + SDKr11 running multi-configuration on a variety of device configurations. I was able to instigate a crash but now after running some more tests it is not crashing. I will continue to investigate and hopefully come up with some useful logs for review. Mike Elkin
added a comment - Looking at this more you are a bit right if ADB has already been started. However when ADB has to be started up the stability seems to go way down. To replicate you'd need to kill the adb daemon, setup a many-executor node, and kick off a multi-config emulator job that will consume all of the executors. I've noticed that a lot of the jobs seem to hang and in some cases the adb server will die off.
Mike Elkin
added a comment - Looking at this more you are a bit right if ADB has already been started. However when ADB has to be started up the stability seems to go way down. To replicate you'd need to kill the adb daemon, setup a many-executor node, and kick off a multi-config emulator job that will consume all of the executors. I've noticed that a lot of the jobs seem to hang and in some cases the adb server will die off. 
Jørgen Tjernø
added a comment - I've submitted a proposed fix for this here:
https://github.com/jorgenpt/android-emulator-plugin/commit/0702320d18b152d360b80d77f189342d2c8296cf
It's a part of this pull request:
https://github.com/jenkinsci/android-emulator-plugin/pull/8
The change makes it run a private adb for each emulator, which should fix all problems like these. 
Jørgen Tjernø
added a comment - I've submitted a proposed fix for this here:
https://github.com/jorgenpt/android-emulator-plugin/commit/0702320d18b152d360b80d77f189342d2c8296cf
It's a part of this pull request:
https://github.com/jenkinsci/android-emulator-plugin/pull/8
The change makes it run a private adb for each emulator, which should fix all problems like these. Version 2.0 has now been released, which includes Jørgen's feature.
One instance of ADB crashing no longer affects other builds running in parallel.
The plugin never tries to kill the adb daemon.
I also run jobs with many executors on Linux (SDK Tools r11, multiple Ubuntu amd64 slaves, 4 executors per slave) without having this problem.
However, I have seen this issue in the past where adb crashes at the end of a job. Since
JENKINS-7354, I haven't had any further reports of it though, and adb does seem to be a bit more well behaved these days.But as I still don't have any way of reproducing this, and don't have any indication of why adb fails (or any sort of gdb stacktrace from when it does), there's not really much I can do about this...
For reference, does this happen for every job, or only sporadically? Has it always happened, or only since upgrading the Android SDK, for example? Is there anything particular given in the build log output once it's finished and crashes adb?