Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11952

Emulator occasionally hangs indefinitely on startup, unable to connect adb

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Android-Emulator 1.19-SNAPSHOT (from git)
      Jenkins 1.438
      SDK r15

      Sometimes the emulator will fail to start up, this seems to only (or mostly) happen when snapshots are enabled. When this happens, it'll go in a loop of, until it times out:

      $ /opt/android-sdk/platform-tools/adb connect emu:38996,48597
      $ /opt/android-sdk/platform-tools/adb -s emulator-38996 shell getprop dev.bootcomplete

      This seems to be because sometimes the adb server never moves the device from the "offline" to the "online" status. Forcing a reconnect seems to fix this - like restarting the adb server.

          [JENKINS-11952] Emulator occasionally hangs indefinitely on startup, unable to connect adb

          P Oh added a comment -

          forgot to add, plugin version: 2.13-SNAPSHOT (private-5bdae99c-jenkins)

          P Oh added a comment - forgot to add, plugin version: 2.13-SNAPSHOT (private-5bdae99c-jenkins)

          akosmaroy added a comment -

          I have similar issues as reported by others:

          $ /Users/build/.jenkins/tools/android-sdk/platform-tools/adb -s emulator-5847 shell getprop dev.bootcomplete
          error: device offline
          

          installing the current snapshot HPI file manually seems to have solved the problem

          akosmaroy added a comment - I have similar issues as reported by others: $ /Users/build/.jenkins/tools/android-sdk/platform-tools/adb -s emulator-5847 shell getprop dev.bootcomplete error: device offline installing the current snapshot HPI file manually seems to have solved the problem

          I'm seeing the same thing, even with the latest snapshot plugin (2.13-SNAPSHOT (private-67377031-jenkins). In my case, I wouldn't say it's occasional - i'd say it's more like 90% of the time the emulator will fail to respond. I have Jenkins slaves running on Debian EC2 instances - not using any emulator snapshots, every run is fresh with a clean workspace, so I haven't been able to figure out the difference as to why sometimes they are starting correctly and more often not.

          Here's an example of a sanitized log from a successful startup:
          ------------------------------------

          Deleting project workspace... Cloning the remote Git repository
          Cloning repository git@github.com:####/####.droid.git
          > git init /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test # timeout=10
          Fetching upstream changes from git@github.com:####/####.droid.git
          > git --version # timeout=10
          using GIT_SSH to set credentials
          > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/:refs/remotes/origin/
          > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10
          > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
          > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10
          Fetching upstream changes from git@github.com:####/####.droid.git
          using GIT_SSH to set credentials
          > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/:refs/remotes/origin/
          > git rev-parse origin/instrumentation-test^

          {commit} # timeout=10
          Checking out Revision #### (origin/instrumentation-test)
          > git config core.sparsecheckout # timeout=10
          > git checkout -f ####
          > git rev-list #### # timeout=10
          Starting xvnc
          [instrumentation-test] $ vncserver :63 -localhost -nolisten tcp

          New 'slave20150223:63 ()' desktop is slave20150223:63

          Starting applications specified in /var/lib/jenkins/.vnc/xstartup
          Log file is /var/lib/jenkins/.vnc/slave20150223:63.log

          $ /var/lib/jenkins/workspace/android-sdk/tools/android list target
          [android] Using Android SDK: /var/lib/jenkins/workspace/android-sdk
          [android] Creating Android AVD: /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test/.android/avd/hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation.avd
          [android] /var/lib/jenkins/workspace/android-sdk/tools/android create avd -f -a -s WVGA800 -n hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -t android-19 --abi armeabi-v7a
          [android] Setting hardware properties:
          hw.audioInput: no
          [android] Waiting 10 seconds before starting emulator...
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server
          * daemon not running. starting it now on port 9881 *
          * daemon started successfully *
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server
          [android] Starting Android emulator
          $ /var/lib/jenkins/workspace/android-sdk/tools/emulator -no-boot-anim -ports 9879,9880 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -no-snapshot-load -no-snapshot-save -no-audio -gpu off
          Failed to create Context 0x3005
          emulator: WARNING: Could not initialize OpenglES emulation, using software renderer.
          emulator: emulator window was out of view and was recentered

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880
          connected to localhost:9880
          [android] Waiting for emulator to finish booting...
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 shell getprop init.svc.bootanim
          error: device offline
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 shell getprop init.svc.bootanim
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 logcat -v time
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880
          [android] Emulator is ready for use (took 20 seconds)
          [Gradle] - Launching build.
          [instrumentation-test] $ /var/lib/jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.2.1_####.Droid_/gradle-2.2.1/bin/gradle --debug --info --stacktrace assemble
          Starting Build

          ----------------------------


          And here's it failing:

          ----------------------------

          Deleting project workspace... done

          Cloning the remote Git repository
          Cloning repository git@github.com:####
          > git init /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test # timeout=10
          Fetching upstream changes from git@github.com:####/####.droid.git
          > git --version # timeout=10
          using GIT_SSH to set credentials
          > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/:refs/remotes/origin/
          > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10
          > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
          > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10
          Fetching upstream changes from git@github.com:####/####.droid.git
          using GIT_SSH to set credentials
          > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/:refs/remotes/origin/
          > git rev-parse origin/instrumentation-test^{commit}

          # timeout=10
          Checking out Revision #### (origin/instrumentation-test)
          > git config core.sparsecheckout # timeout=10
          > git checkout -f ####
          > git rev-list #### # timeout=10
          Starting xvnc
          [instrumentation-test] $ vncserver :95 -localhost -nolisten tcp

          New 'slave20150223:95 ()' desktop is slave20150223:95

          Starting applications specified in /var/lib/jenkins/.vnc/xstartup
          Log file is /var/lib/jenkins/.vnc/slave20150223:95.log

          $ /var/lib/jenkins/workspace/android-sdk/tools/android list target
          [android] Using Android SDK: /var/lib/jenkins/workspace/android-sdk
          [android] Creating Android AVD: /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test/.android/avd/hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation.avd
          [android] /var/lib/jenkins/workspace/android-sdk/tools/android create avd -f -a -s WVGA800 -n hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -t android-19 --abi armeabi-v7a
          [android] Setting hardware properties:
          hw.audioInput: no
          [android] Waiting 10 seconds before starting emulator...

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server

          • daemon not running. starting it now on port 9227 *
          • daemon started successfully *
            $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server
            [android] Starting Android emulator
            $ /var/lib/jenkins/workspace/android-sdk/tools/emulator -no-boot-anim -ports 9225,9226 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -no-snapshot-load -no-snapshot-save -no-audio -gpu off

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226
          connected to localhost:9226
          [android] Waiting for emulator to finish booting...
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim
          error: device offline
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim
          error: device offline
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim
          error: device offline
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb disconnect localhost:9226
          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226

          $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim
          error: device offline

          ------------

          In the failure case, it will sit there repeatedly cycling through the commands and the error: device offline until it times out and the job aborts.

          Nicholas Paufler added a comment - I'm seeing the same thing, even with the latest snapshot plugin (2.13-SNAPSHOT (private-67377031-jenkins). In my case, I wouldn't say it's occasional - i'd say it's more like 90% of the time the emulator will fail to respond. I have Jenkins slaves running on Debian EC2 instances - not using any emulator snapshots, every run is fresh with a clean workspace, so I haven't been able to figure out the difference as to why sometimes they are starting correctly and more often not. Here's an example of a sanitized log from a successful startup: ------------------------------------ Deleting project workspace... Cloning the remote Git repository Cloning repository git@github.com:####/####.droid.git > git init /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test # timeout=10 Fetching upstream changes from git@github.com:####/####.droid.git > git --version # timeout=10 using GIT_SSH to set credentials > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/ :refs/remotes/origin/ > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10 Fetching upstream changes from git@github.com:####/####.droid.git using GIT_SSH to set credentials > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/ :refs/remotes/origin/ > git rev-parse origin/instrumentation-test^ {commit} # timeout=10 Checking out Revision #### (origin/instrumentation-test) > git config core.sparsecheckout # timeout=10 > git checkout -f #### > git rev-list #### # timeout=10 Starting xvnc [instrumentation-test] $ vncserver :63 -localhost -nolisten tcp New 'slave20150223:63 ()' desktop is slave20150223:63 Starting applications specified in /var/lib/jenkins/.vnc/xstartup Log file is /var/lib/jenkins/.vnc/slave20150223:63.log $ /var/lib/jenkins/workspace/android-sdk/tools/android list target [android] Using Android SDK: /var/lib/jenkins/workspace/android-sdk [android] Creating Android AVD: /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test/.android/avd/hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation.avd [android] /var/lib/jenkins/workspace/android-sdk/tools/android create avd -f -a -s WVGA800 -n hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -t android-19 --abi armeabi-v7a [android] Setting hardware properties: hw.audioInput: no [android] Waiting 10 seconds before starting emulator... $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server * daemon not running. starting it now on port 9881 * * daemon started successfully * $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server [android] Starting Android emulator $ /var/lib/jenkins/workspace/android-sdk/tools/emulator -no-boot-anim -ports 9879,9880 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -no-snapshot-load -no-snapshot-save -no-audio -gpu off Failed to create Context 0x3005 emulator: WARNING: Could not initialize OpenglES emulation, using software renderer. emulator: emulator window was out of view and was recentered $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880 connected to localhost:9880 [android] Waiting for emulator to finish booting... $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 shell getprop init.svc.bootanim error: device offline $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880 $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 shell getprop init.svc.bootanim $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9880 logcat -v time $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9880 [android] Emulator is ready for use (took 20 seconds) [Gradle] - Launching build. [instrumentation-test] $ /var/lib/jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.2.1_####.Droid_/gradle-2.2.1/bin/gradle --debug --info --stacktrace assemble Starting Build ---------------------------- And here's it failing: ---------------------------- Deleting project workspace... done Cloning the remote Git repository Cloning repository git@github.com:#### > git init /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test # timeout=10 Fetching upstream changes from git@github.com:####/####.droid.git > git --version # timeout=10 using GIT_SSH to set credentials > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/ :refs/remotes/origin/ > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 > git config remote.origin.url git@github.com:####/####.droid.git # timeout=10 Fetching upstream changes from git@github.com:####/####.droid.git using GIT_SSH to set credentials > git -c core.askpass=true fetch --tags --progress git@github.com:####/####.droid.git +refs/heads/ :refs/remotes/origin/ > git rev-parse origin/instrumentation-test^{commit} # timeout=10 Checking out Revision #### (origin/instrumentation-test) > git config core.sparsecheckout # timeout=10 > git checkout -f #### > git rev-list #### # timeout=10 Starting xvnc [instrumentation-test] $ vncserver :95 -localhost -nolisten tcp New 'slave20150223:95 ()' desktop is slave20150223:95 Starting applications specified in /var/lib/jenkins/.vnc/xstartup Log file is /var/lib/jenkins/.vnc/slave20150223:95.log $ /var/lib/jenkins/workspace/android-sdk/tools/android list target [android] Using Android SDK: /var/lib/jenkins/workspace/android-sdk [android] Creating Android AVD: /var/lib/jenkins/workspace/####.Droid-branches/instrumentation-test/.android/avd/hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation.avd [android] /var/lib/jenkins/workspace/android-sdk/tools/android create avd -f -a -s WVGA800 -n hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -t android-19 --abi armeabi-v7a [android] Setting hardware properties: hw.audioInput: no [android] Waiting 10 seconds before starting emulator... $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server daemon not running. starting it now on port 9227 * daemon started successfully * $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb start-server [android] Starting Android emulator $ /var/lib/jenkins/workspace/android-sdk/tools/emulator -no-boot-anim -ports 9225,9226 -prop persist.sys.language=en -prop persist.sys.country=US -avd hudson_en-US_160_WVGA_android-19_armeabi-v7a_instrumentation -no-snapshot-load -no-snapshot-save -no-audio -gpu off $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226 connected to localhost:9226 [android] Waiting for emulator to finish booting... $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim error: device offline $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226 $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim error: device offline $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226 $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim error: device offline $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb disconnect localhost:9226 $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb connect localhost:9226 $ /var/lib/jenkins/workspace/android-sdk/platform-tools/adb -s localhost:9226 shell getprop init.svc.bootanim error: device offline ------------ In the failure case, it will sit there repeatedly cycling through the commands and the error: device offline until it times out and the job aborts.

          Erik Ingman added a comment -

          We have solved this on our end, it seems to work every time. According to the Android Emulator docs "The console port number must be an even integer between 5554 and 5584, inclusive. <port>+1 must also be free and will be reserved for ADB." So we've created a PR, see https://github.com/jenkinsci/android-emulator-plugin/pull/48

          /Erik @ www.blocket.se

          Erik Ingman added a comment - We have solved this on our end, it seems to work every time. According to the Android Emulator docs "The console port number must be an even integer between 5554 and 5584, inclusive. <port>+1 must also be free and will be reserved for ADB." So we've created a PR, see https://github.com/jenkinsci/android-emulator-plugin/pull/48 /Erik @ www.blocket.se

          pipen: I tried this change a month or so ago after somebody mentioned that documentation to me, but I found that it made no difference.

          From re-reading the Android source code, I couldn't see any obvious technical reason why the ports have to be in that form (and only the "-port" flag, not "-ports" enforces it anyway), the "maximum" port number hasn't been 5584 for a long time, adb does recognise emulators started on high ports, and I also looked through lots of build logs for evidence that the user port being even and in that range makes a difference — I didn't find anything conclusive (in fact, emulator startup was more likely to fail in that case).

          In addition, the code in this pull request doesn't guarantee that the first port will be even, so I'll have to be skeptical about your claims
          But if it turns out that this change nevertheless helps, then that's good to know. So what are the results you are seeing? Which OS are the emulator builds running on, and what is the emulator config? How many builds were failing previously, compared to now, and is that with builds whose user port is actually even?

          Christopher Orr added a comment - pipen : I tried this change a month or so ago after somebody mentioned that documentation to me, but I found that it made no difference. From re-reading the Android source code, I couldn't see any obvious technical reason why the ports have to be in that form (and only the "-port" flag, not "-ports" enforces it anyway), the "maximum" port number hasn't been 5584 for a long time, adb does recognise emulators started on high ports, and I also looked through lots of build logs for evidence that the user port being even and in that range makes a difference — I didn't find anything conclusive (in fact, emulator startup was more likely to fail in that case). In addition, the code in this pull request doesn't guarantee that the first port will be even, so I'll have to be skeptical about your claims But if it turns out that this change nevertheless helps, then that's good to know. So what are the results you are seeing? Which OS are the emulator builds running on, and what is the emulator config? How many builds were failing previously, compared to now, and is that with builds whose user port is actually even?

          Joe Hansche added a comment -

          I tend to agree with orrc. Out of the last ~7 builds that were attempted, these were the results:

          • -ports 5743,5744 – Worked
          • -ports 5849,5850 – Failed to start
          • -ports 5705,5706 – Failed to start
          • -ports 5886,5887 – Worked
          • -ports 5719,5720 – Worked
          • -ports 5619,5620 – Failed to start
          • -ports 5866,5867 – Worked
          • -ports 5822,5823 – Worked

          So this means it does not seem to be affected by the even/odd pairs.

          Joe Hansche added a comment - I tend to agree with orrc . Out of the last ~7 builds that were attempted, these were the results: -ports 5743,5744 – Worked -ports 5849,5850 – Failed to start -ports 5705,5706 – Failed to start -ports 5886,5887 – Worked -ports 5719,5720 – Worked -ports 5619,5620 – Failed to start -ports 5866,5867 – Worked -ports 5822,5823 – Worked So this means it does not seem to be affected by the even/odd pairs.

          Version 2.13 of the plugin has been released, which includes the updates discussed here.

          Christopher Orr added a comment - Version 2.13 of the plugin has been released, which includes the updates discussed here.

          I'm marking this issue as closed, though I'm aware of the likelihood that not everything is working brilliantly on every machine, all the time.

          But if there are still obvious issues in 2.13, let's please open a clean issue.

          Christopher Orr added a comment - I'm marking this issue as closed, though I'm aware of the likelihood that not everything is working brilliantly on every machine, all the time. But if there are still obvious issues in 2.13, let's please open a clean issue.

          Andy Piper added a comment -

          This does not work.

          andypiper:platform-tools andy$ ./adb -s localhost:5554 shell getprop init.svc.bootanim
          error: device 'localhost:5554' not found

          Maybe something changed in the latest releases of android, but this mechanism is now invalid as a means of connecting to the emulator

          Andy Piper added a comment - This does not work. andypiper:platform-tools andy$ ./adb -s localhost:5554 shell getprop init.svc.bootanim error: device 'localhost:5554' not found Maybe something changed in the latest releases of android, but this mechanism is now invalid as a means of connecting to the emulator

          Andy Piper added a comment -

          Does not work on current android releases.

          Android Debug Bridge version 1.0.32
          Revision 57224c5cff69-android

          Andy Piper added a comment - Does not work on current android releases. Android Debug Bridge version 1.0.32 Revision 57224c5cff69-android

            Unassigned Unassigned
            jorgenpt Jørgen Tjernø
            Votes:
            10 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated: