Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8830

Hudson AIX slave build is getting stuck on Ant build execution.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Incomplete
    • Component/s: remoting, ssh-slaves-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I am getting stuck build on my AIX slave. I can run the Ant build command sucessfully from the slave's workspace by SSHing into the machine and executing it manually, but it will not run via Hudson.

      Per instructions here: http://wiki.hudson-ci.org/display/HUDSON/Build+is+hanging

      Relevant Thread Dumps on Master:

      Channel reader thread: RnDAIX

      "Channel reader thread: RnDAIX" Id=2428 Group=main WAITING on java.lang.Object@2d1a002d
      at java.lang.Object.wait(Native Method)

      • waiting on java.lang.Object@2d1a002d
        at java.lang.Object.wait(Object.java:485)
        at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
        at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:947)

      Executor #0 for RnDAIX : executing autoInstall-V6R2012AIX #31

      "Executor #0 for RnDAIX : executing autoInstall-V6R2012AIX #31" Id=76 Group=main WAITING on hudson.remoting.UserRequest@40ad47e1
      at java.lang.Object.wait(Native Method)

      • waiting on hudson.remoting.UserRequest@40ad47e1
        at java.lang.Object.wait(Object.java:485)
        at hudson.remoting.Request$1.get(Request.java:206)
        at hudson.remoting.Request$1.get(Request.java:172)
        at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
        at hudson.Proc$RemoteProc.join(Proc.java:351)
        at hudson.Launcher$ProcStarter.join(Launcher.java:280)
        at hudson.tasks.Ant.perform(Ant.java:216)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:622)
        at hudson.model.Build$RunnerImpl.build(Build.java:172)
        at hudson.model.Build$RunnerImpl.doRun(Build.java:137)
        at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:420)
        at hudson.model.Run.run(Run.java:1362)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:145)

      Pipe writer thread: RnDAIX

      "Pipe writer thread: RnDAIX" Id=2729 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ffedd7f
      at sun.misc.Unsafe.park(Native Method)

      • waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ffedd7f
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

      stderr copier for remote agent on RnDAIX

      "stderr copier for remote agent on RnDAIX" Id=2427 Group=main WAITING on java.lang.Object@601a17d
      at java.lang.Object.wait(Native Method)

      • waiting on java.lang.Object@601a17d
        at java.lang.Object.wait(Object.java:485)
        at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213)
        at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171)
        at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)

      Here is some info on the AIX slave:

      Name ↓
      Value
      com.ibm.cpu.endian big
      com.ibm.oti.configuration scar
      com.ibm.oti.jcl.build 20080208_0008
      com.ibm.oti.vm.bootstrap.library.path /usr/java5/jre/bin
      com.ibm.oti.vm.library.version 23
      com.ibm.util.extralibs.properties
      com.ibm.vm.bitmode 32
      file.encoding ISO8859-1
      file.separator /
      ibm.signalhandling.rs false
      ibm.signalhandling.sigchain true
      ibm.signalhandling.sigint true
      ibm.system.encoding ISO8859-1
      invokedviajava
      java.assistive ON
      java.awt.fonts
      java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
      java.awt.printerjob sun.print.PSPrinterJob
      java.class.path slave.jar
      java.class.version 49.0
      java.compiler j9jit23
      java.ext.dirs /usr/java5/jre/lib/ext
      java.fullversion J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20080315 (JIT enabled) J9VM - 20080314_17962_bHdSMr JIT - 20080130_0718ifx2_r8 GC - 200802_08
      java.home /usr/java5/jre
      java.io.tmpdir /tmp/
      java.jcl.version 20080314
      java.library.path /usr/java5/jre/bin:/usr/java5/jre/bin:/usr/java5/jre/bin/classic:/usr/java5/jre/bin:/usr/java5/jre/bin/j9vm:/usr/lib
      java.runtime.name Java(TM) 2 Runtime Environment, Standard Edition
      java.runtime.version pap32dev-20080315 (SR7)
      java.specification.name Java Platform API Specification
      java.specification.vendor Sun Microsystems Inc.
      java.specification.version 1.5
      java.util.prefs.PreferencesFactory java.util.prefs.FileSystemPreferencesFactory
      java.vendor IBM Corporation
      java.vendor.url http://www.ibm.com/
      java.version 1.5.0
      java.vm.info J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20080315 (JIT enabled) J9VM - 20080314_17962_bHdSMr JIT - 20080130_0718ifx2_r8 GC - 200802_08
      java.vm.name IBM J9 VM
      java.vm.specification.name Java Virtual Machine Specification
      java.vm.specification.vendor Sun Microsystems Inc.
      java.vm.specification.version 1.0
      java.vm.vendor IBM Corporation
      java.vm.version 2.3
      jxe.current.romimage.version 9
      jxe.lowest.romimage.version 9
      line.separator
      os.arch ppc
      os.name AIX
      os.version 6.1
      path.separator :
      sun.arch.data.model 32
      sun.boot.class.path /usr/java5/jre/lib/vm.jar:/usr/java5/jre/lib/core.jar:/usr/java5/jre/lib/charsets.jar:/usr/java5/jre/lib/graphics.jar:/usr/java5/jre/lib/security.jar:/usr/java5/jre/lib/ibmpkcs.jar:/usr/java5/jre/lib/ibmorb.jar:/usr/java5/jre/lib/ibmcfw.jar:/usr/java5/jre/lib/ibmorbapi.jar:/usr/java5/jre/lib/ibmjcefw.jar:/usr/java5/jre/lib/ibmjgssprovider.jar:/usr/java5/jre/lib/ibmjsseprovider2.jar:/usr/java5/jre/lib/ibmjaaslm.jar:/usr/java5/jre/lib/ibmcertpathprovider.jar:/usr/java5/jre/lib/server.jar:/usr/java5/jre/lib/xml.jar
      sun.boot.library.path /usr/java5/jre/bin
      sun.io.unicode.encoding UnicodeBig
      sun.java2d.fontpath
      sun.jnu.encoding ISO8859-1
      svnkit.ssh2.persistent false
      user.country US
      user.dir /hudson
      user.home /
      user.language en
      user.name root
      user.timezone GMT-06:00
      user.variant
      Environment Variables

      Name ↓
      Value
      _ /usr/java5/jre/bin/java
      AIXTHREAD_SCOPE S
      AUTHSTATE files
      HOME /
      IBM_JAVA_COMMAND_LINE java -jar slave.jar
      IBM_JVM_AIXTHREAD_SCOPE_NEW_VALUE S
      IBM_JVM_CHANGED_ENVVARS_340144 LIBPATH,AIXTHREAD_SCOPE,LDR_CNTRL
      IBM_JVM_LDR_CNTRL_NEW_VALUE MAXDATA=0XA0000000@DSA
      IBM_JVM_LIBPATH_NEW_VALUE /usr/java5/jre/bin:/usr/java5/jre/bin/classic:/usr/java5/jre/bin
      LANG en_US
      LC__FASTMSG true
      LDR_CNTRL MAXDATA=0XA0000000@DSA
      LIBPATH /usr/java5/jre/bin:/usr/java5/jre/bin/classic:/usr/java5/jre/bin:/usr/java5/jre/bin/j9vm
      LOCPATH /usr/lib/nls/loc
      LOGIN root
      LOGNAME root
      MAIL /var/spool/mail/root
      NLSPATH /usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
      ODMDIR /etc/objrepos
      OLDPWD /
      PATH /usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java5/jre/bin:/usr/java5/bin
      PWD /hudson
      SHELL /usr/bin/bash
      SHLVL 1
      SSH_CLIENT 10.1.2.12 43907 22
      SSH_CONNECTION 10.1.2.12 43907 10.1.2.10 22
      TZ MST6
      USER root

      Slave dump:

      Thread Dump

      Channel reader thread: channel

      java.io.FileInputStream.readBytes(Native Method)
      java.io.FileInputStream.read(FileInputStream.java:229)
      java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
      java.io.BufferedInputStream.read(BufferedInputStream.java:246)
      java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2220)
      java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2510)
      java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2520)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:363)
      hudson.remoting.Channel$ReaderThread.run(Channel.java:947)
      main

      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:199)
      hudson.remoting.Channel.join(Channel.java:728)
      hudson.remoting.Launcher.main(Launcher.java:418)
      hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:364)
      hudson.remoting.Launcher.run(Launcher.java:204)
      hudson.remoting.Launcher.main(Launcher.java:166)
      Ping thread for channel hudson.remoting.Channel@7d9a7d9a:channel

      java.lang.Thread.sleep(Native Method)
      java.lang.Thread.sleep(Thread.java:938)
      hudson.remoting.PingThread.run(PingThread.java:86)
      Pipe writer thread: channel

      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:169)
      java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1793)
      java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:379)
      java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:485)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:689)
      java.lang.Thread.run(Thread.java:810)
      pool-1-thread-1

      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:199)
      java.lang.UNIXProcess.waitFor(UNIXProcess.java:309)
      hudson.Proc$LocalProc.join(Proc.java:226)
      hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:795)
      hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:767)
      hudson.remoting.UserRequest.perform(UserRequest.java:114)
      hudson.remoting.UserRequest.perform(UserRequest.java:48)
      hudson.remoting.Request$2.run(Request.java:270)
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:432)
      java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
      java.util.concurrent.FutureTask.run(FutureTask.java:138)
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
      java.lang.Thread.run(Thread.java:810)
      pool-1-thread-2

      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:197)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:792)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1107)
      java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:306)
      java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:458)
      java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:490)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:689)
      java.lang.Thread.run(Thread.java:810)
      pool-1-thread-3

      java.lang.Thread.getStackTraceImpl(Native Method)
      java.lang.Thread.getStackTrace(Thread.java:1144)
      java.lang.Thread.getAllStackTraces(Thread.java:1172)
      hudson.Functions.dumpAllThreads(Functions.java:779)
      hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
      hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:81)
      hudson.remoting.UserRequest.perform(UserRequest.java:114)
      hudson.remoting.UserRequest.perform(UserRequest.java:48)
      hudson.remoting.Request$2.run(Request.java:270)
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:432)
      java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
      java.util.concurrent.FutureTask.run(FutureTask.java:138)
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
      java.lang.Thread.run(Thread.java:810)
      process reaper java.lang.UNIXProcess@54565456

      java.lang.UNIXProcess.waitForProcessExit(Native Method)
      java.lang.UNIXProcess.access$900(UNIXProcess.java:56)
      java.lang.UNIXProcess$2$1.run(UNIXProcess.java:271)
      Signal Dispatcher

      com.ibm.misc.SignalDispatcher.waitForSignal(Native Method)
      com.ibm.misc.SignalDispatcher.run(SignalDispatcher.java:84)
      Finalizer thread

      Gc Slave Thread

      JIT Compilation Thread

        Attachments

          Issue Links

            Activity

            Hide
            dougborg Doug Borg added a comment -

            I think this may be related to JENKINS-8420. I bet the build isn't actually getting stuck; it just appears so since there is no log output. I will upgrade to 1.399 and test this later today.

            Show
            dougborg Doug Borg added a comment - I think this may be related to JENKINS-8420 . I bet the build isn't actually getting stuck; it just appears so since there is no log output. I will upgrade to 1.399 and test this later today.
            Hide
            evernat evernat added a comment -

            Hi Doug,
            You said you will test. Any news now?

            Show
            evernat evernat added a comment - Hi Doug, You said you will test. Any news now?
            Hide
            evernat evernat added a comment -

            No response from the reporter, so resolving as incomplete.

            Show
            evernat evernat added a comment - No response from the reporter, so resolving as incomplete.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              dougborg Doug Borg
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: