-
Bug
-
Resolution: Unresolved
-
Major
-
Ubuntu Linux 11.04 x86
-
Powered by SuggestiMate
Sometimes, when doing a build, it starts to hang at the SVN update. Clicking the abort-button does not have any effect, the build processor doing the build is thus blocked.
Only restarting Jenkins helps then.
We are doing the SVN update via a external script as a workaround, which isn't really convenient.
[JENKINS-13538] Build hangs at SVN update occasionally
I just did a apt-get update on the Ubuntu slave and rebooted it. It seems to have cleared up my hanging build.
I still have this problem in 1.491 (don't have more recent yet), but already tried recent svn plugin versions (1.40-1.45).
I found out that using in svn urls http instead of https does help avoiding the hanging issue.
I've tested this running 2 jobs (basically just doing svn update) using the exact same url every 2 minutes.
This is basically a workaround since it may mean the plugin (or svnkit) has some problem with handling https urls.
If the Slave was invoked in older versions of Jenkins or Hudson, it would have created some (supporting) files in the node's workspace directory.
In my setup, If i clear those files and invoke the slave agent again, it checks-out properly without hanging.
I am still facing this issue. How did you guys come around this issue.
Jenkins ver. 1.531
Jenkins Subversion Plug-in - 2.2
subversion server :svn, version 1.5.6 (r36142), workspace version "svn, version 1.6.11 (r934486)
Jenkins server - CentOS Linux release 6.0 (Final)
Jenkins Slave - CentOS Linux release 6.0 (Final)
Jenkins Slave - Solaris 10 10/08 s10x_u6wos_07b X86
exact same issue. suddenly the svn co hangs right after the "At revision" message when checking out on ssh slave (debian). None of the build steps is executed.
master: v1.583
slave: debian, oracle jdk1.6.0_45
updating java from 1.6 => 1.7 on the slave node did the trick.
Experiencing this issue, and now desperately seeking a resolution. Have attempted with multiple versions of Jenkins and the SVN plugin.
One thing I've identified, regards SVN sometimes hanging at "At revision" and then failing, but not for other builds, is that (for me) this is consistently happening when SVN detects a change in the repository.
i.e. If SVN detects no change in the repository, the build will succeed. If SVN does detect a change, the build will hang at "At revision x" indefinitely.
This has the effect, that every time a change to the repository is committed, the first Jenkins build following will fail. All subsequent builds (where no change is detected) will then succeed without issue, until another change is committed.
Does this match behaviour other people are experiencing with this issue?
Master
Jenkins - 1.631
OS - Ubuntu Linux
Java - 1.7.0_75
SVN Plugin - 2.5.3
Slave
Unix slave, version 2.52
OS - Mac OS X Yosemite 10.10.5
Java - 1.8.0_45
Console output of failed build:
Started by user Robbie Building remotely on mac-mini-slave (unity ios mac unity-ios) in workspace <redacted> Cleaning local Directory . Checking out svn+ssh://<redacted> at revision '2015-10-12T18:31:07.755 +0100' A Assets AU Assets/TestScene.unity <redacted> U . At revision 17 *< Will hang here for some time, until I presume the build attempt times out or is cancelled* hudson.util.IOException2: revision check failed on svn+ssh://<redacted> at hudson.scm.SubversionChangeLogBuilder.buildModule(SubversionChangeLogBuilder.java:196) at hudson.scm.SubversionChangeLogBuilder.run(SubversionChangeLogBuilder.java:123) at hudson.scm.SubversionSCM.calcChangeLog(SubversionSCM.java:726) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:861) at hudson.scm.SCM.checkout(SCM.java:485) at hudson.model.AbstractProject.checkout(AbstractProject.java:1277) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) Caused by: org.tmatesoft.svn.core.SVNException: svn: E210004: Malformed network data at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.svn.SVNReader.readChar(SVNReader.java:473) at org.tmatesoft.svn.core.internal.io.svn.SVNReader.skipWhiteSpace(SVNReader.java:485) at org.tmatesoft.svn.core.internal.io.svn.SVNReader.readTuple(SVNReader.java:287) at org.tmatesoft.svn.core.internal.io.svn.SVNReader.parse(SVNReader.java:241) at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.read(SVNConnection.java:276) at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.authenticate(SVNConnection.java:174) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.authenticate(SVNRepositoryImpl.java:1276) at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.logImpl(SVNRepositoryImpl.java:728) at org.tmatesoft.svn.core.io.SVNRepository.log(SVNRepository.java:1038) at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:181) at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:35) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1259) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:968) at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:873) at hudson.scm.SubversionChangeLogBuilder.buildModule(SubversionChangeLogBuilder.java:184) ... 12 more Skipped archiving because build is not successful Publish artifacts to S3 Bucket Using S3 profile: Jenkins S3 Profile Publish artifacts to S3 Bucket Skipping publishing on S3 because build failed Finished: FAILURE
For successful builds:
At revision 16 no change for svn+ssh://<redacted> since the previous build Piping unity Editor.log from /Users/jenkins/Library/Logs/Unity/Editor.log <redacted> *(will then proceed to build successfully without issue)* Finished: SUCCESS
Does anyone have an update on this one? We experience those issue as well. Have some pipeline jobs running regularly to reproduce it. Unfortunetly the timeout step is not able to kill the job run in the case of an error. The only thing that helps then will be using the hard kill.
This also happens on our side. We are doing the checkout inside the job file, like this
checkout changelog: false, poll: false, scm: [$class: 'SubversionSCM', additionalCredentials: [], excludedCommitMessages: '', excludedRegions: '', excludedRevprop: '', excludedUsers: '', filterChangelog: false, ignoreDirPropChanges: false, includedRegions: '', locations: [[cancelProcessOnExternalsFail: true, credentialsId: 'blabla', depthOption: 'infinity', ignoreExternalsOption: true, local: './SVNTranslation', remote: 'https://blabla/SVNTranslation']], quietOperation: true, workspaceUpdater: [$class: 'UpdateUpdater']]
It hangs with no timeout, intervention is needed to abort, or else is hanging like this. I'm also using a linux machine.
I also found another workaround - renaming the job. If this is done, probably it doesn't happen anymore because there is a complete new workspace in that case. But after more running of the same job - more attempts to update checkout - it happens again. I'm not sure if it happens immediatley after the job is aborted, but i think there is a problem concerning the workspace somehow. Can't tell exactly.
I have a build job that is hanging the same way (Jenkins 1.466 and 1.474). My Jenkins master is running on RHEL5 and the slave Ubuntu (Linux wookie 2.6.32-38-generic #83-Ubuntu SMP Wed Jan 4 11:12:07 UTC 2012 x86_64 GNU/Linux).
Running top on the slave computer shows the build slave process ranging from 190% to 220% CPU.
Here's the thread dump on the slave:
Thread Dump
Channel reader thread: channel
"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at hudson.remoting.Command.readFrom(Command.java:90)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
main
"main" Id=1 Group=main WAITING on hudson.remoting.Channel@452bb7e0
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at hudson.remoting.Channel.join(Channel.java:785)
at hudson.remoting.Launcher.main(Launcher.java:420)
at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366)
at hudson.remoting.Launcher.run(Launcher.java:206)
at hudson.remoting.Launcher.main(Launcher.java:168)
Ping thread for channel hudson.remoting.Channel@452bb7e0:channel
"Ping thread for channel hudson.remoting.Channel@452bb7e0:channel" Id=10 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:86)
Pipe writer thread: channel
"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@79fcfede
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
pool-1-svnkit-thread-1
"pool-1-svnkit-thread-1" Id=17 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@9257e6f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
pool-1-thread-1
"pool-1-thread-1" Id=11 Group=main TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.sleepForTimestamp(SVNFileUtil.java:1350)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.sleepForTimeStamp(SVNBasicDelegate.java:316)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:575)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:965)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1221)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:292)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:781)
at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:85)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:144)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:789)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:770)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:753)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2196)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Number of locked synchronizers = 1
pool-1-thread-4
"pool-1-thread-4" Id=19 Group=main RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:433)
at hudson.Functions.getThreadInfos(Functions.java:873)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Number of locked synchronizers = 1
svn log copier
"svn log copier" Id=16 Group=main TIMED_WAITING on java.io.PipedInputStream@7c8fae19
at java.lang.Object.wait(Native Method)
at java.io.PipedInputStream.read(PipedInputStream.java:310)
at java.io.PipedInputStream.read(PipedInputStream.java:361)
at java.io.InputStream.read(InputStream.java:85)
at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)
Timer-0
"Timer-0" Id=15 Group=main TIMED_WAITING on java.util.TaskQueue@3cef2b32
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
Finalizer
"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@2f8ffdc4
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Reference Handler
"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@165d6741
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Signal Dispatcher
"Signal Dispatcher" Id=4 Group=system RUNNABLE