Loading...

XML

Word

Printable

Type: Bug
Resolution: Fixed
Priority: Blocker
Component/s: core, subversion-plugin
Labels:
None
Environment:
Linux kriek 2.6.26-2-amd64 #1 SMP Sun Jun 20 20:16:30 UTC 2010 x86_64 GNU/Linux
java version "1.6.0_19"
Java(TM) SE Runtime Environment (build 1.6.0_19-b04)
Java HotSpot(TM) 64-Bit Server VM (build 16.2-b04, mixed mode)

Similar Issues:

Show

A few weeks ago we upgraded from 1.372 to 1.381 which solved a significant amount of issues (many thanks for that!).
Along with the upgrade of Hudson, we upgraded the Subversion Hudson plugin from 1.18 to 1.20.
Shortly after the upgrade of these two, we started noticing that Hudson was not triggering builds, despite the fact that Subversion was indeed notifying it of commits. We have the SCM Subversion polling set to:

*/30 * * * *

thus, almost completely relying on the post-commit-hook to trigger the build (as opposed to the SCM polling getting stuffed up gradually).

The Subversion post-commit-hook works correctly and is sending a request, as can be seen below.
This is what the successful output should look like:

--17:30:49--  http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit?rev=71423
           => `/tmp/svn-notify-commit-post'
Resolving hudson... 192.168.0.41
Connecting to hudson[192.168.0.41]:8082... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0

    0K                                                           0.00 B/s

This is the sort of output we currently have:

--18:09:09--  http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit?rev=71424
           => `/tmp/svn-notify-commit-post'
Resolving hudson... 192.168.0.41
Connecting to hudson[192.168.0.41]:8082... connected.
HTTP request sent, awaiting response... 500 READER
18:09:09 ERROR 500: READER.

This is an excerpt from the Jetty log:

2010-11-03 14:09:56.888:WARN:/hudson:Error while serving http://hudson:8082/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:259)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:126)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:65)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:79)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:453)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:135)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1216)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:421)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:456)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:924)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
	at org.eclipse.jetty.server.Server.handle(Server.java:335)
	at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
	at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
	at java.lang.Thread.run(Thread.java:619)
2010-11-03 14:09:56.889:WARN::/hudson/subversion/c5784c02-6b0d-0410-910a-a7a69de433da/notifyCommit
java.lang.IllegalStateException: READER
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:508)
	at org.eclipse.jetty.server.Request.extractParameters(Request.java:251)
	at org.eclipse.jetty.server.Request.getParameter(Request.java:648)
	at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:169)
	at hudson.scm.SubversionRepositoryStatus.doNotifyCommit(SubversionRepositoryStatus.java:84)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:259)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:126)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:65)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:79)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:30)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:359)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:537)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:453)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:135)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1216)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:421)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:456)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:924)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
	at org.eclipse.jetty.server.Server.handle(Server.java:335)
	at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
	at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
	at java.lang.Thread.run(Thread.java:619)
Nov 3, 2010 2:11:03 PM hudson.scm.PerJobCredentialStore <init>
INFO: Failed to retrieve Subversion credentials from /local/hudson/jobs/FAS-6199/subversion.credentials
hudson.util.IOException2: Unable to read /local/hudson/jobs/FAS-6199/subversion.credentials
	at hudson.XmlFile.unmarshal(XmlFile.java:152)
	at hudson.scm.PerJobCredentialStore.<init>(PerJobCredentialStore.java:38)
	at hudson.scm.SubversionSCM$DescriptorImpl.createAuthenticationProvider(SubversionSCM.java:1659)
	at hudson.scm.SubversionSCM$DescriptorImpl.getRepository(SubversionSCM.java:1858)
	at hudson.scm.SubversionSCM$DescriptorImpl.checkRepositoryPath(SubversionSCM.java:1831)
	at hudson.scm.SubversionSCM.repositoryLocationsNoLongerExist(SubversionSCM.java:1998)
	at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1045)
	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:353)
	at hudson.scm.SCM.poll(SCM.java:370)
	at hudson.model.AbstractProject.poll(AbstractProject.java:1218)
	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: com.thoughtworks.xstream.converters.ConversionException: No InvocationHandler specified for dynamic proxy
---- Debugging information ----
class               : com.thoughtworks.xstream.mapper.DynamicProxyMapper$DynamicProxy
required-type       : com.thoughtworks.xstream.mapper.DynamicProxyMapper$DynamicProxy
path                : /hudson.scm.PerJobCredentialStore
line number         : 13
-------------------------------
	at com.thoughtworks.xstream.converters.extended.DynamicProxyConverter.unmarshal(DynamicProxyConverter.java:92)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:137)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:33)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:926)
	at hudson.util.XStream2.unmarshal(XStream2.java:80)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:912)
	at hudson.XmlFile.unmarshal(XmlFile.java:148)
	... 18 more

The error above looks like a problem with the deserialization of the per-module subversion.credentials file below:

<?xml version='1.0' encoding='UTF-8'?>
<hudson.scm.PerJobCredentialStore resolves-to="dynamic-proxy">
  <h class="hudson.remoting.RemoteInvocationHandler" serialization="custom">
    <hudson.remoting.RemoteInvocationHandler>
      <default>
        <autoUnexportByCaller>false</autoUnexportByCaller>
        <goingHome>false</goingHome>
        <oid>9271</oid>
        <userProxy>true</userProxy>
      </default>
    </hudson.remoting.RemoteInvocationHandler>
  </h>
</hudson.scm.PerJobCredentialStore>

Rolling back to 1.18 did not solve the problem. In addition, I tried the following:

Create a new job which checks out a project from the same Subversion repository.

Commit a changed file.
The build worked and got triggered. So then I figured I should stop Hudson and write a simple script to replace all per-project credential files (they all have the same content, except there is a serious difference between a working one and one of an older project). This did not work. For some reason Hudson would just re-overwrite these files with the old broken ones. Below you can see a subversion.credentials file for a working module:

<?xml version='1.0' encoding='UTF-8'?>
<hudson.scm.PerJobCredentialStore>
  <credentials class="hashtable">
    <entry>
      <string>svn+ssh://subversion</string>
      <hudson.scm.SubversionSCM_-DescriptorImpl_-SshPublicKeyCredential>
        <userName>some_user</userName>
        <passphrase></passphrase>
        <id>85c531bf25d7ae61</id>
      </hudson.scm.SubversionSCM_-DescriptorImpl_-SshPublicKeyCredential>
    </entry>
  </credentials>
</hudson.scm.PerJobCredentialStore>

As you can see the above files have little in common.

I have also tried upgrading to 1.384. That didn't fix it either and actually caused nodes to die, so I rolled back to 1.381.

I have also looked at the PerJobCredentialStore class. I didn't find anything suspicious there (in fact, if I recall correctly, this class hasn't been changed in about half a year).

A fix for this would be highly appreciated, as we have some 200 modules which now rely solely on a build triggered every half hour.

Please advise and many thanks in advance,

Martin

Assignee:: Kohsuke Kawaguchi

Reporter:: carlspring

Votes:: 2 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2010-11-09 16:13

Updated:: 2011-01-06 17:16

Resolved:: 2011-01-06 17:14

Details

Description

Attachments

Activity

People

Dates