-
Bug
-
Resolution: Duplicate
-
Major
-
None
I was investigating why my jenkins.log file is 13GB (even after rotating daily, and it's only halfway through the day currently), and noticed that the majority of the file (so far I've only managed to confirm up to the first 1 million lines of the file) comes from an exception emitted by the Multiple-SCMs plugin:
org.xml.sax.SAXParseException: XML document structures must start and end within the same entity. at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser.parse(MultiSCMChangeLogParser.java:140) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser$LogSplitter.endElement(MultiSCMChangeLogParser.java:107) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser.parse(MultiSCMChangeLogParser.java:140) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:826) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:800) at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:362) at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1389) at hudson.model.User.getProjects(User.java:409) at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21) at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) at org.jenkinsci.plugins.gravatar.UserGravatarResolver.findAvatarFor(UserGravatarResolver.java:67) at org.jenkinsci.plugins.gravatar.GravatarCheckAsyncPeriodicWork.execute(GravatarCheckAsyncPeriodicWork.java:56) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:636)
The way I confirmed up to the first 1 million lines is using head/tail:
# head /var/log/jenkins/jenkins.log -n 1000000 | tail at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
I'm trying to use wc -l to figure out exactly how many lines are in the file to begin with, and after several minutes of waiting, I see the current log file is 160 million lines and 13 GB.
On further checks, I'm now confirming that up to 50 million lines is the same exception:
# head /var/log/jenkins/jenkins.log -n 50000000 | tail at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser.parse(MultiSCMChangeLogParser.java:140) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser$LogSplitter.endElement(MultiSCMChangeLogParser.java:107) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
And just for kicks, I checked the end of the first 100 million lines... Yup, you guessed it:
# head /var/log/jenkins/jenkins.log -n 100000000 | tail at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
Here is an example of one of my builds' changelog.xml file (I've removed most of the internal information). The SCM polling we use is GitSCM:
<multi-scm-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch origin/HEAD, between f2e9e5b564641405d5eccabd2f60fd74d05eb10f and f2e9e5b564641405d5eccabd2f60fd74d05eb10f Changes in branch origin/master, between f2e9e5b564641405d5eccabd2f60fd74d05eb10f and f2e9e5b564641405d5eccabd2f60fd74d05eb10f ]]> </sub-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch base/master, between dc78f64a3ae4a006aacd83e6d247ef9770c910bd and dc78f64a3ae4a006aacd83e6d247ef9770c910bd ]]> </sub-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch common/master, between f1df64664f9a4573a812a307e22e365010ce6d02 and c6c5cb40bbc2f1dda11c8c47756adcb28b41961e commit c6c5cb40bbc2f1dda11c8c47756adcb28b41961e tree 83e64f5dda4e20d664e02afc5bf7fa04f5cbcf98 parent f1df64664f9a4573a812a307e22e365010ce6d02 author XXXX 1339444234 -0400 committer XXXX 1339604443 -0400 XXXX Change-Id: I35edfcb6b5dee97c8eed0b17218b8f440917c11c Reviewed-on: http://gerrit.xxx :100644 100644 0c5996cf2cafb64fd4643ce8cb7babc0b71d557b cefd6dbfbd9634a2784c42ec87119b3977c15af0 M Path/Files.. :100644 100644 c46e7c64a04520d4143a257cb826cc18de306610 eb254e580056a7b2b67e315dc7fe90f55f661c76 M Path/Files.. :100644 100644 66f985386cf7c3b9df445cfe6d5fc94a0ddd295e 33e4c22951a6f61034b7bf1ad9fd49f79a45daee M Path/Files.. ... ]]> </sub-log> </multi-scm-log>
The sheer size of this log file causes logrotate to flush all kernel cache buffers while it has to compress the log file, and causes the memory consumption pattern to look like the attached graph.
- is related to
-
-
- Open
-
[JENKINS-14099] Excessive exception logging
I tried to remove the Gravatar plugin, to see if that was part of the problem (since it's at the bottom of the stack), and after restarting, I'm now seeing the same exception, with a different trace:
org.xml.sax.SAXParseException: XML document structures must start and end within the same entity. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.scanData(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser.parse(MultiSCMChangeLogParser.java:140) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser$LogSplitter.endElement(MultiSCMChangeLogParser.java:107) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.jenkinsci.plugins.multiplescms.MultiSCMChangeLogParser.parse(MultiSCMChangeLogParser.java:140) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:832) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:806) at hudson.model.View$People.isApplicable(View.java:674) at hudson.model.View.hasPeople(View.java:596) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTOrNode.value(ASTOrNode.java:61) at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54) at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71) at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488) at org.kohsuke.stapler.Stapler.service(Stapler.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:171) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636)
Still coming from Multiple SCMs plugin, just a different entry this time.
On further investigation, it looks like this is caused by using a combination of "Multiple SCMs" and the "Template Project" plugins.
On the project(s) where this happens, I have this type of project configuration:
- Template-Project
- SCM: Multiple SCMs
- Git: git://repo1
- Git: git://repo2
- SCM: Multiple SCMs
- Real-Project
- SCM: Multiple SCMs
- Use SCM from another project: Template-Project
- Git: git://repo3
- SCM: Multiple SCMs
The error occurs on an XML file that looks like this:
<multi-scm-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch origin/master, between 9c4074dcff82c18ed73fac20a99d6f607c841124 and 6fe4697a307b79944336ba8c3668fdb4c7c3deb3 commit 6fe4697a307b79944336ba8c3668fdb4c7c3deb3 tree 1ae70d49e3188b26239550591ae19a8e94cfe842 parent 9c4074dcff82c18ed73fac20a99d6f607c841124 author XXX 1339536117 -0400 committer XXX 1339623892 -0400 INTER-147 title on profile page Change-Id: Ic7e5acf8d0a854024548adab639e8497e861b5f1 :100644 100644 74c6e402f2243fce6b6c298ecf46e4b108994a2f 0ef7427344375aa5fea5b44d2ec12934e940f558 M path/to/files.. :100644 100644 087006cc767d0ff5b5480ffdc2a92540e56dc8c3 cf3d1021387d00cc5bce12ae0ccf80e699612bbe M path/to/files.. ]]> </sub-log> <sub-log scm="hudson.plugins.templateproject.ProxySCM"> <![CDATA[<multi-scm-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch base/HEAD, between 895a00c4d17312a78f2626e781c0492751685396 and 895a00c4d17312a78f2626e781c0492751685396 Changes in branch base/master, between 895a00c4d17312a78f2626e781c0492751685396 and 895a00c4d17312a78f2626e781c0492751685396 ]]> </sub-log> <sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch common/HEAD, between ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 and ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 Changes in branch common/master, between ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 and ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 ]]> </sub-log> </multi-scm-log> ]]> </sub-log> </multi-scm-log>
In reality, the individual XML files are:
- repo1:
<sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch common/HEAD, between ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 and ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 Changes in branch common/master, between ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 and ce491ecc5ee0c9d83f3fc3b57033af9b686d4bd7 ]]> </sub-log> - repo2:
<sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch base/HEAD, between 895a00c4d17312a78f2626e781c0492751685396 and 895a00c4d17312a78f2626e781c0492751685396 Changes in branch base/master, between 895a00c4d17312a78f2626e781c0492751685396 and 895a00c4d17312a78f2626e781c0492751685396 ]]> </sub-log> - repo3:
<sub-log scm="hudson.plugins.git.GitSCM"> <![CDATA[Changes in branch origin/master, between 9c4074dcff82c18ed73fac20a99d6f607c841124 and 6fe4697a307b79944336ba8c3668fdb4c7c3deb3 commit 6fe4697a307b79944336ba8c3668fdb4c7c3deb3 tree 1ae70d49e3188b26239550591ae19a8e94cfe842 parent 9c4074dcff82c18ed73fac20a99d6f607c841124 author XXX 1339536117 -0400 committer XXX 1339623892 -0400 INTER-147 title on profile page Change-Id: Ic7e5acf8d0a854024548adab639e8497e861b5f1 :100644 100644 74c6e402f2243fce6b6c298ecf46e4b108994a2f 0ef7427344375aa5fea5b44d2ec12934e940f558 M path/to/files.. :100644 100644 087006cc767d0ff5b5480ffdc2a92540e56dc8c3 cf3d1021387d00cc5bce12ae0ccf80e699612bbe M path/to/files.. ]]> </sub-log>
But each sub-log entry combines its own log data into a CDATA section – including hudson.plugins.templateproject.ProxySCM. ProxySCM is encapsulating a multi-scm-log containing to GitSCM sub-logs. Since each sub-log is wrapped in CDATA, the ProxySCM CDATA is ended early, and that results in an invalid XML file.
The root cause is resolved in JENKINS-14537 (pending the code being merged and an update released). However, these stack traces will still be dumped for any existing changelog.xml files that contain the bad formatting. The fix only works for files created after the fix has been released and installed. It would be possible to transform old changelog.xml files into the new format to try to resolve that, but it may not be 100% perfect. I will post back with a set of commands that would help do that, once the update is released.
The error indicates a truncated XML file (probably 0 byte file, which appears to gets created in some error conditions.
I think the first step is for MultiSCMChangeLogParser to report a file name when it fails to parse a faile.