-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Hudson 1.347, Mercurial plugin 1.25, Tomcat 6.0.20, Linux 2.6.18
-
Powered by SuggestiMate
Since we upgraded to Hudson 1.347 and the Mercurial plugin 1.25 the SCM polling stops working once a day and we have to restart Tomcat. These are the symptoms:
- The "Last Mercurial Polling Log" of any job shows an empty output, without any "hg incoming" command as usual:
Last Mercurial Polling Log Started on Feb 25, 2010 6:21:24 AM
- The Tomcat catalina.out log shows the following error all the time:
Feb 25, 2010 7:57:24 AM hudson.triggers.SCMTrigger$Runner runPolling SEVERE: Failed to record SCM polling java.lang.AbstractMethodError at hudson.scm.SCM.poll(SCM.java:344) at hudson.model.AbstractProject.poll(AbstractProject.java:1137) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:346) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
Thanks.
- is blocking
-
JENKINS-2180 Quiet period keeps running without building
-
- Closed
-
- is duplicated by
-
JENKINS-5866 AbstractMethodError from SCM.poll
-
- Closed
-
-
JENKINS-5882 ERROR: Failed to record SCM polling
-
- Closed
-
- is related to
-
JENKINS-7546 Getting AbstractMethodError on all top-level (non job) pages after 1.377 upgrade
-
- Resolved
-
[JENKINS-5756] Poll SCM fails with java.lang.AbstractMethodError
After upgrading to 1.348 it continues happening to me. But now the exception is also shown in the SCM polling log:
Started on Mar 4, 2010 7:40:41 PM Failed to record SCM polling java.lang.AbstractMethodError at hudson.scm.SCM.poll(SCM.java:344) at hudson.model.AbstractProject.poll(AbstractProject.java:1137) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:348) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
We have the same problem after updating to latest Hudson version (and after updating SVN plugin):
Failed to record SCM polling java.lang.AbstractMethodError at hudson.scm.SCM.poll(SCM.java:344) at hudson.model.AbstractProject.poll(AbstractProject.java:1137) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:348) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
There's now the same problem with SVN:
Failed to record SCM polling java.lang.ClassCastException: hudson.scm.SCMRevisionState$None cannot be cast to hudson.scm.SVNRevisionState at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1051) at hudson.scm.SCM.poll(SCM.java:342) at hudson.model.AbstractProject.poll(AbstractProject.java:1150) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:348) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
Hm, I actually do not believe this is a mercurial plugin problem, then. I am running Hudson 1.349 with Mercurial Plugin 1.25 on Linux (Winstone, java version "1.6.0_17" and I am getting these errors for the SVN-plugin:
SEVERE: Failed to record SCM polling
java.lang.ClassCastException: hudson.scm.SCMRevisionState$None cannot be cast to hudson.scm.SVNRevisionState
at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1051)
After switching back from subversion-plugin 1.12 to 1.11 the error is gone for me, so I switch the component to the subversion-plugin.
There is a fix for the AbstractMethodError in http://github.com/kohsuke/hudson/blob/1.346/core/src/main/java/hudson/scm/SCM.java#L340, so this should be gone from Hudson 1.346 on.
@Tomcat Users: are you sure tomcat cleaned the webapps directory correctly?
Regards
Mirko
These are two separate (albeit related) issues - I've seen the AbstractMethodError pop up with the git plugin, with only version 1.11 of the subversion plugin installed. They're both related to the SCM changes for better quiet period support, but I'm pretty sure they're actually different.
@mfriedenhagen - sadly, that fixed a different AbstractMethodError. This one's cropping up on the return in the catch there.
For what it's worth, I can reproduce this issue with the Subversion plugin version at 1.11. However note that it's just installed, I'm only using the Mercurial plugin.
Which of the stacktraces are you seeing? The AbstractMethodError or the ClassCastException?
Moving this to core - see JENKINS-5827 for the Subversion ClassCastException.
This issue doesn't seem confusing enough, just to make it worse, with 1.349 and svn 1.12 (no mercurial installed at all) I'm getting:
ERROR: Failed to record SCM polling [abcd:AAAAVx+LCAAAAAAAAASmjNKU4PiIQ+08vOaTBb895aBtbqvOWtSEp4tKMD8nVMz/PCaAgBF2FTYp6C22JmJgZIAAR6pDYALL70maGi2iEEKTYMzhAYp1WF6Nq==[0mjava.lang.ClassCastException: hudson.scm.SCMRevisionState$None cannot be cast to hudson.scm.SVNRevisionState at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1051) at hudson.scm.SCM.poll(SCM.java:342) at hudson.model.AbstractProject.poll(AbstractProject.java:1162) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:348) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
(Big bit of base64 cowardly damaged as I have no idea what it contains)
I upgrade by stoptomcat && rm -rf hudson hudson.war && wget hudson.war && starttomcat, so it's unlikely to be latent unpacked classes?
Again, see JENKINS-5827 for the Subversion ClassCastException you're running into, faux.
I've tried both mercurial 1.25 and git 0.8, on Hudson 1.348/1.349/1.350-SNAPSHOT but so far I'm failing to reproduce the problem.
hudson.scm.SCM.poll(SCM.java:344) calls into a method that's not abstract (the method is defined in the same class), so I'm puzzled as to why this can happen.
I managed to reproduce the problem. I eventually wrote a small test case outside Hudson, and I now think this is a HotSpot problem.
I filed bug 6933067, which should show up in BugParade eventually. The text of the bug report is as follows:
AbstractMethodError escapes try/catch block.
I think I've found a bug in JVM wrt a call to an abstract method.
I have the following classes:
public abstract class SCM { public abstract void foo(); public void bar() {} public void poll() { try { System.out.println("Calling a possible abstract method"); foo(); System.out.println("Done"); } catch (AbstractMethodError x) { System.out.println("Caught AME"); bar(); System.out.println("Resuming"); } } } public class SCM1 extends SCM { @Override public void foo() {} }
There's also a separate SCM2 class, which is compiled separately with the different version of the SCM class that doesn't have any abstract method.
public class SCM2 extends SCM {}
Just to check sanity, the de-compiled SCM2 class looks like this:
% javap -verbose test.SCM2 Compiled from "SCM2.java" public class test.SCM2 extends test.SCM SourceFile: "SCM2.java" minor version: 0 major version: 50 Constant pool: const #1 = Method #3.#10; // test/SCM."<init>":()V const #2 = class #11; // test/SCM2 const #3 = class #12; // test/SCM const #4 = Asciz <init>; const #5 = Asciz ()V; const #6 = Asciz Code; const #7 = Asciz LineNumberTable; const #8 = Asciz SourceFile; const #9 = Asciz SCM2.java; const #10 = NameAndType #4:#5;// "<init>":()V const #11 = Asciz test/SCM2; const #12 = Asciz test/SCM; { public test.SCM2(); Code: Stack=1, Locals=1, Args_size=1 0: aload_0 1: invokespecial #1; //Method test/SCM."<init>":()V 4: return LineNumberTable: line 3: 0 }
Finally, I have the driver class to put the whole thing together. Notice that in App the call with SCM1 is commented out for now.
public class App { public static void main( String[] args ) { // test(new SCM1()); test(new SCM2()); } public static void test(SCM scm) { try { scm.poll(); } catch (Throwable e) { e.printStackTrace(); } } }
This results in the following output. Nothing surprising.
% java test.App
Calling a possible abstract method
Caught AME
Resuming
But if I bring back the "test(new SCM1())" line, I get this:
% java test.App Calling a possible abstract method Done Calling a possible abstract method java.lang.AbstractMethodError at test.SCM.poll(SCM.java:20) at test.App.test(App.java:17) at test.App.main(App.java:12)
SCM.java line 20 refers to the end bracket of the poll() method. As you can see, AbstractMethodError is escaping the try/catch block. I tested this with JDK6 u15 b03, but I have reports from users of my program that this problem happens on other versions of JRE.
Furthermore, when I tried the same program with the debug build of JDK6u18, I get the following assertion failure, which made me believe that this is a bug in JVM:
% /usr/local/java6u18-debug/bin/java test.App Calling a possible abstract method Done Calling a possible abstract method =============== DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_leaf_base: last_sp != NULL ================ # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/methodOop.cpp:164 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/BUILD_AREA/jdk6_18/hotspot/src/share/vm/oops/methodOop.cpp:164), pid=24412, tid=140588300966160 # Error: assert(is_native() && bcp == code_base() || contains(bcp) || is_error_reported(),"bcp doesn't belong to this method") # # JRE version: 6.0_18-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13-fastdebug mixed mode linux-amd64 ) # An error report file with more information is saved as: # /home/kohsuke/ws/hudson/investigations/ame/hs_err_pid24412.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # Current thread is 140588300966160 Dumping core ...
Given that AbstractMethodError is supposed to be raised by the invokevirtual instruction, I think this is a violation of the JVM spec.
Now to the work around.
I first thought that the problem was that HotSpot was optimizing it too much. I thought it was seeing that the left hand side of the foo() call was always 'this', and so maybe it's replacing the method body with "throw new AbstractMethodError()".
But with a bit more experiment, I think there's clearly more than that going on, as the following definition fails, too:
public void poll() throws Exception { try { System.out.println("Calling a possible abstract method"); // long and convoluted way of getting 'this', in a way HotSpot shouldn't see. SCM s = (SCM)SCM.class.getMethod("idem",Object.class).invoke(null,this); s.foo(); System.out.println("Done"); } catch (AbstractMethodError x) { System.out.println("Caught AME"); bar(); System.out.println("Resuming"); } } public static Object idem(Object o) { return o; }
The easiest work around thus far is to add one more pointless indirection like this:
public void poll() throws Exception { try { System.out.println("Calling a possible abstract method"); _foo(); System.out.println("Done"); } catch (AbstractMethodError x) { System.out.println("Caught AME"); bar(); System.out.println("Resuming"); } } private void _foo() { foo(); }
Code changed in hudson
User: : kohsuke
Path:
trunk/hudson/main/core/src/main/java/hudson/scm/SCM.java
trunk/www/changelog.html
http://jenkins-ci.org/commit/28424
Log:
[FIXED JENKINS-5756] in 1.350 by introducing a pointless function to work around what appears to be a HotSpot problem.
A safer fix might be to avoid catching AME to begin with by using Class.getDeclaredMethod to check whether the implementor defines a given method or not. Cleaner still would have been to not introduce new abstract methods into existing classes, but rather to define SCM2 extends SCM with the new methods (which would also permit old SCM impls to continue to compile against a newer hudson-core.jar).
Code changed in hudson
User: : abayer
Path:
trunk/hudson/main/core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/28470
Log:
[JENKINS-5827, JENKINS-5756] This approach fixes both bugs in one fell swoop
Code changed in hudson
User: : abayer
Path:
trunk/hudson/main/core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/28471
Log:
[JENKINS-5827, JENKINS-5756] Whoops - missed something I had intended.
Code changed in hudson
User: : kohsuke
Path:
trunk/hudson/main/core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/28474
Log:
Resurrecting the original fix for JENKINS-5756, as I'm still worried that bugparade 6933067 is related to some sort of the HotSpot optimization.
Code changed in hudson
User: : kohsuke
Path:
branches/rc/core/src/main/java/hudson/model/AbstractProject.java
branches/rc/core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/28499
Log:
Do the same protection to _calcRevisionsFromBuild to avoid JENKINS-5756
Cannot reproduce using your test program in 5u22, 6u37, or 7u9 on Linux—nor in 6u18, your originally reported platform. So are these workarounds still needed?
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/ClassicPluginStrategy.java
core/src/main/java/hudson/ExtensionFinder.java
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/model/AbstractItem.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/queue/Executables.java
core/src/main/java/hudson/model/queue/Tasks.java
core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/jenkins/253943e13ef16a9e14b71ef07a30e51e6a66e38f
Log:
JENKINS-5756 Removing workarounds for HotSpot bug.
https://bugs.openjdk.java.net/browse/JDK-6933067 claims to be fixed as of Java 6, our baseline.
Integrated in jenkins_main_trunk #3703
JENKINS-5756 Removing workarounds for HotSpot bug. (Revision 253943e13ef16a9e14b71ef07a30e51e6a66e38f)
Result = SUCCESS
Jesse Glick : 253943e13ef16a9e14b71ef07a30e51e6a66e38f
Files :
- core/src/main/java/hudson/model/View.java
- core/src/main/java/hudson/model/AbstractProject.java
- core/src/main/java/hudson/model/queue/Executables.java
- core/src/main/java/hudson/FilePath.java
- core/src/main/java/hudson/scm/SCM.java
- core/src/main/java/hudson/ExtensionFinder.java
- core/src/main/java/hudson/model/queue/Tasks.java
- core/src/main/java/hudson/model/AbstractItem.java
- core/src/main/java/hudson/ClassicPluginStrategy.java
Code changed in jenkins
User: Oliver Gondža
Path:
core/src/main/java/hudson/ClassicPluginStrategy.java
core/src/main/java/hudson/ExtensionFinder.java
core/src/main/java/hudson/model/AbstractItem.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/queue/Executables.java
core/src/main/java/hudson/model/queue/Tasks.java
core/src/main/java/hudson/scm/SCM.java
http://jenkins-ci.org/commit/jenkins/04880780d72bb6e03bd9b788d3019661ac462bfe
Log:
JENKINS-5756 Removing workarounds for HotSpot bug.
https://bugs.openjdk.java.net/browse/JDK-6933067 claims to be fixed as of Java 6, our baseline.
(cherry picked from commit 253943e13ef16a9e14b71ef07a30e51e6a66e38f)
Conflicts:
core/src/main/java/hudson/FilePath.java
I'm having the same problem with hudson 1.348 (I'm not sure if I had previously installed versione 1.347) and mercurial plugin 1.25