• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Jenkins 2.261
    • Jenkins 2.262

      With the newest update of jenkins 2.261, my builds started to never finish  .

      Pipelines builds or classical builds all stay stucked indefinitely on the warning analysis step.

       

      Classical build(when arriving to the post build steps):

      // code placeholder
      05:45:54 [Cobertura] Setting Build to unstable.
      05:45:54 [Cobertura] Setting Build to unstable.
      05:45:54 
      08:13:44 ERROR: Build step failed with exception
      08:13:44 java.lang.InterruptedException
      08:13:44  at java.lang.Object.wait(Native Method)
      08:13:44  at hudson.remoting.Request.call(Request.java:176)
      08:13:44  at hudson.remoting.Channel.call(Channel.java:1000)
      08:13:44  at hudson.FilePath.act(FilePath.java:1070)
      08:13:44  at hudson.FilePath.act(FilePath.java:1059)
      08:13:44  at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scanInWorkspace(ReportScanningTool.java:149)
      08:13:44 Caused: edu.hm.hafner.analysis.ParsingCanceledException: Canceling parsing since build has been aborted.
      08:13:44  at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scanInWorkspace(ReportScanningTool.java:159)
      08:13:44  at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scan(ReportScanningTool.java:131)
      08:13:44  at io.jenkins.plugins.analysis.core.steps.IssuesScanner.scan(IssuesScanner.java:101)
      08:13:44  at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.scanWithTool(IssuesRecorder.java:681)
      08:13:44  at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.record(IssuesRecorder.java:646)
      08:13:44  at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:617)
      08:13:44  at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:595)
      08:13:44  at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      08:13:44  at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
      08:13:44  at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
      08:13:44  at hudson.model.Build$BuildExecution.post2(Build.java:177)
      08:13:44  at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
      08:13:44  at hudson.model.Run.execute(Run.java:1932)
      08:13:44  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      08:13:44  at hudson.model.ResourceController.execute(ResourceController.java:97)
      08:13:44  at hudson.model.Executor.run(Executor.java:428)
      08:13:44 Build step 'Record compiler warnings and static analysis results' marked build as failure
      

       

      And a pipeline build:

      [Pipeline] recordIssuesAborted by 
       C.dric
      [Pipeline] }[Pipeline] // stage[Pipeline] }[Pipeline] // node[Pipeline] }[Pipeline] // timestamps[Pipeline] End of Pipeline[Checks API] No suitable checks publisher found.
       java.lang.InterruptedException
       at java.lang.Object.wait(Native Method)
       at hudson.remoting.Request.call(Request.java:176)
       at hudson.remoting.Channel.call(Channel.java:1000)
       at hudson.FilePath.act(FilePath.java:1070)
       at hudson.FilePath.act(FilePath.java:1059)
       at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scanInWorkspace(ReportScanningTool.java:149)
       Caused: edu.hm.hafner.analysis.ParsingCanceledException: Canceling parsing since build has been aborted.
       at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scanInWorkspace(ReportScanningTool.java:159)
       at io.jenkins.plugins.analysis.core.model.ReportScanningTool.scan(ReportScanningTool.java:131)
       at io.jenkins.plugins.analysis.core.steps.IssuesScanner.scan(IssuesScanner.java:101)
       at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.scanWithTool(IssuesRecorder.java:681)
       at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.record(IssuesRecorder.java:646)
       at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:617)
       at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:1024)
       at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:985)
       at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Finished: ABORTED
      

      I had to abot those builds manually after a few hours for the first and about 30 minutes for the second. Of course, those parts were done previously very fast.

       

      Downgrading to jenkins 2.260 fixes the problem.

          [JENKINS-63933] Jobs got stuck during execution on agents

          We are having the same issue.

          Joseph Witthuhn added a comment - We are having the same issue.

          Tim Jacomb added a comment - - edited

          can someone attach a threaddump while it's hung?

          probably from both the controller (aka master) and agent

          Tim Jacomb added a comment - - edited can someone attach a threaddump while it's hung? probably from both the controller (aka master) and agent

          Oleg Nenashev added a comment -

          Yes, it is impossible to diagnose it without the actual hanging execution. I am trying to reproduce it locally at the moment

          Oleg Nenashev added a comment - Yes, it is impossible to diagnose it without the actual hanging execution. I am trying to reproduce it locally at the moment

          Andre Crespo added a comment -

          I have problems just on Maven Projects. After git fetch completes, I see a line "Parsing POMs" that stucks on this position.

          Andre Crespo added a comment - I have problems just on Maven Projects. After git fetch completes, I see a line "Parsing POMs" that stucks on this position.

          Oleg Nenashev added a comment - - edited

          I was able to reproduce it. It happens due to JENKINS-63937 as far as I can tell, at least on Java 11

          Oct 15, 2020 3:54:01 PM hudson.remoting.AbstractByteBufferCommandTransport processCommand WARNING: Failed to construct Command in channel agent java.lang.ClassNotFoundException: jenkins.util.java.ClassNotFoundNoStackTraceException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:686) at hudson.remoting.ObjectInputStreamEx.resolveClass(ObjectInputStreamEx.java:63) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at hudson.remoting.Command.readFromObjectStream(Command.java:155) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:202) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189) at hudson.remoting.Engine$1AgentEndpoint.onMessage(Engine.java:583) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.notifyMessageHandlers(TyrusSession.java:576) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusEndpointWrapper.onPartialMessage(TyrusEndpointWrapper.java:1072) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusWebSocket.onFragment(TyrusWebSocket.java:171) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.frame.BinaryFrame.respond(BinaryFrame.java:91) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.ProtocolHandler.process(ProtocolHandler.java:807) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.TyrusClientEngine$TyrusReadHandler.handle(TyrusClientEngine.java:747) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter.processRead(ClientFilter.java:226) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:299) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter._read(TransportFilter.java:283) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter.access$100(TransportFilter.java:78) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:301) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          

          Oleg Nenashev added a comment - - edited I was able to reproduce it. It happens due to  JENKINS-63937  as far as I can tell, at least on Java 11 Oct 15, 2020 3:54:01 PM hudson.remoting.AbstractByteBufferCommandTransport processCommand WARNING: Failed to construct Command in channel agent java.lang.ClassNotFoundException: jenkins.util.java.ClassNotFoundNoStackTraceException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang. ClassLoader .loadClass( ClassLoader .java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang. ClassLoader .loadClass( ClassLoader .java:357) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:348) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:686) at hudson.remoting.ObjectInputStreamEx.resolveClass(ObjectInputStreamEx.java:63) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at hudson.remoting.Command.readFromObjectStream(Command.java:155) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:202) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189) at hudson.remoting.Engine$1AgentEndpoint.onMessage(Engine.java:583) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.notifyMessageHandlers(TyrusSession.java:576) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusEndpointWrapper.onPartialMessage(TyrusEndpointWrapper.java:1072) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusWebSocket.onFragment(TyrusWebSocket.java:171) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.frame.BinaryFrame.respond(BinaryFrame.java:91) at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.ProtocolHandler.process(ProtocolHandler.java:807) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.TyrusClientEngine$TyrusReadHandler.handle(TyrusClientEngine.java:747) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter.processRead(ClientFilter.java:226) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:299) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter._read(TransportFilter.java:283) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter.access$100(TransportFilter.java:78) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:301) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)

          We are seeing this issue on Java 8.

          Joseph Witthuhn added a comment - We are seeing this issue on Java 8.

          Oleg Nenashev added a comment -

          I confirm the regression is caused by JENKINS-63937, https://github.com/jenkinsci/jenkins/pull/4999 resolves this issue as well. I will continue analysis, but for  now I suggest reverting the patch and doing out of order release

          Oleg Nenashev added a comment - I confirm the regression is caused by  JENKINS-63937 , https://github.com/jenkinsci/jenkins/pull/4999  resolves this issue as well. I will continue analysis, but for  now I suggest reverting the patch and doing out of order release

          Oleg Nenashev added a comment -

          New weekly release is in progress. It will take a few hous until the release is done

          Oleg Nenashev added a comment - New weekly release is in progress. It will take a few hous until the release is done

          Joseph Witthuhn added a comment - - edited

          Do we believe that this is fixed in 2.262, which includes that pull request? If so, will we have to upgrade our agent JAR files as well to implement this fix?

          Joseph Witthuhn added a comment - - edited Do we believe that this is fixed in 2.262, which includes that pull request? If so, will we have to upgrade our agent JAR files as well to implement this fix?

          Oleg Nenashev added a comment -

          Yes, it should be fixed. No agent upgrade required

          Oleg Nenashev added a comment - Yes, it should be fixed. No agent upgrade required

            oleg_nenashev Oleg Nenashev
            cdric cdric julien
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: