Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71227

Agent on AIX stops working with Jenkins 2.387.3

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • core, ssh-agent-plugin
    • None
    • AIX 7.2, JDK Semeru 11.0.17

      After upgrading Jenkins 2.387.2 to 2.387.3 the SSH agent on an AIX 7.2 machine sort of stops working. It is still connected, but unable to do any file operations. Starting a FreeStyle job fails. Even listing a workspace content is impossible.

      The log of the agent contains the following entries:

      Remoting version: 3107.v665000b_51092
      Launcher: SSHLauncher
      Communication Protocol: Standard in/out
      This is a Unix agent
      WARNING: An illegal reflective access operation has occurred
      WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
      WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
      WARNING: All illegal access operations will be denied in a future release
      Evacuated stdout
      May 08, 2023 2:20:02 PM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:hudson.FilePath$Exists@33285262
      java.lang.ExceptionInInitializerError
          at java.base/java.lang.J9VMInternals.ensureError(J9VMInternals.java:206)
          at java.base/java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:195)
          at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
          at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1445)
          at java.base/jdk.internal.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
          at java.base/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:186)
          at java.base/java.lang.reflect.Field.acquireFieldAccessor(Field.java:1105)
          at java.base/java.lang.reflect.Field.getFieldAccessor(Field.java:1086)
          at java.base/java.lang.reflect.Field.getLong(Field.java:630)
          at java.base/java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1637)
          at java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:367)
          at java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:355)
          at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
          at java.base/java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:355)
          at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:98)
          at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:95)
          at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:73)
          at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:70)
          at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:234)
          at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:216)
          at java.base/java.lang.ClassValue.get(ClassValue.java:122)
          at java.base/java.io.ClassCache.get(ClassCache.java:84)
          at java.base/java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:336)
          at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:542)
          at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2141)
          at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1991)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2322)
          at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1808)
          at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2624)
          at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2516)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2349)
          at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1808)
          at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:573)
          at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:483)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:289)
          at hudson.remoting.UserRequest.perform(UserRequest.java:189)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:377)
          at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:839)
      Caused by: sun.nio.fs.UnixException: A file, file system or message queue is no longer available.

      Trying to start a job on the agent results in the following stack traces in the job's log:

      00:35:24 Building remotely on aix[...] (OSSL30-AIX7_64 AIX7_64) in workspace /[...]
      00:35:24 sun.nio.fs.UnixException: No such file or directory
      00:35:24 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to aix[...]
      00:35:24 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
      00:35:24 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      00:35:24 		at hudson.remoting.Channel.call(Channel.java:1000)
      00:35:24 		at hudson.FilePath.act(FilePath.java:1192)
      00:35:24 		at hudson.FilePath.act(FilePath.java:1181)
      00:35:24 		at hudson.FilePath.mkdirs(FilePath.java:1372)
      00:35:24 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1234)
      00:35:24 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
      00:35:24 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
      00:35:24 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
      00:35:24 		at hudson.model.Run.execute(Run.java:1900)
      00:35:24 		at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
      00:35:24 		at hudson.model.ResourceController.execute(ResourceController.java:101)
      00:35:24 		at hudson.model.Executor.run(Executor.java:442)
      00:35:24 Caused: java.lang.NoClassDefFoundError: hudson.FilePath (initialization failure)
      00:35:24 	at java.base@11.0.17/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:156)
      00:35:24 	at java.base@11.0.17/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
      00:35:24 	at java.base@11.0.17/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1445)
      00:35:24 	at java.base@11.0.17/jdk.internal.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
      00:35:24 	at java.base@11.0.17/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:186)
      00:35:24 	at java.base@11.0.17/java.lang.reflect.Field.acquireFieldAccessor(Field.java:1105)
      00:35:24 	at java.base@11.0.17/java.lang.reflect.Field.getFieldAccessor(Field.java:1086)
      00:35:24 	at java.base@11.0.17/java.lang.reflect.Field.getLong(Field.java:630)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1637)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:367)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:355)
      00:35:24 	at java.base@11.0.17/java.security.AccessController.doPrivileged(AccessController.java:691)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:355)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:98)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:95)
      00:35:24 	at java.base@11.0.17/java.io.ClassCache$1.computeValue(ClassCache.java:73)
      00:35:24 	at java.base@11.0.17/java.io.ClassCache$1.computeValue(ClassCache.java:70)
      00:35:24 	at java.base@11.0.17/java.lang.ClassValue.getFromHashMap(ClassValue.java:234)
      00:35:24 	at java.base@11.0.17/java.lang.ClassValue.getFromBackup(ClassValue.java:216)
      00:35:24 	at java.base@11.0.17/java.lang.ClassValue.get(ClassValue.java:122)
      00:35:24 	at java.base@11.0.17/java.io.ClassCache.get(ClassCache.java:84)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:336)
      00:35:24 	at java.base@11.0.17/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:542)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2141)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1991)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2322)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1808)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2624)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2516)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2349)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1808)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readObject(ObjectInputStream.java:573)
      00:35:24 	at java.base@11.0.17/java.io.ObjectInputStream.readObject(ObjectInputStream.java:483)
      00:35:24 	at app//hudson.remoting.UserRequest.deserialize(UserRequest.java:289)
      00:35:24 	at app//hudson.remoting.UserRequest.perform(UserRequest.java:189)
      00:35:24 	at app//hudson.remoting.UserRequest.perform(UserRequest.java:54)
      00:35:24 	at app//hudson.remoting.Request$2.run(Request.java:377)
      00:35:24 	at app//hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
      00:35:24 	at java.base@11.0.17/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      00:35:24 	at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      00:35:24 	at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      00:35:24 	at java.base@11.0.17/java.lang.Thread.run(Thread.java:839)
      00:35:24 Caused: java.io.IOException: Remote call on aix[...] failed
      00:35:24 	at hudson.remoting.Channel.call(Channel.java:1004)
      00:35:24 	at hudson.FilePath.act(FilePath.java:1192)
      00:35:24 	at hudson.FilePath.act(FilePath.java:1181)
      00:35:24 	at hudson.FilePath.mkdirs(FilePath.java:1372)
      00:35:24 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1234)
      00:35:24 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
      00:35:24 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
      00:35:24 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
      00:35:24 	at hudson.model.Run.execute(Run.java:1900)
      00:35:24 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
      00:35:24 	at hudson.model.ResourceController.execute(ResourceController.java:101)
      00:35:24 	at hudson.model.Executor.run(Executor.java:442)

      After returning to Jenkins 2.387.2 everything is fine again. The plugins are all up-to-date.

      : The illegal reflective access warning still appears, though:

      Remoting version: 3107.v665000b_51092
      Launcher: SSHLauncher
      Communication Protocol: Standard in/out
      This is a Unix agent
      WARNING: An illegal reflective access operation has occurred
      WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
      WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
      WARNING: All illegal access operations will be denied in a future release
      Evacuated stdout
      ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
      java.lang.NoSuchMethodException: java.lang.Class.getDeclaredFields0(boolean)
      	at java.base@11.0.17/java.lang.Class.newNoSuchMethodException(Class.java:649)
      	at java.base@11.0.17/java.lang.Class.throwExceptionOrReturnNull(Class.java:1447)
      	at java.base@11.0.17/java.lang.Class.getMethodHelper(Class.java:1533)
      	at java.base@11.0.17/java.lang.Class.getDeclaredMethod(Class.java:1121)
      	at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.getModifiers(EnvInjectMasterEnvVarsSetter.java:28)
      	at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:55)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to aix[...]
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      		at hudson.remoting.Channel.call(Channel.java:1000)
      		at hudson.FilePath.act(FilePath.java:1283)
      		at org.jenkinsci.plugins.envinject.EnvInjectComputerListener.onOnline(EnvInjectComputerListener.java:144)
      		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)
      		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:438)
      		at hudson.plugins.sshslaves.SSHLauncher.startAgent(SSHLauncher.java:641)
      		at hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:458)
      		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      		at java.base/java.lang.Thread.run(Thread.java:829)
      Caused: java.lang.RuntimeException
      	at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:62)
      	at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:19)
      	at app//hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at app//hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at app//hudson.remoting.Request$2.run(Request.java:377)
      	at app//hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
      	at java.base@11.0.17/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base@11.0.17/java.lang.Thread.run(Thread.java:839)
      Agent successfully connected and online

      Please advise, thank you. – Matthias

       

            Unassigned Unassigned
            makr Matthias Kraft
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: