Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56474

agent stuck in infinite loop at connect time on FreeBSD/arm with openjdk 8

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • remoting

      Starting an agent (both via ssh and from the command line) on a FreeBSD-12/arm64 agent sends the slave in an infinite loop for over five minutes. If started over ssh, jenkins will try launching a second (and third and so on) copy of the agent to the same result.

      For some reason it did connect once, but I can't reproduce the correct behaviour. I've tried deleting and recreating the host, no changes.

      Attaching stdout, stderr, remoting log and two thread dumps taken a few minutes after startup.

      It would seem that the slave is looping forever in hudson.remoting.ClassFilter$RegExpClassFilter.add - maybe the fact that it's running on an interpreter-mode JVM makes it super inefficient at parsing regexes? Either way, it's been running at 100% CPU for over 15 minutes now and Jenkins still doesn't see the connection as up 

        1. err
          2 kB
        2. out
          0.1 kB
        3. remoting.log.0
          1 kB
        4. tdump.1
          13 kB
        5. tdump.2
          13 kB
        6. tdump.3
          13 kB

          [JENKINS-56474] agent stuck in infinite loop at connect time on FreeBSD/arm with openjdk 8

          Jeff Thompson added a comment -

          I'm not too surprised that class loading, filtering, and pattern matching would be slow on an interpreted-only JVM. Those can be some intensive operations. Commonly the agent and the operations it needs to perform can be resource intensive. If you're running into that much resource issues just getting it up and going, it may be difficult for it perform the builds, particularly if other plugins are configured for the job.

          I think your best approach is to figure out how to speed up your processes. If you can get away from an interpreted-only JVM, that would be a good step. Otherwise you may try throwing hardware or other resources at to get it to run faster.

          Jeff Thompson added a comment - I'm not too surprised that class loading, filtering, and pattern matching would be slow on an interpreted-only JVM. Those can be some intensive operations. Commonly the agent and the operations it needs to perform can be resource intensive. If you're running into that much resource issues just getting it up and going, it may be difficult for it perform the builds, particularly if other plugins are configured for the job. I think your best approach is to figure out how to speed up your processes. If you can get away from an interpreted-only JVM, that would be a good step. Otherwise you may try throwing hardware or other resources at to get it to run faster.

          Jeff Thompson added a comment -

          Since there has been a lack of response for quite a while on this, I'm going to close this issue. If you have additional information to help move this along, feel free to provide it and re-open as needed.

          Jeff Thompson added a comment - Since there has been a lack of response for quite a while on this, I'm going to close this issue. If you have additional information to help move this along, feel free to provide it and re-open as needed.

          I believe I saw the same issue here.  Jenkins is running on a CentOS 7 box and I'm trying to configure a FreeBSD 13-CURRENT aarch64 as a builder node.  Using the default openjdk package (version 8.232.09) I got the same problem.  I decided then to go ahead and upgrade it to the latest version available on FreeBSD ports tree, which is 8.242.07.1.

          After that, it takes like 5 minutes to start, but after that it works.  It throws these messages on console:

           

          <===[JENKINS REMOTING CAPACITY]===>

          channel started

          <===[JENKINS REMOTING CAPACITY]===>

          channel started

          Remoting version: 3.36.1This is a Unix agentFeb 12, 2020 11:40:39 AM hudson.remoting.UserRequest performWARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@4c81d622java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/freebsd-aarch64/libjnidispatch.so) not found in resource path ([]) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          Agent successfully connected and online

           

          Renato Botelho added a comment - I believe I saw the same issue here.  Jenkins is running on a CentOS 7 box and I'm trying to configure a FreeBSD 13-CURRENT aarch64 as a builder node.  Using the default openjdk package (version 8.232.09) I got the same problem.  I decided then to go ahead and upgrade it to the latest version available on FreeBSD ports tree, which is 8.242.07.1. After that, it takes like 5 minutes to start, but after that it works.  It throws these messages on console:   <=== [JENKINS REMOTING CAPACITY] ===> channel started <=== [JENKINS REMOTING CAPACITY] ===> channel started Remoting version: 3.36.1This is a Unix agentFeb 12, 2020 11:40:39 AM hudson.remoting.UserRequest performWARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@4c81d622java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/freebsd-aarch64/libjnidispatch.so) not found in resource path ([]) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Agent successfully connected and online  

            jthompson Jeff Thompson
            kinkie kinkie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: