Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29171

Abort after load all jobs in Solaris/Sparc

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      Jenkins 1.617,1.6091.517 tried on Java 7 and Java 8
      SunOS myserver 5.10 Generic_Virtual sun4v sparc sun4v
      Oracle Solaris 10 1/13 s10s_u11wos_24a SPARC
    • Similar Issues:

      Description

      Trying to start jenkins on Solaris/Sparc and it shows abort after the Loaded all jobs.

      java -jar jenkins.war --httpPort=1810 -Xdebug

      INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
      INFO: Started initialization
      INFO: Listed all plugins
      INFO: Prepared all plugins
      INFO: Started all plugins
      INFO: Augmented all extensions
      INFO: Loaded all jobs
      Abort

      This works fine on Linux but fails with both java 7 and 8 on Solaris/Sparc. I can not find work around on web. The thread dump does not produce any output.

        Attachments

          Activity

          Hide
          fred1912 Frederic B added a comment - - edited

          Hello,
          I have the exact same issue with jenkins 1.609.2, java 6 or 7 on Solaris/SPARC 10.

          Last thread in the pstack of the core dump :

          -----------------  lwp# 53 / thread# 53  --------------------
           fefcecb8 ___lwp_cond_wait (1389ac8, 1389ab0, 0, feb5934c, 55000, 0) + 8
           fe85a2a0 __1cGParkerEpark6Mbx_v_ (34358, feb38358, 63c000, feb04000, 1389ab0, 2) + 49c
           fe968714 Unsafe_Park (46444, 0, feb44f7c, feb481b4, 63c000, feb04000) + 280
           fbc0f960 * sun/misc/Unsafe.park(ZJ)V+0
           fbc0f90c * sun/misc/Unsafe.park(ZJ)V+0
           fbc06a84 * java/util/concurrent/locks/LockSupport.park(Ljava/lang/Object;)V+14 (line 370)
           fbc06a84 * java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await()V+42 (line 4080)
           fbc0761c * java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take()Ljava/util/concurrent/RunnableScheduledFuture;+98 (line 2190)
           fbc06748 * java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take()Ljava/lang/Object;+1 (line 1614)
           fbc072e0 * java/util/concurrent/ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+156 (line 2056)
           fbc06748 * java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17 (line 2202)
           fbc06a84 * java/util/concurrent/ThreadPoolExecutor$Worker.run()V+5 (line 1206)
           fbc0761c * java/lang/Thread.run()V+11 (line 1443)
           fbc0021c * StubRoutines (1)
           fe171744 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (b46ffb70, 63c000, b46ffab8, e, 4, b46ff910) + 300
           fe5a64a4 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_ (b46ffb70, b46ff9f0, 1758fd0, 992800, b46ffab8, ff66d444) + 158
           fe5a6510 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_6pnGThread__v_ (b46ffb70, b46ffb6c, b46ffb64, 57768, 57960, 63c000) + 5c
           fe222ca0 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (655600, 55734, b7826038, 1758fc4, feb59734, 55400) + 160
           fe940ffc __1cKJavaThreadRthread_main_inner6M_v_ (63c000, 35, 1157d28, 0, b46ffc00, 0) + 3c
           fe85065c java_start (63c000, fe21e3ec, feb04000, feb4a300, 32, 63c400) + 290
           fefcaee0 _lwp_start (0, 0, 0, 0, 0, 0)
          

          By launching jenkins activating various verbose option, I can see the loaded libraries before crash :
          java -Xcheck:jni -server -verbose:jni -verbose:class -jar jenkins.war

          [Loaded com.sun.jna.NativeMappedConverter from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar]
          [Dynamic-linking native method com.sun.jna.Native.invokePointer ... JNI]
          [Loaded org.codehaus.groovy.runtime.IteratorClosureAdapter from file:/users/jenkins/.jenkins/war/WEB-INF/lib/groovy-all-1.8.9.jar]
          [Loaded org.codehaus.groovy.runtime.metaclass.MissingPropertyExceptionNoStack from file:/users/jenkins/.jenkins/war/WEB-INF/lib/groovy-all-1.8.9.jar]
          [Loaded org.jvnet.solaris.libzfs.LibZFS$1 from file:/users/jenkins/.jenkins/war/WEB-INF/lib/libzfs-0.5.jar]
          [Loaded com.sun.jna.CallbackReference$DefaultCallbackProxy from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar]
          [Loaded com.sun.jna.CallbackResultContext from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar]
          [Loaded com.sun.jna.CallbackParameterContext from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar]
          [Loaded com.sun.jna.win32.DLLCallback from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar]
          [Dynamic-linking native method com.sun.jna.Native.createNativeCallback ... JNI]
          [Dynamic-linking native method com.sun.jna.Native._getPointer ... JNI]
          [Dynamic-linking native method com.sun.jna.Native.invokeInt ... JNI]
          

          It seems that jenkins tries to load a win32 dll for JNA library, maybe Solaris is not properly detected.

          Any help would be much appreciated as it is a blocker for our project and I cannot find anything to help on the web...

          Thanks.

          Show
          fred1912 Frederic B added a comment - - edited Hello, I have the exact same issue with jenkins 1.609.2, java 6 or 7 on Solaris/SPARC 10. Last thread in the pstack of the core dump : ----------------- lwp# 53 / thread# 53 -------------------- fefcecb8 ___lwp_cond_wait (1389ac8, 1389ab0, 0, feb5934c, 55000, 0) + 8 fe85a2a0 __1cGParkerEpark6Mbx_v_ (34358, feb38358, 63c000, feb04000, 1389ab0, 2) + 49c fe968714 Unsafe_Park (46444, 0, feb44f7c, feb481b4, 63c000, feb04000) + 280 fbc0f960 * sun/misc/Unsafe.park(ZJ)V+0 fbc0f90c * sun/misc/Unsafe.park(ZJ)V+0 fbc06a84 * java/util/concurrent/locks/LockSupport.park(Ljava/lang/ Object ;)V+14 (line 370) fbc06a84 * java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await()V+42 (line 4080) fbc0761c * java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take()Ljava/util/concurrent/RunnableScheduledFuture;+98 (line 2190) fbc06748 * java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take()Ljava/lang/ Object ;+1 (line 1614) fbc072e0 * java/util/concurrent/ThreadPoolExecutor.getTask()Ljava/lang/ Runnable ;+156 (line 2056) fbc06748 * java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17 (line 2202) fbc06a84 * java/util/concurrent/ThreadPoolExecutor$Worker.run()V+5 (line 1206) fbc0761c * java/lang/ Thread .run()V+11 (line 1443) fbc0021c * StubRoutines (1) fe171744 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (b46ffb70, 63c000, b46ffab8, e, 4, b46ff910) + 300 fe5a64a4 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_ (b46ffb70, b46ff9f0, 1758fd0, 992800, b46ffab8, ff66d444) + 158 fe5a6510 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_6pnGThread__v_ (b46ffb70, b46ffb6c, b46ffb64, 57768, 57960, 63c000) + 5c fe222ca0 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (655600, 55734, b7826038, 1758fc4, feb59734, 55400) + 160 fe940ffc __1cKJavaThreadRthread_main_inner6M_v_ (63c000, 35, 1157d28, 0, b46ffc00, 0) + 3c fe85065c java_start (63c000, fe21e3ec, feb04000, feb4a300, 32, 63c400) + 290 fefcaee0 _lwp_start (0, 0, 0, 0, 0, 0) By launching jenkins activating various verbose option, I can see the loaded libraries before crash : java -Xcheck:jni -server -verbose:jni -verbose:class -jar jenkins.war [Loaded com.sun.jna.NativeMappedConverter from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar] [Dynamic-linking native method com.sun.jna.Native.invokePointer ... JNI] [Loaded org.codehaus.groovy.runtime.IteratorClosureAdapter from file:/users/jenkins/.jenkins/war/WEB-INF/lib/groovy-all-1.8.9.jar] [Loaded org.codehaus.groovy.runtime.metaclass.MissingPropertyExceptionNoStack from file:/users/jenkins/.jenkins/war/WEB-INF/lib/groovy-all-1.8.9.jar] [Loaded org.jvnet.solaris.libzfs.LibZFS$1 from file:/users/jenkins/.jenkins/war/WEB-INF/lib/libzfs-0.5.jar] [Loaded com.sun.jna.CallbackReference$DefaultCallbackProxy from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar] [Loaded com.sun.jna.CallbackResultContext from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar] [Loaded com.sun.jna.CallbackParameterContext from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar] [Loaded com.sun.jna.win32.DLLCallback from file:/users/jenkins/.jenkins/war/WEB-INF/lib/jna-4.1.0.jar] [Dynamic-linking native method com.sun.jna.Native.createNativeCallback ... JNI] [Dynamic-linking native method com.sun.jna.Native._getPointer ... JNI] [Dynamic-linking native method com.sun.jna.Native.invokeInt ... JNI] It seems that jenkins tries to load a win32 dll for JNA library, maybe Solaris is not properly detected. Any help would be much appreciated as it is a blocker for our project and I cannot find anything to help on the web... Thanks.
          Hide
          info781 Steven Moslin added a comment -

          The only information I have is that I moved it to Solaris 11 (on sparc) and it works, so I think it is a Solaris 10 bug. Hope that helps someone.

          Steve

          Show
          info781 Steven Moslin added a comment - The only information I have is that I moved it to Solaris 11 (on sparc) and it works, so I think it is a Solaris 10 bug. Hope that helps someone. Steve
          Hide
          danielbeck Daniel Beck added a comment -

          Did older versions of Jenkins work properly on Solaris 10? E.g. 1.585? How about 1.586? The latter is when Jenkins upgraded JNA from 3.x to 4.x.

          Show
          danielbeck Daniel Beck added a comment - Did older versions of Jenkins work properly on Solaris 10? E.g. 1.585? How about 1.586? The latter is when Jenkins upgraded JNA from 3.x to 4.x.
          Hide
          fred1912 Frederic B added a comment -

          Hello,

          The issue was on all version of jenkins I've tested, from older version to newest ones.
          The solution has been found by contacting Oracle Solaris support, which I post here for anyone who has the same issue :

          I've noticed a pattern of failure in both core dumps, as follows:

          ff1829f0 abort (0, 1, ff3fa0c4, ffb04, ff285518, 0) + 110
          ff3e6b38 _brand_abort (0, ff3e8670, ff3e8630, 27b, ff3fac0c, ff3fa000) + 30
          ff3e3110 zfs_ioctl (b57ba870, 10, 5a04, b57ba8f0, ff3e8664, ff3fa000) + 5c

          In a solaris 10 branded zone, any call to a zfs ioctl causes the process to abort as the ZFS ioctls have changed in the solaris 11 kernel so cannot be called...

          This is coming from the following routine:

          fbc072e0 * $Proxy32.zfs_iter_root(Lorg/jvnet/solaris/libzfs/jna/libzfs_handle_t;Lorg/jvnet/solaris/libzfs/jna/libzfs$zfs_iter_f;Lcom/sun/jna/Pointer;)I+24
          fbc07384 * org/jvnet/solaris/libzfs/LibZFS.roots()Ljava/util/List;+25 (line 131)
          fbc06748 * hudson/os/solaris/ZFSInstaller.shouldBeActive()Z+30 (line 204)
          fbc0655c * hudson/os/solaris/ZFSInstaller.<init>()V+6 (line 152)
          fbc06a84 * hudson/os/solaris/ZFSInstaller.init()Lhudson/model/AdministrativeMonitor;+102 (line 598)

          I suggest you try starting the app with -Dhudson.os.solaris.ZFSInstaller.disabled=true as this might by-pass the code that causes the core dump. This comes from:
          https://wiki.eclipse.org/Using_Hudson/Features_controlled_by_system_properties

          By adding the option -Dhudson.os.solaris.ZFSInstaller.disabled=true, jenkins is launching properly.

          Show
          fred1912 Frederic B added a comment - Hello, The issue was on all version of jenkins I've tested, from older version to newest ones. The solution has been found by contacting Oracle Solaris support, which I post here for anyone who has the same issue : I've noticed a pattern of failure in both core dumps, as follows: ff1829f0 abort (0, 1, ff3fa0c4, ffb04, ff285518, 0) + 110 ff3e6b38 _brand_abort (0, ff3e8670, ff3e8630, 27b, ff3fac0c, ff3fa000) + 30 ff3e3110 zfs_ioctl (b57ba870, 10, 5a04, b57ba8f0, ff3e8664, ff3fa000) + 5c In a solaris 10 branded zone, any call to a zfs ioctl causes the process to abort as the ZFS ioctls have changed in the solaris 11 kernel so cannot be called... This is coming from the following routine: fbc072e0 * $Proxy32.zfs_iter_root(Lorg/jvnet/solaris/libzfs/jna/libzfs_handle_t;Lorg/jvnet/solaris/libzfs/jna/libzfs$zfs_iter_f;Lcom/sun/jna/Pointer;)I+24 fbc07384 * org/jvnet/solaris/libzfs/LibZFS.roots()Ljava/util/List;+25 (line 131) fbc06748 * hudson/os/solaris/ZFSInstaller.shouldBeActive()Z+30 (line 204) fbc0655c * hudson/os/solaris/ZFSInstaller.<init>()V+6 (line 152) fbc06a84 * hudson/os/solaris/ZFSInstaller.init()Lhudson/model/AdministrativeMonitor;+102 (line 598) I suggest you try starting the app with -Dhudson.os.solaris.ZFSInstaller.disabled=true as this might by-pass the code that causes the core dump. This comes from: https://wiki.eclipse.org/Using_Hudson/Features_controlled_by_system_properties By adding the option -Dhudson.os.solaris.ZFSInstaller.disabled=true, jenkins is launching properly.
          Hide
          info781 Steven Moslin added a comment -

          That worked for me on Solaris 10/Sparc as well. Frederic, thanks for the information, it is a big help.

          Show
          info781 Steven Moslin added a comment - That worked for me on Solaris 10/Sparc as well. Frederic, thanks for the information, it is a big help.

            People

            Assignee:
            info781 Steven Moslin
            Reporter:
            info781 Steven Moslin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: