Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53924

Jenkins 2.136+ is not restartable with GLIBC 2.5

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • core
    • None
    • Red Hat Enterprise Linux Server release 5.11 (Tikanga)
      java version "1.8.0_181"
      Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
      Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

      When Jenkins 2.136 (or newer) starts up, the log lists problem with loading libc library for com.sun.akuma.CLibrary:

      Oct 05, 2018 4:08:34 PM jenkins.diagnosis.HsErrPidList <init>
      WARNING: Failed to list up hs_err_pid files
      java.lang.UnsatisfiedLinkError: /tmp/jna-578977101/jna607378441395487448.tmp: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna-578977101/jna607378441395487448.tmp)
              at java.lang.ClassLoader$NativeLibrary.load(Native Method)
              at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
              at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
              at java.lang.Runtime.load0(Runtime.java:809)
              at java.lang.System.load(System.java:1086)
              at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947)
              at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922)
              at com.sun.jna.Native.<clinit>(Native.java:190)
              at com.sun.akuma.CLibrary.<clinit>(CLibrary.java:89)
              at com.sun.akuma.JavaVMArguments.resolvePID(JavaVMArguments.java:128)
              at com.sun.akuma.JavaVMArguments.ofLinux(JavaVMArguments.java:116)
              at com.sun.akuma.JavaVMArguments.of(JavaVMArguments.java:104)
              at com.sun.akuma.JavaVMArguments.current(JavaVMArguments.java:92)
      ...
      

      When visiting /safeRestart URL, there is no option to restart and instead there is a message saying: "Jenkins cannot restart itself as currently configured."

      Attempt to restart from command line fails with:

      $ java -jar jenkins-cli.jar -s <jenkins_url> -ssh -user <user> safe-restart
      
      ERROR: Unexpected exception occurred while performing safe-restart command.
      java.lang.NoClassDefFoundError: Could not initialize class com.sun.akuma.CLibrary
              at com.sun.akuma.JavaVMArguments.resolvePID(JavaVMArguments.java:128)
              at com.sun.akuma.JavaVMArguments.ofLinux(JavaVMArguments.java:116)
              at com.sun.akuma.JavaVMArguments.of(JavaVMArguments.java:104)
              at com.sun.akuma.JavaVMArguments.current(JavaVMArguments.java:92)
              at hudson.lifecycle.UnixLifecycle.<init>(UnixLifecycle.java:55)
              at hudson.lifecycle.Lifecycle.get(Lifecycle.java:94)
              at jenkins.model.Jenkins.getLifecycle(Jenkins.java:2538)
              at org.jenkins.ci.plugins.saferestart.JenkinsUtils.canRestart(JenkinsUtils.java:37)
              at org.jenkins.ci.plugins.saferestart.SafeRestartRootAction.getIconFileName(SafeRestartRootAction.java:47)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
              at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
              at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
              at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
              at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
              at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
              at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
              at org.apache.commons.jexl.parser.ASTTernaryNode.value(ASTTernaryNode.java:41)
              at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
              at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56)
              at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
              at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
              at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:121)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
              at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
              at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
              at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
              at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
              at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
              at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
              at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
              at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:801)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
              at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
              at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
              at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
              at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
              at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
              at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
              at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
              at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
              at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
              at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
              at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:531)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
              at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
              at winstone.BoundedExecutorService.lambda$scheduleNext$0(BoundedExecutorService.java:80)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      Caused: hudson.lifecycle.RestartNotSupportedException: Failed to obtain the command line arguments of the process
              at hudson.lifecycle.UnixLifecycle.verifyRestartable(UnixLifecycle.java:104)
              at jenkins.model.Jenkins.safeRestart(Jenkins.java:4216)
              at jenkins.model.Jenkins.doSafeRestart(Jenkins.java:4176)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at hudson.cli.declarative.MethodBinder.call(MethodBinder.java:114)
              at hudson.cli.declarative.CLIRegisterer$1.main(CLIRegisterer.java:225)
              at org.jenkinsci.main.modules.sshd.CLICommandAdapter$1.run(CLICommandAdapter.java:37)
              at org.jenkinsci.main.modules.sshd.AsynchronousCommand$1.run(AsynchronousCommand.java:112)
              at java.lang.Thread.run(Thread.java:748)
      

      This is a regression introduced in 2.136. JENKINS-52771 introduced new version of JNA library, which requires GLIBC 2.7 to start. I saw a similar issue JENKINS-10354 was fixed some time ago with recompilation of JNA library against older glibc fixed the issue. Would it be possible to fix it the same way now? RHEL5 extended support is up to 2020.

          [JENKINS-53924] Jenkins 2.136+ is not restartable with GLIBC 2.5

          Oleg Nenashev added a comment -

          Sames as JENKINS-53832, but for RHEL

          Oleg Nenashev added a comment - Sames as JENKINS-53832 , but for RHEL

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - Documentation patch: https://github.com/jenkins-infra/jenkins.io/pull/1815

          Gordon Daugherty added a comment - - edited

          As info, I had this same issue and found that the cause was that Linux's 'noexec' setting was enabled on the directory that was being used as my JVM's JNI tmpdir. The fix was to use "-Djna.tmpdir=<some path that has execute capability>" in the java command line used to start Jenkins.

           

          Specifically, I added this to my start script:

          #Java's JNA tmpdir cannot be 'noexec' or Jenkins' restart functionality won't work so we're relocating it.

          mkdir -p ${JENKINS_HOME}/tmp
          TMPDIR_ARG=-Djna.tmpdir=${JENKINS_HOME}/tmp

          nohup $JAVA_HOME/bin/java $TMPDIR_ARG -jar jenkins.war <various other arguments omitted as those will differ for you> &

          Gordon Daugherty added a comment - - edited As info, I had this same issue and found that the cause was that Linux's 'noexec' setting was enabled on the directory that was being used as my JVM's JNI tmpdir. The fix was to use "-Djna.tmpdir=<some path that has execute capability>" in the java command line used to start Jenkins.   Specifically, I added this to my start script: #Java's JNA tmpdir cannot be 'noexec' or Jenkins' restart functionality won't work so we're relocating it. mkdir -p ${JENKINS_HOME}/tmp TMPDIR_ARG=-Djna.tmpdir=${JENKINS_HOME}/tmp nohup $JAVA_HOME/bin/java $TMPDIR_ARG -jar jenkins.war <various other arguments omitted as those will differ for you> &

            Unassigned Unassigned
            raspy Krzysztof Malinowski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: