Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49038

Jenkins does not start due to a deadlock

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • claim-plugin, script-security-plugin
    • None
    • Windows Server 2012 R2
      Jenkins 2.102
      Oracle Java HotSpot(TM) 64-Bit Server VM version 25.144-b01

    Description

      Recent changes on the affected server:
      Jenkins-Update to 2.102, including all installed plugins
      Update for WIndows 2012 R2

      Jenkins refuses to start due to a thread deadlock during the startup process. This deadlock also affects HTTP request workers; so the server does not respond to HTTP requests anymore.

      The stacks below show the deadlock between thread pool-6-thread-16 (a.k.a. GitSCM.onLoaded) and pool-6-thread-11. The last stack shows how RequestHandlerThread1 is blocked by the lock owned by GitSCM.onLoaded.

      ------------------------------------------------------------------------------
      pool-6-thread-16

      Name: GitSCM.onLoaded
      State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@29986f0 owned by: PreventRefreshFilter.initAutoRefreshFilter
      Total blocked: 91 Total waited: 39

      Stack trace:
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:160)
      com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:136)
      hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
      com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
      com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
      com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
      com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
      hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:386)
      hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:377)
      hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:482)
      hudson.ExtensionList.load(ExtensionList.java:366)
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:304)

      • locked hudson.ExtensionList$Lock@5cfdc450
        hudson.ExtensionList.iterator(ExtensionList.java:158)
        jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1519)
        hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:1859)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      pool-6-thread-11

      Name: PreventRefreshFilter.initAutoRefreshFilter
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 10 Total waited: 41

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      hudson.ExtensionList.get(ExtensionList.java:149)
      org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval.get(ScriptApproval.java:97)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.configuring(SecureGroovyScript.java:126)
      org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.readResolve(SecureGroovyScript.java:102)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      hudson.util.XStream2.unmarshal(XStream2.java:147)
      hudson.util.XStream2.unmarshal(XStream2.java:118)
      com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      hudson.XmlFile.unmarshal(XmlFile.java:179)
      hudson.XmlFile.unmarshal(XmlFile.java:162)
      hudson.model.Descriptor.load(Descriptor.java:893)

      • locked hudson.plugins.claim.ClaimConfig@1d46c412
        hudson.plugins.claim.ClaimConfig.<init>(ClaimConfig.java:24)
        hudson.plugins.claim.ClaimConfig$$FastClassByGuice$$74f9760b.newInstance(<generated>)
        com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
        hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:424)
        com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        jenkins.ProxyInjector.getInstance(ProxyInjector.java:98)
        hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:124)
        hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:102)
        hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
        jenkins.model.Jenkins$5.runTask(Jenkins.java:1063)
        org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
        org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

      ------------------------------------------------------------------------------
      RequestHandlerThread1

      Name: Handling GET / from 10.10.15.31 : RequestHandlerThread1 HudsonIsLoading/index.jelly
      State: BLOCKED on hudson.ExtensionList$Lock@5cfdc450 owned by: GitSCM.onLoaded
      Total blocked: 1 Total waited: 0

      Stack trace:
      hudson.ExtensionList.ensureLoaded(ExtensionList.java:303)
      hudson.ExtensionList.iterator(ExtensionList.java:158)
      jenkins.JenkinsHttpSessionListener.sessionCreated(JenkinsHttpSessionListener.java:50)
      org.eclipse.jetty.server.session.SessionHandler.newHttpSession(SessionHandler.java:810)
      org.eclipse.jetty.server.Request.getSession(Request.java:1550)
      org.eclipse.jetty.server.Request.getSession(Request.java:1523)
      javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:279)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
      org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
      org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:169)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      org.eclipse.jetty.server.Server.handle(Server.java:564)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
      org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
      org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
      org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
      org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
      org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
      org.eclipse.jetty.io.ManagedSelector$$Lambda$16/549293029.run(Unknown Source)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-55361 DescriptorExtensionList not locking correctly, leading to deadlocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-54974 Jenkins does not start due to a deadlock after upgrade from 2.121.2.2 to 2.138.2.2

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            flybeetlecricket Fly Cricket added a comment - - edited

            As it turns out the deadlock does not occur when the Script Security Plugin plugin is being disabled.

            This can be done by creating an empty file `plugins/script-security.jpi.disabled`.

            flybeetlecricket Fly Cricket added a comment - - edited As it turns out the deadlock does not occur when the Script Security Plugin plugin is being disabled. This can be done by creating an empty file `plugins/script-security.jpi.disabled`.
            flybeetlecricket Fly Cricket added a comment -

            Further analysis shows that the deadlock can also be avoided by disabling the claim plugin

            flybeetlecricket Fly Cricket added a comment - Further analysis shows that the deadlock can also be avoided by disabling the  claim plugin
            oleg_nenashev Oleg Nenashev added a comment -

            It seems that calling "configuring()" in injector is generally risky. CC jglick who may point to the root cause or guidelines

            oleg_nenashev Oleg Nenashev added a comment - It seems that calling "configuring()" in injector is generally risky. CC jglick who may point to the root cause or guidelines
            jglick Jesse Glick added a comment -

            Guice madness, no clue. ClaimConfig$$FastClassByGuice$$74f9760b.newInstance

            jglick Jesse Glick added a comment - Guice madness, no clue. ClaimConfig$$FastClassByGuice$$74f9760b.newInstance
            flybeetlecricket Fly Cricket added a comment -

            Going back from Claim Plugin version 2.13.1 to 2.12 seems to solve the problem.

            flybeetlecricket Fly Cricket added a comment - Going back from Claim Plugin version 2.13.1 to 2.12 seems to solve the problem.
            greybird Arnaud TAMAILLON added a comment -

            Probably because in the 2.13, I introduced a new injection:

            @Initializer(after = InitMilestone.PLUGINS_STARTED)
public static synchronized void initAutoRefreshFilter(ClaimConfig config) throws ServletException {

            I could not reproduce locally on my first attempts, can you tell me if the issue is occuring everytime, and if not, the frequence of occurence?

            greybird Arnaud TAMAILLON added a comment - Probably because in the 2.13, I introduced a new injection: @Initializer(after = InitMilestone.PLUGINS_STARTED) public static synchronized void initAutoRefreshFilter(ClaimConfig config) throws ServletException { I could not reproduce locally on my first attempts, can you tell me if the issue is occuring everytime, and if not, the frequence of occurence?
            flybeetlecricket Fly Cricket added a comment -

            The issue is occuring everytime.

            flybeetlecricket Fly Cricket added a comment - The issue is occuring everytime.
            greybird Arnaud TAMAILLON added a comment -

            flybeetlecricket, can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ?

            I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.

            greybird Arnaud TAMAILLON added a comment - flybeetlecricket , can you test https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fclaim-plugin/detail/PR-53/1/artifacts version please, to tell me if situation is better ? I'm still not managing to reproduce, but am under the impression that it is probably linked to the actual list of plugins and maybe threads used by the task reactor lib, as on my dev pc, the deadlock does not occur.
            flybeetlecricket Fly Cricket added a comment -

            I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.

             

            flybeetlecricket Fly Cricket added a comment - I've uploaded the HPI via the Jenkins GUI. Jenkins could be restarted without a deadlock. A second restart also worked.  
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Arnaud
            Path:
            src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java
            http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698
            Log:
            JENKINS-49038 Jenkins does not start due to a deadlock

            the idea is to get the ClaimConfig retrieval out of guice logic.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Arnaud Path: src/main/java/hudson/plugins/claim/http/PreventRefreshFilter.java http://jenkins-ci.org/commit/claim-plugin/16e22544affcff18b8e4b4ae01036ad3482de698 Log: JENKINS-49038 Jenkins does not start due to a deadlock the idea is to get the ClaimConfig retrieval out of guice logic.
            greybird Arnaud TAMAILLON added a comment -

            Thanks for your help!

            Fixed in 2.14.1

            greybird Arnaud TAMAILLON added a comment - Thanks for your help! Fixed in 2.14.1

            People

              greybird Arnaud TAMAILLON
              flybeetlecricket Fly Cricket
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: