Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54026

Database maybe alredy in use "Server is running"(pipeline+maven)

XMLWordPrintable

       

      I have problem on Jenkins, when i use maven in pipeline script :

      sh "cd artifact-info-grabber && mvn clean package -s ../settings.xml -Dmaven.test.skip=true"

      But any time i have error:

       

      ERROR: [withMaven] pipelineGraphPublisher - WARNING: Exception recording ru.sbrf.bdp.devops.apivalidation:core:jar:1.0.0-SNAPSHOT on build, skiporg.jenkinsci.plugins.pipeline.maven.util.RuntimeSqlException: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Server is running". Possible solutions: close all other connection(s); use the server mode; SQL statement:10.116.241.122:44405/1665de891a9ce518d0a1383bd8942359493f04ca42f [90020-196]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.store.FileLock.checkServer(FileLock.java:257)
        at org.h2.store.FileLock.lock(FileLock.java:129)
        at org.h2.engine.Database.open(Database.java:662)
        at org.h2.engine.Database.openDatabase(Database.java:276)
        at org.h2.engine.Database.<init>(Database.java:270)
        at org.h2.engine.Engine.openSession(Engine.java:64)
        at org.h2.engine.Engine.openSession(Engine.java:176)
        at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154)
        at org.h2.engine.Engine.createSession(Engine.java:137)
        at org.h2.server.TcpServerThread.run(TcpServerThread.java:148)
        at java.lang.Thread.run(Thread.java:748) 
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.getOrCreateBuildPrimaryKey(PipelineMavenPluginH2Dao.java:424)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.recordDependency(PipelineMavenPluginH2Dao.java:114)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginMonitoringDao.recordDependency(PipelineMavenPluginMonitoringDao.java:33)
        at org.jenkinsci.plugins.pipeline.maven.publishers.PipelineGraphPublisher.recordDependencies(PipelineGraphPublisher.java:189)
        at org.jenkinsci.plugins.pipeline.maven.publishers.PipelineGraphPublisher.process(PipelineGraphPublisher.java:93)
        at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:118)
        at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution.java:1050)
        at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124)
        at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:349)
        at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
        at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Server is running". Possible solutions: close all other connection(s); use the server mode; SQL statement:10.116.241.122:44405/1665de891a9ce518d0a1383bd8942359493f04ca42f [90020-196]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.store.FileLock.checkServer(FileLock.java:257)
        at org.h2.store.FileLock.lock(FileLock.java:129)
        at org.h2.engine.Database.open(Database.java:662)
        at org.h2.engine.Database.openDatabase(Database.java:276)
        at org.h2.engine.Database.<init>(Database.java:270)
        at org.h2.engine.Engine.openSession(Engine.java:64)
        at org.h2.engine.Engine.openSession(Engine.java:176)
        at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154)
        at org.h2.engine.Engine.createSession(Engine.java:137)
        at org.h2.server.TcpServerThread.run(TcpServerThread.java:148)
        at java.lang.Thread.run(Thread.java:748) 
        at org.h2.engine.SessionRemote.done(SessionRemote.java:629)
        at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:144)
        at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:453)
        at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:367)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:116)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:100)
        at org.h2.Driver.connect(Driver.java:69)
        at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188)
        at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351)
        at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383)
        at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:226)
        at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:198)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.getOrCreateBuildPrimaryKey(PipelineMavenPluginH2Dao.java:369)
        ... 35 more
 ERROR: [withMaven] WARNING Exception executing Maven reporter 'Pipeline Graph Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.PipelineGraphPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org org.jenkinsci.plugins.pipeline.maven.util.RuntimeSqlException: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Server is running". Possible solutions: close all other connection(s); use the server mode; SQL statement:10.116.241.122:44405/1665de891a9ce518d0a1383bd8942359493f04ca42f [90020-196]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.store.FileLock.checkServer(FileLock.java:257)
        at org.h2.store.FileLock.lock(FileLock.java:129)
        at org.h2.engine.Database.open(Database.java:662)
        at org.h2.engine.Database.openDatabase(Database.java:276)
        at org.h2.engine.Database.<init>(Database.java:270)
        at org.h2.engine.Engine.openSession(Engine.java:64)
        at org.h2.engine.Engine.openSession(Engine.java:176)
        at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154)
        at org.h2.engine.Engine.createSession(Engine.java:137)
        at org.h2.server.TcpServerThread.run(TcpServerThread.java:148)
        at java.lang.Thread.run(Thread.java:748) 
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.getOrCreateBuildPrimaryKey(PipelineMavenPluginH2Dao.java:424)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.recordGeneratedArtifact(PipelineMavenPluginH2Dao.java:198)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginMonitoringDao.recordGeneratedArtifact(PipelineMavenPluginMonitoringDao.java:57)
        at org.jenkinsci.plugins.pipeline.maven.publishers.PipelineGraphPublisher.recordGeneratedArtifacts(PipelineGraphPublisher.java:229)
        at org.jenkinsci.plugins.pipeline.maven.publishers.PipelineGraphPublisher.process(PipelineGraphPublisher.java:94)
        at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:118)
        at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution.java:1050)
        at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124)
        at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:349)
        at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
        at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Server is running". Possible solutions: close all other connection(s); use the server mode; SQL statement:10.116.241.122:44405/1665de891a9ce518d0a1383bd8942359493f04ca42f [90020-196]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
        at org.h2.message.DbException.get(DbException.java:179)
        at org.h2.message.DbException.get(DbException.java:155)
        at org.h2.store.FileLock.checkServer(FileLock.java:257)
        at org.h2.store.FileLock.lock(FileLock.java:129)
        at org.h2.engine.Database.open(Database.java:662)
        at org.h2.engine.Database.openDatabase(Database.java:276)
        at org.h2.engine.Database.<init>(Database.java:270)
        at org.h2.engine.Engine.openSession(Engine.java:64)
        at org.h2.engine.Engine.openSession(Engine.java:176)
        at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154)
        at org.h2.engine.Engine.createSession(Engine.java:137)
        at org.h2.server.TcpServerThread.run(TcpServerThread.java:148)
        at java.lang.Thread.run(Thread.java:748) 
        at org.h2.engine.SessionRemote.done(SessionRemote.java:629)
        at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:144)
        at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:453)
        at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:367)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:116)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:100)
        at org.h2.Driver.connect(Driver.java:69)
        at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:188)
        at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:351)
        at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:383)
        at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:226)
        at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:198)
        at org.jenkinsci.plugins.pipeline.maven.dao.PipelineMavenPluginH2Dao.getOrCreateBuildPrimaryKey(PipelineMavenPluginH2Dao.java:369)
... 35 more

       

      maybe someone know how to solve the problem. Maybe can increase pool of connections? But i dont't know how.

       

        1. JENKINS-54026.png
          JENKINS-54026.png
          27 kB
        2. jenkins-error.jpg
          jenkins-error.jpg
          176 kB

          [JENKINS-54026] Database maybe alredy in use "Server is running"(pipeline+maven)

          Cyrille Le Clerc added a comment -

          Could you use NFS or a shared file system of any kind?

          Cyrille Le Clerc added a comment - Could you use NFS or a shared file system of any kind?

          Anatoliy Pavlov added a comment -

          I don't kmow, we work on openstack(vm), and in system use "local" disck,  but we work whith DRBD for backup jenkins(configuration and log).

           

           

          Anatoliy Pavlov added a comment - I don't kmow, we work on openstack(vm), and in system use "local" disck,  but we work whith DRBD for backup jenkins(configuration and log).    

          Cyrille Le Clerc added a comment -

          xed did you fix temporarily the problem? restarting the master?

          • How often do this problem happen?
          • Can you please share the database details (number of rows...) displayed in "manage jenkins / global tools configuration"
          • Is this a very loaded server? do you have many Maven pipelines running in parallel?
          • If the problem is happening once again, can you consider starting the H2 database as standalone process?
            • The H2 database is located in $JENKINS_HOME/jenkins-jobs/jenkins-jobs.mv.db, it can be started as a standalone process and then the connection parameters are entered in the "advanced" section of "manage jenkins / global tools configuration / pipeline maven configuration"

          Cyrille Le Clerc added a comment - xed did you fix temporarily the problem? restarting the master? How often do this problem happen? Can you please share the database details (number of rows...) displayed in "manage jenkins / global tools configuration" Is this a very loaded server? do you have many Maven pipelines running in parallel? If the problem is happening once again, can you consider starting the H2 database as standalone process? The H2 database is located in $JENKINS_HOME/jenkins-jobs/jenkins-jobs.mv.db, it can be started as a standalone process and then the connection parameters are entered in the "advanced" section of "manage jenkins / global tools configuration / pipeline maven configuration"

          Anatoliy Pavlov added a comment -

          The problem is temporary, if start job several times.

          • Verry offen, I could not find the sequence. But we have a problem with 1 of 10 launches.
          • No, now time "manage jenkins / global tools configuration / pipeline maven configuration", displayed  error "Server is running"
          • We have high load jenkins for ci. many development team parrallel use maven, pipelain, free-style jobs for build and testing deploy java-applications. More than 35k builds(with failed and aborted) per day.
          • ok, we will think about it

           

           

          Anatoliy Pavlov added a comment - The problem is temporary, if start job several times. Verry offen, I could not find the sequence. But we have a problem with 1 of 10 launches. No, now time "manage jenkins / global tools configuration / pipeline maven configuration", displayed  error "Server is running" We have high load jenkins for ci. many development team parrallel use maven, pipelain, free-style jobs for build and testing deploy java-applications. More than 35k builds(with failed and aborted) per day. ok, we will think about it    

          Cyrille Le Clerc added a comment -

          xed this problem seems to be caused by a kind of "file lock" (1).

          • Could you have a kind of "zombie process" that is keeping a lock on the h2 file? a Jenkins process not yet properly terminated? an antivirus?
          • Could it be an initialisation issue with a thread safety issue due to a big workload at startup?
          • Is it also happening some time after the master has successfully started?

          (1) https://github.com/h2database/h2database/blob/version-1.4.196/h2/src/main/org/h2/store/FileLock.java#L257

          Cyrille Le Clerc added a comment - xed this problem seems to be caused by a kind of "file lock" (1). Could you have a kind of "zombie process" that is keeping a lock on the h2 file? a Jenkins process not yet properly terminated? an antivirus? Could it be an initialisation issue with a thread safety issue due to a big workload at startup? Is it also happening some time after the master has successfully started? (1) https://github.com/h2database/h2database/blob/version-1.4.196/h2/src/main/org/h2/store/FileLock.java#L257

          Anatoliy Pavlov added a comment - - edited

          We check "zombie process", but don't found them on master jenkins server.

          Before, the errors returned a few hours after the restart jenkins, now we are working successfully already secondary day.

          We fixed problem after update "Pipeline Maven Integration Plugin" with 3.5.12 to 3.5.13 and restart jenkins.

           

           

           

           

          Anatoliy Pavlov added a comment - - edited We check "zombie process", but don't found them on master jenkins server. Before, the errors returned a few hours after the restart jenkins, now we are working successfully already secondary day. We fixed problem after update "Pipeline Maven Integration Plugin" with 3.5.12 to 3.5.13 and restart jenkins.        

          Cyrille Le Clerc added a comment -

          FYI I'm working at optimising the database queries to significantly reduce the workload on the database. This optimisation is very likely to reduce the need of concurrent connections to the database.

          Cyrille Le Clerc added a comment - FYI I'm working at optimising the database queries to significantly reduce the workload on the database. This optimisation is very likely to reduce the need of concurrent connections to the database.

          Cyrille Le Clerc added a comment -

          MySQL database support introduced in https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.6.0-beta-2

          Cyrille Le Clerc added a comment - MySQL database support introduced in https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.6.0-beta-2

          Cyrille Le Clerc added a comment - - edited

          MySQL database support available in 3.6.0.

          xed Can you use a MySQL database with the Pipeline Maven Plugin instead of the embedded H2 database as documented in "Pipeline Maven Plugin Setup > Using a MySQL Database (since 3.6.0)" ?

          Cyrille Le Clerc added a comment - - edited MySQL database support available in 3.6.0. xed Can you use a MySQL database with the Pipeline Maven Plugin instead of the embedded H2 database as documented in " Pipeline Maven Plugin Setup > Using a MySQL Database (since 3.6.0) " ?

            cleclerc Cyrille Le Clerc
            xed Anatoliy Pavlov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: