Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61830

Pipeline Maven Integration: migrate from H2 to MySQL/PostgreSQL

      Hi,

      We have been using the Pipeline Maven Integration plugin with the default H2 database for a while now and we are considering to switch to MySQL or PostgreSQL (as recommended).

      I've read the plugin wiki that describes how to configure it to use either MySQL or PostgreSQL but I could not find more information regarding how to actually migrate from H2.

      Does the data from the H2 database need to be migrated to the new database or can it be discarded? Would this affect the users in any way?

      Are there any other things to consider before switching to a new DB or can it be done "on the fly"? Any possible issues that might come up while/after the switch?

      Thanks and Best regards,

      Sebastian

          [JENKINS-61830] Pipeline Maven Integration: migrate from H2 to MySQL/PostgreSQL

          I've switched from H2 to PostgreSQL. Maybe it will be useful for someone so, mentioning this here.

          Copied the data from H2 to PostgreSQL by exporting it as CSV using the H2 CSVWRITE command and then importing it using the COPY .., FROM PostgreSQL commands.

          I've had to disable all triggers before doing this due to the constraints and re-enabling them after the data was imported.

          Sebastian Apro added a comment - I've switched from H2 to PostgreSQL. Maybe it will be useful for someone so, mentioning this here. Copied the data from H2 to PostgreSQL by exporting it as CSV using the H2 CSVWRITE command and then importing it using the COPY .., FROM PostgreSQL commands. I've had to disable all triggers before doing this due to the constraints and re-enabling them after the data was imported.

          Dear,

           

          I've done latelly this migration but I had to rollback to H2.

          Explanation, I've done export CSV from H2 then import to postgresSQL using this order to manage dependancies :

          1- build_results

          2- Version

          3- Jobs

          4- master

          5- build

          6- artifact

          7- build_upstream

          8- parent_project

          9- Depedancy

          10- generated_maven

           

          Unfortunatly after the restart of my instance I've got this message :

          Found errors after H2 migration ; org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "jenkins_build_pkey"

          I tried twice, using import with truncate option, but with no luck...

          Question : what happens if we start from an empty DB ? I have a huge instance with 10K Jobs and load of pipelines, will I lose lot of informations, and what kind of ?

           

          Frédéric Meyrou added a comment - Dear,   I've done latelly this migration but I had to rollback to H2. Explanation, I've done export CSV from H2 then import to postgresSQL using this order to manage dependancies : 1- build_results 2- Version 3- Jobs 4- master 5- build 6- artifact 7- build_upstream 8- parent_project 9- Depedancy 10- generated_maven   Unfortunatly after the restart of my instance I've got this message : Found errors after H2 migration ; org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "jenkins_build_pkey" I tried twice, using import with truncate option, but with no luck... Question : what happens if we start from an empty DB ? I have a huge instance with 10K Jobs and load of pipelines, will I lose lot of informations, and what kind of ?  

          I've tried the same. It looks like the issue is the ID fields for tables are generated via sequence, so when they get reimported by hand the IDs end up wrong, so the foreign key constraints result in failure.

          There's nothing in any documentation to say how to change from the default H2 to a real database despite this being really quite strongly recommended. There really ought to be some advice in that documentation to describe how to actually follow the advise of using a real database.

          Or will Jenkins automatically import the data from the H2 system once a real database is specified?

          Or is it safe to start with a blank database?

          William Whittle added a comment - I've tried the same. It looks like the issue is the ID fields for tables are generated via sequence, so when they get reimported by hand the IDs end up wrong, so the foreign key constraints result in failure. There's nothing in any documentation to say how to change from the default H2 to a real database despite this being really quite strongly recommended. There really ought to be some advice in that documentation to describe how to actually follow the advise of using a real database. Or will Jenkins automatically import the data from the H2 system once a real database is specified? Or is it safe to start with a blank database?

          I agree that we can improve the documentation. Do not hesitate to do a PR otherwise I'll try to do it.

          The plugin isn't designed to transfer the data between 2 databases.

          While I would love to document how to migrate the data I am not a DB specialist and I am really afraid that it will be very different from a DB to another (and maybe across versions).

          The easy solution is to restart from scratch but what does it means ? It means that the plugin will loose the knowledge of which project is producing / consuming the maven articfacts.

          Because of this the triggering won't happen.

          If project B depends on project A, when A is built a build of B won't occur.

          It's exactly the same thing when you create project B, it has to be manually launched once to discover/record the dependency and trigger

          Thus the workaround is to restart from scratch and trigger once manually all your jobs using the pipeline-maven-plugin (or with a groovy if you have a lot of jobs). This will record all the artifacts produced/consumed by your projects and they'll be triggered correctly in the future

           

          Arnaud Héritier added a comment - I agree that we can improve the documentation. Do not hesitate to do a PR otherwise I'll try to do it. The plugin isn't designed to transfer the data between 2 databases. While I would love to document how to migrate the data I am not a DB specialist and I am really afraid that it will be very different from a DB to another (and maybe across versions). The easy solution is to restart from scratch but what does it means ? It means that the plugin will loose the knowledge of which project is producing / consuming the maven articfacts. Because of this the triggering won't happen. If project B depends on project A, when A is built a build of B won't occur. It's exactly the same thing when you create project B, it has to be manually launched once to discover/record the dependency and trigger Thus the workaround is to restart from scratch and trigger once manually all your jobs using the pipeline-maven-plugin (or with a groovy if you have a lot of jobs). This will record all the artifacts produced/consumed by your projects and they'll be triggered correctly in the future  

          Thanks. That confirms what I thought was the case, that there's no support for the system to migrate the data when you change database.

          And I think what you're saying is that it's not a long-term problem to change storage. Yes, we'd lose some dependency knowledge in the short term, but that would be regained as we build over time. So I think the comment that's just needed in the documentation is simply not to worry too much about making the change from H2 to something else, but to be aware that the triggering won't happen to begin with, and that to restore that triggering an administrator would either wait for builds to happen naturally, or just to force the building of the projects so that data is rediscovered.

          I think documenting migration steps are therefore not really worth the effort.

          William Whittle added a comment - Thanks. That confirms what I thought was the case, that there's no support for the system to migrate the data when you change database. And I think what you're saying is that it's not a long-term problem to change storage. Yes, we'd lose some dependency knowledge in the short term, but that would be regained as we build over time. So I think the comment that's just needed in the documentation is simply not to worry too much about making the change from H2 to something else, but to be aware that the triggering won't happen to begin with, and that to restore that triggering an administrator would either wait for builds to happen naturally, or just to force the building of the projects so that data is rediscovered. I think documenting migration steps are therefore not really worth the effort.

            cleclerc Cyrille Le Clerc
            sapro Sebastian Apro
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: