Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-2562

Artifactory is no longer reliably adding releases to the Maven index

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Artifactory periodically needs to regenerate its Maven/Nexus index. It currently takes about 10 minutes for repo.jenkins-ci.org/releases and we kick the process off again within 1 minute of completion.

      update-center2 uses the Maven/Nexus index to determine which releases of core and plugins exist. If something isn't in the index, it is ignored.

      Until fairly recently (February?) the Artifactory Maven index reliably included newly deployed artifacts in the first index whose generation started after the deployment, possible even earlier.

      I.e. if you deploy something on 8:05 AM, and index generation is in 10 minute intervals, it may have been in the index generated from 8:00 AM to 8:10 AM, but will have been in the index generated from 8:10 to 8:20 AM.

      Manual control of the index creation trigger also let us reliably include an artifact on the next run before (useful when staging security updates – suspend index updates until we had copied them to the /releases repo, then immediately kick off an indexing).

      Since ~March 2020, this no longer works.

      First observed during the delivery of the March 25 security updates, it can take several indexing runs for newly deployed artifacts to be picked up.

      I am fairly sure this problem had not yet affected the March 9 security updates.

      As an example, on April 4, an artifact deployed at 7:03 AM got its content indexed at 7:04 AM by ArchiveIndexerImpl, but despite generating new Maven indexes about every 10 minutes, the Artifact only appeared in the  index dated 2020-04-04T08:58:30.212Z, it wasn't yet in the index dated 2020-04-04T08:48:21.192Z.

      2020-04-04 06:50:38,850 [art-exec-881990] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 06:50:38,850 [art-exec-881990] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 06:59:56,571 [art-exec-881990] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:00:01,075 [art-exec-882355] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:00:01,076 [art-exec-882355] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:04:24,432 [art-exec-882543] [INFO ] (o.a.s.a.ArchiveIndexerImpl:146) - Indexing archive: releases:org/jenkins-ci/plugins/fitnesse/1.33/fitnesse-1.33.hpi
      2020-04-04 07:04:24,557 [art-exec-882543] [INFO ] (o.a.s.a.ArchiveIndexerImpl:146) - Indexing archive: releases:org/jenkins-ci/plugins/fitnesse/1.33/fitnesse-1.33.jar
      2020-04-04 07:04:24,896 [art-exec-882543] [INFO ] (o.a.s.a.ArchiveIndexerImpl:146) - Indexing archive: releases:org/jenkins-ci/plugins/fitnesse/1.33/fitnesse-1.33-sources.jar
      2020-04-04 07:08:17,864 [art-exec-882355] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:09:01,198 [art-exec-882724] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:09:01,198 [art-exec-882724] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:16:59,942 [art-exec-882724] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:20:13,328 [art-exec-883160] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:20:13,329 [art-exec-883160] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:28:57,623 [art-exec-883160] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:29:00,946 [art-exec-883504] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:29:00,947 [art-exec-883504] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:37:42,739 [art-exec-883504] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:37:43,926 [art-exec-883840] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:37:43,927 [art-exec-883840] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:46:43,216 [art-exec-883840] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:47:01,055 [art-exec-884203] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:47:01,055 [art-exec-884203] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 07:56:10,359 [art-exec-884203] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 07:57:01,608 [art-exec-884592] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 07:57:01,608 [art-exec-884592] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:05:19,536 [art-exec-884592] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:07:47,801 [art-exec-885013] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:07:47,801 [art-exec-885013] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:16:02,540 [art-exec-885013] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:19:56,343 [art-exec-885486] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:19:56,343 [art-exec-885486] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:28:17,146 [art-exec-885486] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:29:01,010 [art-exec-885841] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:29:01,011 [art-exec-885841] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:38:00,910 [art-exec-885841] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:38:00,988 [art-exec-886188] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:38:00,989 [art-exec-886188] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:48:44,509 [art-exec-886188] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:49:01,709 [art-exec-886626] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:49:01,709 [art-exec-886626] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
      2020-04-04 08:58:53,375 [art-exec-886626] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:191) - Finished Maven indexing...
      2020-04-04 08:59:01,189 [art-exec-887018] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
      2020-04-04 08:59:01,189 [art-exec-887018] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Interestingly, the notification email when watching the repository was sent around the same time the artifact was added to the index (i.e. two hours late). I wonder whether this is related.

          Show
          danielbeck Daniel Beck added a comment - Interestingly, the notification email when watching the repository was sent around the same time the artifact was added to the index (i.e. two hours late). I wonder whether this is related.
          Hide
          danielbeck Daniel Beck added a comment -

          Also, email notifications hadn't worked since Dec 1 2019 for me, and resumed March 17, informing about an artifact added on March 12. This could be an indication when exactly the bad build was deployed.

          Show
          danielbeck Daniel Beck added a comment - Also, email notifications hadn't worked since Dec 1 2019 for me, and resumed March 17, informing about an artifact added on March 12. This could be an indication when exactly the bad build was deployed.
          Hide
          danielbeck Daniel Beck added a comment -

          Update center no longer depends on the Nexus Maven index since https://github.com/jenkins-infra/update-center2/pull/364

          Show
          danielbeck Daniel Beck added a comment - Update center no longer depends on the Nexus Maven index since https://github.com/jenkins-infra/update-center2/pull/364

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            danielbeck Daniel Beck
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: