Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-1730

set up new fallback host for mirrorbrain

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Reopened (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: puppet
    • Labels:
      None
    • Similar Issues:

      Description

      We currently have the problem that the fallback host has extremely limited bandwidth, so that recently plugins, that aren't yet available on mirrors, may not be downloadable for most users.

      (INFRA-1302)

      To prevent this problem, I propose to set up a fallback mirror that is not file based, but rather just contains an htaccess file with redirects to the Artifactory maven repo.

      update-center2 is being extended to generate the htaccess file in https://github.com/jenkins-infra/update-center2/pull/224

      All we need is a new virtualhost or similar, and that to be configured as the fallback mirror in mirrorbrain. Then we regularly rsync the generated htaccess file from update-center2 builds, and done.

      R. Tyler Croy asked that Olivier take care of this, since he has no time.

        Attachments

          Issue Links

            Activity

            Hide
            olblak Olivier Vernin added a comment -

            The blob storage that will host the htaccess files https://github.com/jenkins-infra/azure/pull/60

            Show
            olblak Olivier Vernin added a comment - The blob storage that will host the htaccess files https://github.com/jenkins-infra/azure/pull/60
            Hide
            olblak Olivier Vernin added a comment -

            Jenkins-infra pr to deploy the service https://github.com/jenkins-infra/jenkins-infra/pull/1067

            Show
            olblak Olivier Vernin added a comment - Jenkins-infra pr to deploy the service https://github.com/jenkins-infra/jenkins-infra/pull/1067
            Hide
            olblak Olivier Vernin added a comment -

            Trusted.ci is configured with the correct credential

            Show
            olblak Olivier Vernin added a comment - Trusted.ci is configured with the correct credential
            Show
            olblak Olivier Vernin added a comment - Working http://updates.azure.jenkins.io/plugins/AnchorChain/1.0/AnchorChain.hpi
            Hide
            olblak Olivier Vernin added a comment -

            I am reopening this ticket as I am not sure how to validate this service.

            From updates.azure.jenkins.io:

            And if we want to use updates.azure.jenkins.io as a fallback service for mirrors.jenkins.io, the configuration seems to be here but actually updates.azure.jenkins.io can only be a fallback for mirrors.jenkins.io/plugins

            R. Tyler Croy any thoughs?

            Show
            olblak Olivier Vernin added a comment - I am reopening this ticket as I am not sure how to validate this service. From updates.azure.jenkins.io: http://updates.azure.jenkins.io/plugins/AnchorChain/1.0/AnchorChain.hpi works http://updates.azure.jenkins.io/plugins/AnchorChain/latest/AnchorChain.hpi doesn't works And if we want to use updates.azure.jenkins.io as a fallback service for mirrors.jenkins.io, the configuration seems to be here but actually updates.azure.jenkins.io can only be a fallback for mirrors.jenkins.io/plugins R. Tyler Croy any thoughs?
            Hide
            danielbeck Daniel Beck added a comment -

            http://updates.azure.jenkins.io/plugins/AnchorChain/latest/AnchorChain.hpi doesn't works

            Why would it have to? The redirect to a specific version is defined on updates.j.io level, so this request would never arrive as is at the fallback mirror (or any mirror really):

            $ curl -IL https://updates.jenkins-ci.org/latest/accurev.hpi
            HTTP/1.1 302 Found
            Date: Mon, 06 Aug 2018 12:09:13 GMT
            Server: Apache/2.4.7 (Ubuntu)
            Location: https://updates.jenkins-ci.org/download/plugins/accurev/0.7.17/accurev.hpi
            Content-Type: text/html; charset=iso-8859-1
            
            HTTP/1.1 302 Found
            Date: Mon, 06 Aug 2018 12:09:13 GMT
            Server: Apache/2.4.7 (Ubuntu)
            Location: http://mirrors.jenkins-ci.org/plugins/accurev/0.7.17/accurev.hpi
            Content-Type: text/html; charset=iso-8859-1
            
            HTTP/1.1 302 Found
            Date: Mon, 06 Aug 2018 12:09:14 GMT
            Server: Apache/2.4.7 (Ubuntu)
            X-MirrorBrain-Mirror: ftp-chi.osuosl.org
            X-MirrorBrain-Realm: other
            Link: <http://mirrors.jenkins-ci.org/plugins/accurev/0.7.17/accurev.hpi.meta4>; rel=describedby; type="application/metalink4+xml"
            Link: <http://ftp-chi.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi>; rel=duplicate; pri=1; geo=us
            Link: <http://ftp-nyc.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi>; rel=duplicate; pri=2; geo=us
            Location: http://ftp-chi.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi
            Content-Type: text/html; charset=iso-8859-1
            
            HTTP/1.1 200 OK
            Date: Mon, 06 Aug 2018 12:09:15 GMT
            Server: Apache
            Last-Modified: Mon, 30 Jul 2018 13:51:35 GMT
            ETag: "2648b-57237c27b3c4d"
            Accept-Ranges: bytes
            Content-Length: 156811
            Content-Type: application/vnd.hp-hpid

            Besides that, we don't use latest version links in our distribution. The goal here is basically to fix INFRA-1302, and for that, we always link to specific versions.


            And if we want to use updates.azure.jenkins.io as a fallback service for mirrors.jenkins.io, the configuration seems to be here but actually updates.azure.jenkins.io can only be a fallback for mirrors.jenkins.io/plugins

            It's unclear what this is about.

            AFAIK, war packages get mirrorsync'ed on creation (it's part of KK's script I think), so there's never a period during which mirrors are outdated.

            The same with native packages, and they even get served from Azure in some (all?) situations.

            Show
            danielbeck Daniel Beck added a comment - http://updates.azure.jenkins.io/plugins/AnchorChain/latest/AnchorChain.hpi doesn't works Why would it have to? The redirect to a specific version is defined on updates.j.io level, so this request would never arrive as is at the fallback mirror (or any mirror really): $ curl -IL https://updates.jenkins-ci.org/latest/accurev.hpi HTTP/1.1 302 Found Date: Mon, 06 Aug 2018 12:09:13 GMT Server: Apache/2.4.7 (Ubuntu) Location: https://updates.jenkins-ci.org/download/plugins/accurev/0.7.17/accurev.hpi Content-Type: text/html; charset=iso-8859-1 HTTP/1.1 302 Found Date: Mon, 06 Aug 2018 12:09:13 GMT Server: Apache/2.4.7 (Ubuntu) Location: http://mirrors.jenkins-ci.org/plugins/accurev/0.7.17/accurev.hpi Content-Type: text/html; charset=iso-8859-1 HTTP/1.1 302 Found Date: Mon, 06 Aug 2018 12:09:14 GMT Server: Apache/2.4.7 (Ubuntu) X-MirrorBrain-Mirror: ftp-chi.osuosl.org X-MirrorBrain-Realm: other Link: <http://mirrors.jenkins-ci.org/plugins/accurev/0.7.17/accurev.hpi.meta4>; rel=describedby; type="application/metalink4+xml" Link: <http://ftp-chi.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi>; rel=duplicate; pri=1; geo=us Link: <http://ftp-nyc.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi>; rel=duplicate; pri=2; geo=us Location: http://ftp-chi.osuosl.org/pub/jenkins/plugins/accurev/0.7.17/accurev.hpi Content-Type: text/html; charset=iso-8859-1 HTTP/1.1 200 OK Date: Mon, 06 Aug 2018 12:09:15 GMT Server: Apache Last-Modified: Mon, 30 Jul 2018 13:51:35 GMT ETag: "2648b-57237c27b3c4d" Accept-Ranges: bytes Content-Length: 156811 Content-Type: application/vnd.hp-hpid Besides that, we don't use latest version links in our distribution. The goal here is basically to fix INFRA-1302 , and for that, we always link to specific versions. And if we want to use updates.azure.jenkins.io as a fallback service for mirrors.jenkins.io, the configuration seems to be here but actually updates.azure.jenkins.io can only be a fallback for mirrors.jenkins.io/plugins It's unclear what this is about. AFAIK, war packages get mirrorsync'ed on creation (it's part of KK's script I think), so there's never a period during which mirrors are outdated. The same with native packages, and they even get served from Azure in some (all?) situations.
            Hide
            danielbeck Daniel Beck added a comment -

            INFRA-2293 notes that mirrorbrain goes to archives.jenkins-ci.org even before the mirror has been populated, so the explanation that mirrorbrain rejects a "mirror" without the files present that, IIRC, was the reason we never implemented this, seems to not hold.

            Show
            danielbeck Daniel Beck added a comment - INFRA-2293 notes that mirrorbrain goes to archives.jenkins-ci.org even before the mirror has been populated, so the explanation that mirrorbrain rejects a "mirror" without the files present that, IIRC, was the reason we never implemented this, seems to not hold.
            Hide
            olblak Olivier Vernin added a comment -

            I have some ongoing work to deploy a new mirror on Azure to replicated archives.jenkins.io
            -> https://github.com/jenkins-infra/charts/pull/130
            The idea is to upload every files on an azure file storage directly from sync.sh
            -> https://github.com/jenkins-infra/jenkins-infra/pulls

            Show
            olblak Olivier Vernin added a comment - I have some ongoing work to deploy a new mirror on Azure to replicated archives.jenkins.io -> https://github.com/jenkins-infra/charts/pull/130 The idea is to upload every files on an azure file storage directly from sync.sh -> https://github.com/jenkins-infra/jenkins-infra/pulls

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              danielbeck Daniel Beck
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated: