Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36746

Jenkins 2.7.1 for debian-stable install fails: Size mismatch

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I am not sure if this belongs in Jenkins, website or infra, so I chose Jenkins because it appears the main project. If incorrect, my apologies for my mistake. I am new here.

      Installing Jenkins on Debian stable, as per the official instructions, fails completely. This problem appears to be introduced on 2016/07/14, when Jenkins version 2.7.1 was added to the repository.

      The command apt-get install jenkins will invariably result in:
      E: Failed to fetch http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Size mismatch

      So far, I have:

      • Reproduced this failure on at least 3 freshly installed Debian Stable systems.
      • Reproduced this failure from different geo-locations to rule out a localized failures.
      • Disabled APT caching systems to rule these out as the source of the problem.

      Manually downloading and installing the package does work, so the integrity of the deb package itself appears to be okay. It appears to be a corruption in the repository itself. The result is that installing Jenkins through APT on Debian stable (and maybe elsewhere) has become impossible. Upgrading it is probably also impossible.

      Because I was expecting this would have been noticed by more people already (and reported by now), I was first mostly searching for causes on my own end. I think I have ruled any of those out by now.

      Edit:
      The MD5 of both the manually downloaded (and successfully installed) package and the one that fails to install through APT are the same, so it really has to be a problem with the repository and not the package itself.
      26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb
      26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb.FAILED

      Edit:
      Installing any older version of Jenkins through APT, as a possible workaround, also proved impossible because older versions are not, or no longer, included in the Debian repository. The deb packages themselves exists and can be manually downloaded, but they can not be downloaded through APT. This could be just the result of a good police choice, to not clutter the repository with old versions. However, in this particular case it is rather unfortunate at the same time.

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Context, in case you're unaware: https://jenkins.io/blog/2016/07/14/2-7-1-re-release/

          TBH I'd rather have this problem than that one… still, we need to look into better handling of this case going forward. First time we ever needed something like this AFAIK.

          Show
          danielbeck Daniel Beck added a comment - Context, in case you're unaware: https://jenkins.io/blog/2016/07/14/2-7-1-re-release/ TBH I'd rather have this problem than that one… still, we need to look into better handling of this case going forward. First time we ever needed something like this AFAIK.
          Hide
          elwin Elwin Andriol added a comment -

          Thank you for that extra context, Daniel. I was not aware of that, mostly because I just started using Jenkins. It does confirm my personal findings, so at least I'm glad about that.

          I would not argue too quickly that this problem is less worse that the original problem though. Because currently, and this has apparently so far still not been fixed at all, depending on where you are located and what mirror gets selected for downloading the deb package (which appears to be a process that a user has no control over), Jenkins can simply not be installed at all (at least not through the preferred apt-get install method).

          I would like to stress that such a show stopper (because that is really what it is if the official instruction fail to work) can be very damaging to the first impressions of Jenkins to potential new users. That should not be underestimated IMHO. I happen to be the type that does not like to give up that quickly, but if a particular percentage (I do not know how many mirrors are actually out of sync) of all Debian users who recently tried to install Jenkins have come to the conclusion that this is just impossible, then it could even end up damaging the reputation of Jenkins as a whole (which is of course unrelated in reality, but that might not matter).

          From here on, I think the most important step would be to first make sure that all mirrors get synced ASAP. As for the future, I think that my suggestion to use a packaging version suffix the same way that Debian does for upstream maintained packages, might still not be such a bad idea. This way, changes in files that are not part of the upstream content can still be versioned. The convention here is to add a dash to the upstream version and then a packaging iteration number to it. So, the 2.7.1 package of Jenkins would have been generated as respectively:

          • jenkins_2.7.1-1_all.deb
          • jenkins_2.7.1-2_all.deb

          With such a naming scheme, even with mirrors out of sync, the current problem would not exist and Jenkins could still be installed (even if outdated). Of course, the issue with mirrors being out of sync is still a problem that needs to be address for its own obvious reasons.

          What so far worries me most though, is that apparently nobody is seriously taking up this problem. This problem has now existed for a week and I would not call this just a minor issue. The fact that it got unnoticed for a while is understandable (though it could be argued that proper CI would have prevented that as well). But after I filed this bug report, I still did not see much happening to have this issue swiftly resolved, by somebody with the required privileges to do so. This does in fact worry me. I would not mind putting in effort to resolve this issue, but I expect that I just don't have the required knowledge and neither the credentials for it.

          Of course, I could just have stumbled on that single or just a few obscure mirrors that are out of sync and only causing me problems (I tried installs from both The Netherlands and from Serbia), but until that is established as a fact, I personally would treat this problem as potentially big and rather serious one.

          Show
          elwin Elwin Andriol added a comment - Thank you for that extra context, Daniel. I was not aware of that, mostly because I just started using Jenkins. It does confirm my personal findings, so at least I'm glad about that. I would not argue too quickly that this problem is less worse that the original problem though. Because currently, and this has apparently so far still not been fixed at all , depending on where you are located and what mirror gets selected for downloading the deb package (which appears to be a process that a user has no control over), Jenkins can simply not be installed at all (at least not through the preferred apt-get install method). I would like to stress that such a show stopper (because that is really what it is if the official instruction fail to work) can be very damaging to the first impressions of Jenkins to potential new users. That should not be underestimated IMHO. I happen to be the type that does not like to give up that quickly, but if a particular percentage (I do not know how many mirrors are actually out of sync) of all Debian users who recently tried to install Jenkins have come to the conclusion that this is just impossible, then it could even end up damaging the reputation of Jenkins as a whole (which is of course unrelated in reality, but that might not matter). From here on, I think the most important step would be to first make sure that all mirrors get synced ASAP . As for the future, I think that my suggestion to use a packaging version suffix the same way that Debian does for upstream maintained packages, might still not be such a bad idea. This way, changes in files that are not part of the upstream content can still be versioned. The convention here is to add a dash to the upstream version and then a packaging iteration number to it. So, the 2.7.1 package of Jenkins would have been generated as respectively: jenkins_2.7.1-1_all.deb jenkins_2.7.1-2_all.deb With such a naming scheme, even with mirrors out of sync, the current problem would not exist and Jenkins could still be installed (even if outdated). Of course, the issue with mirrors being out of sync is still a problem that needs to be address for its own obvious reasons. What so far worries me most though, is that apparently nobody is seriously taking up this problem. This problem has now existed for a week and I would not call this just a minor issue. The fact that it got unnoticed for a while is understandable (though it could be argued that proper CI would have prevented that as well). But after I filed this bug report, I still did not see much happening to have this issue swiftly resolved, by somebody with the required privileges to do so. This does in fact worry me. I would not mind putting in effort to resolve this issue, but I expect that I just don't have the required knowledge and neither the credentials for it. Of course, I could just have stumbled on that single or just a few obscure mirrors that are out of sync and only causing me problems (I tried installs from both The Netherlands and from Serbia), but until that is established as a fact, I personally would treat this problem as potentially big and rather serious one.
          Hide
          rtyler R. Tyler Croy added a comment -

          Elwin Andriol the mirrors in the netherlands and serbia were the ones that were so out of date that I disabled them a few days ago (see also our mirrors status page)

          Since last night I have been provisioning, and destroying, new Jenkins masters on EC2 and I haven't yet seen any issues. This is mostly in us-west-2 so I will only be hitting the US mirrors however.

          Unfortunately since the mirrors are pull-based and operated voluntarily the best I can do is identify mirrors which have not pulled updates properly and disable them from our mirror indexing service.

          Show
          rtyler R. Tyler Croy added a comment - Elwin Andriol the mirrors in the netherlands and serbia were the ones that were so out of date that I disabled them a few days ago (see also our mirrors status page ) Since last night I have been provisioning, and destroying, new Jenkins masters on EC2 and I haven't yet seen any issues. This is mostly in us-west-2 so I will only be hitting the US mirrors however. Unfortunately since the mirrors are pull-based and operated voluntarily the best I can do is identify mirrors which have not pulled updates properly and disable them from our mirror indexing service.
          Hide
          elwin Elwin Andriol added a comment -

          I have just tried several new installs from The Netherlands and they now all appear to work.

          For the ones that did not appear to work yesterday, I may have only myself to blame for. It turns out that an apt-proxy server that I disabled/bypassed when this problem first appeared, got turned on and might have been serving the outdated packages (at least yesterday), while they no longer exist on any mirrors.

          My sincere apologies for this mistake on my end.

          Show
          elwin Elwin Andriol added a comment - I have just tried several new installs from The Netherlands and they now all appear to work. For the ones that did not appear to work yesterday, I may have only myself to blame for. It turns out that an apt-proxy server that I disabled/bypassed when this problem first appeared, got turned on and might have been serving the outdated packages (at least yesterday), while they no longer exist on any mirrors. My sincere apologies for this mistake on my end.
          Hide
          rtyler R. Tyler Croy added a comment -

          No problem, I know how frustrating problems like this can be, sometimes it seems like everything breaks at the same time.

          With our partnership with Azure we're going to be migrating from this mirrored setup to a direct CDN which will prevent issues like this in the future.

          Show
          rtyler R. Tyler Croy added a comment - No problem, I know how frustrating problems like this can be, sometimes it seems like everything breaks at the same time. With our partnership with Azure we're going to be migrating from this mirrored setup to a direct CDN which will prevent issues like this in the future.

            People

            Assignee:
            rtyler R. Tyler Croy
            Reporter:
            elwin Elwin Andriol
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: