Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36746

Jenkins 2.7.1 for debian-stable install fails: Size mismatch

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I am not sure if this belongs in Jenkins, website or infra, so I chose Jenkins because it appears the main project. If incorrect, my apologies for my mistake. I am new here.

      Installing Jenkins on Debian stable, as per the official instructions, fails completely. This problem appears to be introduced on 2016/07/14, when Jenkins version 2.7.1 was added to the repository.

      The command apt-get install jenkins will invariably result in:
      E: Failed to fetch http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Size mismatch

      So far, I have:

      • Reproduced this failure on at least 3 freshly installed Debian Stable systems.
      • Reproduced this failure from different geo-locations to rule out a localized failures.
      • Disabled APT caching systems to rule these out as the source of the problem.

      Manually downloading and installing the package does work, so the integrity of the deb package itself appears to be okay. It appears to be a corruption in the repository itself. The result is that installing Jenkins through APT on Debian stable (and maybe elsewhere) has become impossible. Upgrading it is probably also impossible.

      Because I was expecting this would have been noticed by more people already (and reported by now), I was first mostly searching for causes on my own end. I think I have ruled any of those out by now.

      Edit:
      The MD5 of both the manually downloaded (and successfully installed) package and the one that fails to install through APT are the same, so it really has to be a problem with the repository and not the package itself.
      26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb
      26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb.FAILED

      Edit:
      Installing any older version of Jenkins through APT, as a possible workaround, also proved impossible because older versions are not, or no longer, included in the Debian repository. The deb packages themselves exists and can be manually downloaded, but they can not be downloaded through APT. This could be just the result of a good police choice, to not clutter the repository with old versions. However, in this particular case it is rather unfortunate at the same time.

        Attachments

          Activity

          Hide
          rtyler R. Tyler Croy added a comment -

          If you try to wget that debian package URL, can you paste that console output? I'm wondering if you may have encountered some out-of-date mirrors.

          I disabled two mirrors yesterday and I'm curious if that magically fixed your issue or if there is something different going on here

          Show
          rtyler R. Tyler Croy added a comment - If you try to wget that debian package URL, can you paste that console output? I'm wondering if you may have encountered some out-of-date mirrors. I disabled two mirrors yesterday and I'm curious if that magically fixed your issue or if there is something different going on here
          Hide
          elwin Elwin Andriol added a comment -

          The Debian package file is not the problem. Both the one that is being downloaded manually and the one being downloaded through apt-get install have exactly the same checksum and install fine when done so manually through dpkg (after rename the downloaded ). The problem resides in the repository manifest/index files, not the deb package.

          Apparently, I can no longer reproduce the failure, whereas earlier I could reproduce it with 100% accuracy from several different geo-locations.

          So, whatever change took place, it appears to have solved the problem.

          Show
          elwin Elwin Andriol added a comment - The Debian package file is not the problem. Both the one that is being downloaded manually and the one being downloaded through apt-get install have exactly the same checksum and install fine when done so manually through dpkg (after rename the downloaded ). The problem resides in the repository manifest/index files, not the deb package. Apparently, I can no longer reproduce the failure, whereas earlier I could reproduce it with 100% accuracy from several different geo-locations. So, whatever change took place, it appears to have solved the problem.
          Hide
          elwin Elwin Andriol added a comment - - edited

          Sadly, the problem does appear to persist again. I do not know why it did work an hour ago and again stopped working.

          # apt-cache show jenkins
          Package: jenkins
          Priority: extra
          Section: devel
          Installed-Size: 67318
          Maintainer: Kohsuke Kawaguchi <kk@kohsuke.org>
          Architecture: all
          Version: 2.7.1
          Replaces: hudson
          Depends: daemon, adduser, procps, psmisc, net-tools, default-jre-headless (>= 2:1.7) | java7-runtime-headless
          Conflicts: hudson
          Filename: binary/jenkins_2.7.1_all.deb
          Size: 68024840
          MD5sum: f4377650305fbf43b6a0a96e19c0dc87
          SHA1: 1e5725d7678a9a41d33d3ad3d1f96f28a00076c1
          SHA256: 59ca57cba31c9aafdc7a3bf97e2fc46b082a5edcfd937868bc99dbd3b7caeb5e
          SHA512: d5933574995d961df2457b2f3139d95c3176a4cbf71371cba4f2b5f4bf5422d52d2faaa46452bc5840d7d5a508650c9012c9879c26853b06569cec2c4f8e4de5
          Description: continuous integration system
          Jenkins is an application that monitors executions of repeated jobs,
          such as building a software project or jobs run by cron.
          Description-md5: 483e336ea11484aaa0d84b76602263f7
          Homepage: http://jenkins.io/

          # apt-get install jenkins
          ...
          Get:1 http://pkg.jenkins-ci.org/debian-stable/ binary/ jenkins 2.7.1 [68.0 MB]
          Fetched 68.0 MB in 2s (26.8 MB/s)
          E: Failed to fetch http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Size mismatch

          # md5sum /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED
          26eb48db6c1b369c2033e54c5a9062e3 /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED

          _ # wget http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb _
          -2016-07-20 19:55:21- http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb
          Resolving pkg.jenkins-ci.org (pkg.jenkins-ci.org)... 52.202.51.185
          Connecting to pkg.jenkins-ci.org (pkg.jenkins-ci.org)|52.202.51.185|:80... connected.
          HTTP request sent, awaiting response... 302 Found
          Location: http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb [following]
          -2016-07-20 19:55:22- http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb
          Resolving mirrors.jenkins.io (mirrors.jenkins.io)... 52.202.51.185
          Reusing existing connection to pkg.jenkins-ci.org:80.
          HTTP request sent, awaiting response... 302 Found
          Location: http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb [following]
          -2016-07-20 19:55:22- http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb
          Resolving ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)... 137.226.34.46, 2a00:8a60:e012:a00::21
          Connecting to ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)|137.226.34.46|:80... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 68024840 (65M) [application/octet-stream]
          Saving to: ‘jenkins_2.7.1_all.deb’
          ...
          2016-07-20 19:57:31 (515 KB/s) - ‘jenkins_2.7.1_all.deb’ saved [68024840/68024840]

          # md5sum jenkins_2.7.1_all.deb
          f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb

          This time around, it appears that the manually downloaded version has the same checksum as it should have according to the repository index/manifest, but the one that gets downloaded through apt-get install jenkins clearly is not. This could indeed indicate that there is as serious problem with mirrors serving the wrong files.

          I do not know how to investigate from which mirror apt downloads its manifest/index files each time on an apt-get update and neither how I can determine which mirror is used on an apt-get install. Any assistance there would be welcome.

          Show
          elwin Elwin Andriol added a comment - - edited Sadly, the problem does appear to persist again. I do not know why it did work an hour ago and again stopped working. # apt-cache show jenkins Package: jenkins Priority: extra Section: devel Installed-Size: 67318 Maintainer: Kohsuke Kawaguchi <kk@kohsuke.org> Architecture: all Version: 2.7.1 Replaces: hudson Depends: daemon, adduser, procps, psmisc, net-tools, default-jre-headless (>= 2:1.7) | java7-runtime-headless Conflicts: hudson Filename: binary/jenkins_2.7.1_all.deb Size: 68024840 MD5sum: f4377650305fbf43b6a0a96e19c0dc87 SHA1: 1e5725d7678a9a41d33d3ad3d1f96f28a00076c1 SHA256: 59ca57cba31c9aafdc7a3bf97e2fc46b082a5edcfd937868bc99dbd3b7caeb5e SHA512: d5933574995d961df2457b2f3139d95c3176a4cbf71371cba4f2b5f4bf5422d52d2faaa46452bc5840d7d5a508650c9012c9879c26853b06569cec2c4f8e4de5 Description: continuous integration system Jenkins is an application that monitors executions of repeated jobs, such as building a software project or jobs run by cron. Description-md5: 483e336ea11484aaa0d84b76602263f7 Homepage: http://jenkins.io/ # apt-get install jenkins ... Get:1 http://pkg.jenkins-ci.org/debian-stable/ binary/ jenkins 2.7.1 [68.0 MB] Fetched 68.0 MB in 2s (26.8 MB/s) E: Failed to fetch http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Size mismatch # md5sum /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED 26eb48db6c1b369c2033e54c5a9062e3 /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED _ # wget http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb _ - 2016-07-20 19:55:21 - http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Resolving pkg.jenkins-ci.org (pkg.jenkins-ci.org)... 52.202.51.185 Connecting to pkg.jenkins-ci.org (pkg.jenkins-ci.org)|52.202.51.185|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb [following] - 2016-07-20 19:55:22 - http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb Resolving mirrors.jenkins.io (mirrors.jenkins.io)... 52.202.51.185 Reusing existing connection to pkg.jenkins-ci.org:80. HTTP request sent, awaiting response... 302 Found Location: http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb [following] - 2016-07-20 19:55:22 - http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb Resolving ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)... 137.226.34.46, 2a00:8a60:e012:a00::21 Connecting to ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)|137.226.34.46|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 68024840 (65M) [application/octet-stream] Saving to: ‘jenkins_2.7.1_all.deb’ ... 2016-07-20 19:57:31 (515 KB/s) - ‘jenkins_2.7.1_all.deb’ saved [68024840/68024840] # md5sum jenkins_2.7.1_all.deb f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb This time around, it appears that the manually downloaded version has the same checksum as it should have according to the repository index/manifest, but the one that gets downloaded through apt-get install jenkins clearly is not. This could indeed indicate that there is as serious problem with mirrors serving the wrong files. I do not know how to investigate from which mirror apt downloads its manifest/index files each time on an apt-get update and neither how I can determine which mirror is used on an apt-get install . Any assistance there would be welcome.
          Hide
          elwin Elwin Andriol added a comment -

          Additional questions:
          How or why is it even possible that there are mirrors with the same file, as in with the same package name and version, with different checksums?

          Usually, a particular version of a package is supposed to be immutable from the moment it gets released. Even if mirrors end up out of cync, there should never be package files with the same name (and thus also version) but with different content. Or am I missing something?

          I do not know enough about the situation/infrastructure to make any informed statements, but the signs so far do give me reason to worry.

          Show
          elwin Elwin Andriol added a comment - Additional questions: How or why is it even possible that there are mirrors with the same file, as in with the same package name and version, with different checksums? Usually, a particular version of a package is supposed to be immutable from the moment it gets released. Even if mirrors end up out of cync, there should never be package files with the same name (and thus also version) but with different content. Or am I missing something? I do not know enough about the situation/infrastructure to make any informed statements, but the signs so far do give me reason to worry.
          Hide
          elwin Elwin Andriol added a comment - - edited

          I think I have found the source of the problem, or at least a crucial part of it.

          EVENTS:

          • Wed, 06 Jul 2016 12:44:04 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 26eb48db6c1b369c2033e54c5a9062e3
          • Thu, 14 Jul 2016 07:58:06 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 f4377650305fbf43b6a0a96e19c0dc87

          CONCLUSIONS:

          • Apparently, there are (at least) two different versions of the same package file (jenkins_2.7.1_all.deb) and their corresponding repository manifest/index file in circulation.
          • Apparently, the deb file that gets downloaded through apt can come (and apparently often does) from a different origin than the repository manifest/index, resulting in a mismatch.
          • Because two package files got changed (/etc/default/jenkins & /etc/init.d/jenkins), outside of the versioning policy for Jenkins itself, rebuiling the package does appear to have been legit.
          • The usual packaging version suffix for changes outside the scope of the versioning of an upstream package, like e.g. jenkins_2.7.1-1_all.deb and jenkins_2.7.1-2_all.deb, was for some reason not used here.

          SUGGESTED SOLUTION:

          • A swift update of all mirrors that still contain the outdated deb package file and their corresponding Debian repository index/manifest files.
          • Never again rebuild packages with exactly the same version numbers (e.g. by using an extra packaging suffix), or at least ensure a very fast and reliable mirror sync process if this can not be avoided.

          REMARKS:

          • The combined result of badly synched mirrors and deb packages being rebuild (even if with no changes but just different checksums because of a new timestamp), can evidently turn out disasterous.
          • The current situation was probably unforseen, but especially for a CI project, it could still easily discourage a new user like me. Especially a less technical savvy user, who might only notice that they just can't get Jenkins up and running at all.

          PROOF:
          # verify the checksums of the two deb packages we got:
          $ md5sum jenkins_*
          f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb
          26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb.FAILED

          # Unpack both package files:
          $ mkdir jenkins jenkins.FAILED
          $ cd jenkins
          $ ar -x ../jenkins_2.7.1_all.deb
          $ cd ../jenkins.FAILED/
          $ ar -x ../jenkins_2.7.1_all.deb.FAILED
          $ cd ..

          # Unpack control and data archives of 1st deb package:
          $ cd jenkins
          $ mkdir control data
          $ cd control
          $ tar zxf ../control.tar.gz
          $ cd ../data
          $ tar Jxf ../data.tar.xz
          $ cd ../..

          # Unpack control and data archives of 2nd deb package:
          $ cd jenkins.FAILED
          $ mkdir control data
          $ cd control
          $ tar zxf ../control.tar.gz
          $ cd ../data
          $ tar Jxf ../data.tar.xz
          $ cd ../..

          # Inspect 1st package control files:
          $ cd jenkins/control
          $ md5sum *
          c26b730354d8ae1c669c64a7420cff2d conffiles
          935454337cb6099d6bc475f33f0b08d4 control
          d42e0ffd4ce78afc0fd542a92809db8c md5sums
          ddd85780e84b8cf07dc50aab8ef35fc4 postinst
          452b5d5406a95b199ed4eb72a676fdd9 postrm
          c728a327fc7c2ad8699043a4c0ea1e94 prerm
          $ cd ../..

          # Inspect 2nd package control files:
          $ cd jenkins.FAILED/control
          $ md5sum *
          c26b730354d8ae1c669c64a7420cff2d conffiles
          935454337cb6099d6bc475f33f0b08d4 control
          21e3e095b7219ab11a1cca48f5839131 md5sums
          ddd85780e84b8cf07dc50aab8ef35fc4 postinst
          452b5d5406a95b199ed4eb72a676fdd9 postrm
          c728a327fc7c2ad8699043a4c0ea1e94 prerm
          $ cd ../..

          # Compare the entries in both md5sums files:
          $ find . -name md5sums | xargs cat | sort -k 2
          78b54b39b16b4f49d1588a0b39bb1220 usr/share/doc/jenkins/changelog.gz
          a8d87b06e94ded342e4e1902dbab37d6 usr/share/doc/jenkins/changelog.gz
          5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright
          5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright
          207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war
          207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war

          # Please note that jenkins.war is identical in both instances!

          # Inspect the differences in both changelog.gz files:
          $ zdiff jenkins/data/usr/share/doc/jenkins/changelog.gz jenkins.FAILED/data/usr/share/doc/jenkins/changelog.gz
          5c5
          < – Kohsuke Kawaguchi <kk@kohsuke.org> Thu, 14 Jul 2016 07:58:06 -0700

          > – Kohsuke Kawaguchi <kk@kohsuke.org> Wed, 06 Jul 2016 12:44:04 -0700

          # Compare all files to see if there are any other changes:
          $ find . -type f | xargs md5sum | sed 's|\.FAILED||' | sort -k 2
          26eb48db6c1b369c2033e54c5a9062e3 ./jenkins_2.7.1_all.deb
          f4377650305fbf43b6a0a96e19c0dc87 ./jenkins_2.7.1_all.deb
          c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles
          c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles
          935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control
          935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control
          21e3e095b7219ab11a1cca48f5839131 ./jenkins/control/md5sums
          d42e0ffd4ce78afc0fd542a92809db8c ./jenkins/control/md5sums
          ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst
          ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst
          452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm
          452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm
          c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm
          c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm
          1d32e21af04baf0f9747a9e84c592541 ./jenkins/control.tar.gz
          c7fbeb4ef29058e54242b5088aa52d1e ./jenkins/control.tar.gz
          7a7fd1d1e607ff49a0409780472c1ece ./jenkins/data/etc/default/jenkins
          ffb23af22df84fa22e4c6cb02a06e460 ./jenkins/data/etc/default/jenkins
          443c4e9cc5f8e24139270ed5a6cf9d22 ./jenkins/data/etc/init.d/jenkins
          e7d1b32adba36743f4655e2a2e1e19d8 ./jenkins/data/etc/init.d/jenkins
          9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins
          9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins
          0833d8e1b1af7b28f28810a75ba468a0 ./jenkins/data.tar.xz
          b795121ef63336db3602be5e135c1cc6 ./jenkins/data.tar.xz
          78b54b39b16b4f49d1588a0b39bb1220 ./jenkins/data/usr/share/doc/jenkins/changelog.gz
          a8d87b06e94ded342e4e1902dbab37d6 ./jenkins/data/usr/share/doc/jenkins/changelog.gz
          5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright
          5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright
          207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war
          207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war
          3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary
          3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary

          # Inspect changes in /etc/default/jenkins:
          $ diff jenkins/data/etc/default/jenkins jenkins.FAILED/data/etc/default/jenkins
          58a59,60
          > # port for AJP connector (disabled by default)
          > AJP_PORT=-1
          66a69
          > # --ajp13Port=$AJP_PORT
          72c75
          < JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"

          > JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"

          # Inspect changes in /etc/init.d/jenkins
          $ diff jenkins/data/etc/init.d/jenkins jenkins.FAILED/data/etc/init.d/jenkins
          132c132
          < return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java|daemon)' | grep -c . `

          > return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java)' | grep -v defunct | grep -c . `

          Show
          elwin Elwin Andriol added a comment - - edited I think I have found the source of the problem, or at least a crucial part of it. EVENTS: Wed, 06 Jul 2016 12:44:04 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 26eb48db6c1b369c2033e54c5a9062e3 Thu, 14 Jul 2016 07:58:06 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 f4377650305fbf43b6a0a96e19c0dc87 CONCLUSIONS: Apparently, there are (at least) two different versions of the same package file (jenkins_2.7.1_all.deb) and their corresponding repository manifest/index file in circulation. Apparently, the deb file that gets downloaded through apt can come (and apparently often does) from a different origin than the repository manifest/index, resulting in a mismatch. Because two package files got changed (/etc/default/jenkins & /etc/init.d/jenkins), outside of the versioning policy for Jenkins itself, rebuiling the package does appear to have been legit. The usual packaging version suffix for changes outside the scope of the versioning of an upstream package, like e.g. jenkins_2.7.1-1_all.deb and jenkins_2.7.1-2_all.deb, was for some reason not used here. SUGGESTED SOLUTION: A swift update of all mirrors that still contain the outdated deb package file and their corresponding Debian repository index/manifest files. Never again rebuild packages with exactly the same version numbers (e.g. by using an extra packaging suffix), or at least ensure a very fast and reliable mirror sync process if this can not be avoided. REMARKS: The combined result of badly synched mirrors and deb packages being rebuild (even if with no changes but just different checksums because of a new timestamp), can evidently turn out disasterous. The current situation was probably unforseen, but especially for a CI project, it could still easily discourage a new user like me. Especially a less technical savvy user, who might only notice that they just can't get Jenkins up and running at all. PROOF: # verify the checksums of the two deb packages we got: $ md5sum jenkins_* f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb 26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb.FAILED # Unpack both package files: $ mkdir jenkins jenkins.FAILED $ cd jenkins $ ar -x ../jenkins_2.7.1_all.deb $ cd ../jenkins.FAILED/ $ ar -x ../jenkins_2.7.1_all.deb.FAILED $ cd .. # Unpack control and data archives of 1st deb package: $ cd jenkins $ mkdir control data $ cd control $ tar zxf ../control.tar.gz $ cd ../data $ tar Jxf ../data.tar.xz $ cd ../.. # Unpack control and data archives of 2nd deb package: $ cd jenkins.FAILED $ mkdir control data $ cd control $ tar zxf ../control.tar.gz $ cd ../data $ tar Jxf ../data.tar.xz $ cd ../.. # Inspect 1st package control files: $ cd jenkins/control $ md5sum * c26b730354d8ae1c669c64a7420cff2d conffiles 935454337cb6099d6bc475f33f0b08d4 control d42e0ffd4ce78afc0fd542a92809db8c md5sums ddd85780e84b8cf07dc50aab8ef35fc4 postinst 452b5d5406a95b199ed4eb72a676fdd9 postrm c728a327fc7c2ad8699043a4c0ea1e94 prerm $ cd ../.. # Inspect 2nd package control files: $ cd jenkins.FAILED/control $ md5sum * c26b730354d8ae1c669c64a7420cff2d conffiles 935454337cb6099d6bc475f33f0b08d4 control 21e3e095b7219ab11a1cca48f5839131 md5sums ddd85780e84b8cf07dc50aab8ef35fc4 postinst 452b5d5406a95b199ed4eb72a676fdd9 postrm c728a327fc7c2ad8699043a4c0ea1e94 prerm $ cd ../.. # Compare the entries in both md5sums files: $ find . -name md5sums | xargs cat | sort -k 2 78b54b39b16b4f49d1588a0b39bb1220 usr/share/doc/jenkins/changelog.gz a8d87b06e94ded342e4e1902dbab37d6 usr/share/doc/jenkins/changelog.gz 5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright 5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright 207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war 207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war # Please note that jenkins.war is identical in both instances! # Inspect the differences in both changelog.gz files: $ zdiff jenkins/data/usr/share/doc/jenkins/changelog.gz jenkins.FAILED/data/usr/share/doc/jenkins/changelog.gz 5c5 < – Kohsuke Kawaguchi <kk@kohsuke.org> Thu, 14 Jul 2016 07:58:06 -0700 — > – Kohsuke Kawaguchi <kk@kohsuke.org> Wed, 06 Jul 2016 12:44:04 -0700 # Compare all files to see if there are any other changes: $ find . -type f | xargs md5sum | sed 's|\.FAILED||' | sort -k 2 26eb48db6c1b369c2033e54c5a9062e3 ./jenkins_2.7.1_all.deb f4377650305fbf43b6a0a96e19c0dc87 ./jenkins_2.7.1_all.deb c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles 935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control 935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control 21e3e095b7219ab11a1cca48f5839131 ./jenkins/control/md5sums d42e0ffd4ce78afc0fd542a92809db8c ./jenkins/control/md5sums ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst 452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm 452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm 1d32e21af04baf0f9747a9e84c592541 ./jenkins/control.tar.gz c7fbeb4ef29058e54242b5088aa52d1e ./jenkins/control.tar.gz 7a7fd1d1e607ff49a0409780472c1ece ./jenkins/data/etc/default/jenkins ffb23af22df84fa22e4c6cb02a06e460 ./jenkins/data/etc/default/jenkins 443c4e9cc5f8e24139270ed5a6cf9d22 ./jenkins/data/etc/init.d/jenkins e7d1b32adba36743f4655e2a2e1e19d8 ./jenkins/data/etc/init.d/jenkins 9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins 9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins 0833d8e1b1af7b28f28810a75ba468a0 ./jenkins/data.tar.xz b795121ef63336db3602be5e135c1cc6 ./jenkins/data.tar.xz 78b54b39b16b4f49d1588a0b39bb1220 ./jenkins/data/usr/share/doc/jenkins/changelog.gz a8d87b06e94ded342e4e1902dbab37d6 ./jenkins/data/usr/share/doc/jenkins/changelog.gz 5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright 5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright 207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war 207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war 3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary 3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary # Inspect changes in /etc/default/jenkins: $ diff jenkins/data/etc/default/jenkins jenkins.FAILED/data/etc/default/jenkins 58a59,60 > # port for AJP connector (disabled by default) > AJP_PORT=-1 66a69 > # --ajp13Port=$AJP_PORT 72c75 < JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT" — > JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT" # Inspect changes in /etc/init.d/jenkins $ diff jenkins/data/etc/init.d/jenkins jenkins.FAILED/data/etc/init.d/jenkins 132c132 < return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java|daemon)' | grep -c . ` — > return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java)' | grep -v defunct | grep -c . `
          Hide
          danielbeck Daniel Beck added a comment -

          Context, in case you're unaware: https://jenkins.io/blog/2016/07/14/2-7-1-re-release/

          TBH I'd rather have this problem than that one… still, we need to look into better handling of this case going forward. First time we ever needed something like this AFAIK.

          Show
          danielbeck Daniel Beck added a comment - Context, in case you're unaware: https://jenkins.io/blog/2016/07/14/2-7-1-re-release/ TBH I'd rather have this problem than that one… still, we need to look into better handling of this case going forward. First time we ever needed something like this AFAIK.
          Hide
          elwin Elwin Andriol added a comment -

          Thank you for that extra context, Daniel. I was not aware of that, mostly because I just started using Jenkins. It does confirm my personal findings, so at least I'm glad about that.

          I would not argue too quickly that this problem is less worse that the original problem though. Because currently, and this has apparently so far still not been fixed at all, depending on where you are located and what mirror gets selected for downloading the deb package (which appears to be a process that a user has no control over), Jenkins can simply not be installed at all (at least not through the preferred apt-get install method).

          I would like to stress that such a show stopper (because that is really what it is if the official instruction fail to work) can be very damaging to the first impressions of Jenkins to potential new users. That should not be underestimated IMHO. I happen to be the type that does not like to give up that quickly, but if a particular percentage (I do not know how many mirrors are actually out of sync) of all Debian users who recently tried to install Jenkins have come to the conclusion that this is just impossible, then it could even end up damaging the reputation of Jenkins as a whole (which is of course unrelated in reality, but that might not matter).

          From here on, I think the most important step would be to first make sure that all mirrors get synced ASAP. As for the future, I think that my suggestion to use a packaging version suffix the same way that Debian does for upstream maintained packages, might still not be such a bad idea. This way, changes in files that are not part of the upstream content can still be versioned. The convention here is to add a dash to the upstream version and then a packaging iteration number to it. So, the 2.7.1 package of Jenkins would have been generated as respectively:

          • jenkins_2.7.1-1_all.deb
          • jenkins_2.7.1-2_all.deb

          With such a naming scheme, even with mirrors out of sync, the current problem would not exist and Jenkins could still be installed (even if outdated). Of course, the issue with mirrors being out of sync is still a problem that needs to be address for its own obvious reasons.

          What so far worries me most though, is that apparently nobody is seriously taking up this problem. This problem has now existed for a week and I would not call this just a minor issue. The fact that it got unnoticed for a while is understandable (though it could be argued that proper CI would have prevented that as well). But after I filed this bug report, I still did not see much happening to have this issue swiftly resolved, by somebody with the required privileges to do so. This does in fact worry me. I would not mind putting in effort to resolve this issue, but I expect that I just don't have the required knowledge and neither the credentials for it.

          Of course, I could just have stumbled on that single or just a few obscure mirrors that are out of sync and only causing me problems (I tried installs from both The Netherlands and from Serbia), but until that is established as a fact, I personally would treat this problem as potentially big and rather serious one.

          Show
          elwin Elwin Andriol added a comment - Thank you for that extra context, Daniel. I was not aware of that, mostly because I just started using Jenkins. It does confirm my personal findings, so at least I'm glad about that. I would not argue too quickly that this problem is less worse that the original problem though. Because currently, and this has apparently so far still not been fixed at all , depending on where you are located and what mirror gets selected for downloading the deb package (which appears to be a process that a user has no control over), Jenkins can simply not be installed at all (at least not through the preferred apt-get install method). I would like to stress that such a show stopper (because that is really what it is if the official instruction fail to work) can be very damaging to the first impressions of Jenkins to potential new users. That should not be underestimated IMHO. I happen to be the type that does not like to give up that quickly, but if a particular percentage (I do not know how many mirrors are actually out of sync) of all Debian users who recently tried to install Jenkins have come to the conclusion that this is just impossible, then it could even end up damaging the reputation of Jenkins as a whole (which is of course unrelated in reality, but that might not matter). From here on, I think the most important step would be to first make sure that all mirrors get synced ASAP . As for the future, I think that my suggestion to use a packaging version suffix the same way that Debian does for upstream maintained packages, might still not be such a bad idea. This way, changes in files that are not part of the upstream content can still be versioned. The convention here is to add a dash to the upstream version and then a packaging iteration number to it. So, the 2.7.1 package of Jenkins would have been generated as respectively: jenkins_2.7.1-1_all.deb jenkins_2.7.1-2_all.deb With such a naming scheme, even with mirrors out of sync, the current problem would not exist and Jenkins could still be installed (even if outdated). Of course, the issue with mirrors being out of sync is still a problem that needs to be address for its own obvious reasons. What so far worries me most though, is that apparently nobody is seriously taking up this problem. This problem has now existed for a week and I would not call this just a minor issue. The fact that it got unnoticed for a while is understandable (though it could be argued that proper CI would have prevented that as well). But after I filed this bug report, I still did not see much happening to have this issue swiftly resolved, by somebody with the required privileges to do so. This does in fact worry me. I would not mind putting in effort to resolve this issue, but I expect that I just don't have the required knowledge and neither the credentials for it. Of course, I could just have stumbled on that single or just a few obscure mirrors that are out of sync and only causing me problems (I tried installs from both The Netherlands and from Serbia), but until that is established as a fact, I personally would treat this problem as potentially big and rather serious one.
          Hide
          rtyler R. Tyler Croy added a comment -

          Elwin Andriol the mirrors in the netherlands and serbia were the ones that were so out of date that I disabled them a few days ago (see also our mirrors status page)

          Since last night I have been provisioning, and destroying, new Jenkins masters on EC2 and I haven't yet seen any issues. This is mostly in us-west-2 so I will only be hitting the US mirrors however.

          Unfortunately since the mirrors are pull-based and operated voluntarily the best I can do is identify mirrors which have not pulled updates properly and disable them from our mirror indexing service.

          Show
          rtyler R. Tyler Croy added a comment - Elwin Andriol the mirrors in the netherlands and serbia were the ones that were so out of date that I disabled them a few days ago (see also our mirrors status page ) Since last night I have been provisioning, and destroying, new Jenkins masters on EC2 and I haven't yet seen any issues. This is mostly in us-west-2 so I will only be hitting the US mirrors however. Unfortunately since the mirrors are pull-based and operated voluntarily the best I can do is identify mirrors which have not pulled updates properly and disable them from our mirror indexing service.
          Hide
          elwin Elwin Andriol added a comment -

          I have just tried several new installs from The Netherlands and they now all appear to work.

          For the ones that did not appear to work yesterday, I may have only myself to blame for. It turns out that an apt-proxy server that I disabled/bypassed when this problem first appeared, got turned on and might have been serving the outdated packages (at least yesterday), while they no longer exist on any mirrors.

          My sincere apologies for this mistake on my end.

          Show
          elwin Elwin Andriol added a comment - I have just tried several new installs from The Netherlands and they now all appear to work. For the ones that did not appear to work yesterday, I may have only myself to blame for. It turns out that an apt-proxy server that I disabled/bypassed when this problem first appeared, got turned on and might have been serving the outdated packages (at least yesterday), while they no longer exist on any mirrors. My sincere apologies for this mistake on my end.
          Hide
          rtyler R. Tyler Croy added a comment -

          No problem, I know how frustrating problems like this can be, sometimes it seems like everything breaks at the same time.

          With our partnership with Azure we're going to be migrating from this mirrored setup to a direct CDN which will prevent issues like this in the future.

          Show
          rtyler R. Tyler Croy added a comment - No problem, I know how frustrating problems like this can be, sometimes it seems like everything breaks at the same time. With our partnership with Azure we're going to be migrating from this mirrored setup to a direct CDN which will prevent issues like this in the future.

            People

            Assignee:
            rtyler R. Tyler Croy
            Reporter:
            elwin Elwin Andriol
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: