If you try to wget that debian package URL, can you paste that console output? I'm wondering if you may have encountered some out-of-date mirrors.

I disabled two mirrors yesterday and I'm curious if that magically fixed your issue or if there is something different going on here

The Debian package file is not the problem. Both the one that is being downloaded manually and the one being downloaded through apt-get install have exactly the same checksum and install fine when done so manually through dpkg (after rename the downloaded ). The problem resides in the repository manifest/index files, not the deb package.

Apparently, I can no longer reproduce the failure, whereas earlier I could reproduce it with 100% accuracy from several different geo-locations.

So, whatever change took place, it appears to have solved the problem.

Sadly, the problem does appear to persist again. I do not know why it did work an hour ago and again stopped working.

# apt-cache show jenkins
Package: jenkins
Priority: extra
Section: devel
Installed-Size: 67318
Maintainer: Kohsuke Kawaguchi <kk@kohsuke.org>
Architecture: all
Version: 2.7.1
Replaces: hudson
Depends: daemon, adduser, procps, psmisc, net-tools, default-jre-headless (>= 2:1.7) | java7-runtime-headless
Conflicts: hudson
Filename: binary/jenkins_2.7.1_all.deb
Size: 68024840
MD5sum: f4377650305fbf43b6a0a96e19c0dc87
SHA1: 1e5725d7678a9a41d33d3ad3d1f96f28a00076c1
SHA256: 59ca57cba31c9aafdc7a3bf97e2fc46b082a5edcfd937868bc99dbd3b7caeb5e
SHA512: d5933574995d961df2457b2f3139d95c3176a4cbf71371cba4f2b5f4bf5422d52d2faaa46452bc5840d7d5a508650c9012c9879c26853b06569cec2c4f8e4de5
Description: continuous integration system
Jenkins is an application that monitors executions of repeated jobs,
such as building a software project or jobs run by cron.
Description-md5: 483e336ea11484aaa0d84b76602263f7
Homepage: http://jenkins.io/

# apt-get install jenkins
...
Get:1 http://pkg.jenkins-ci.org/debian-stable/ binary/ jenkins 2.7.1 [68.0 MB]
Fetched 68.0 MB in 2s (26.8 MB/s)
E: Failed to fetch http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb Size mismatch

# md5sum /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED
26eb48db6c1b369c2033e54c5a9062e3 /var/cache/apt/archives/partial/jenkins_2.7.1_all.deb.FAILED

_ # wget http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb _
-2016-07-20 19:55:21- http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.7.1_all.deb
Resolving pkg.jenkins-ci.org (pkg.jenkins-ci.org)... 52.202.51.185
Connecting to pkg.jenkins-ci.org (pkg.jenkins-ci.org)|52.202.51.185|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb [following]
-2016-07-20 19:55:22- http://mirrors.jenkins.io/debian-stable/jenkins_2.7.1_all.deb
Resolving mirrors.jenkins.io (mirrors.jenkins.io)... 52.202.51.185
Reusing existing connection to pkg.jenkins-ci.org:80.
HTTP request sent, awaiting response... 302 Found
Location: http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb [following]
-2016-07-20 19:55:22- http://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.7.1_all.deb
Resolving ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)... 137.226.34.46, 2a00:8a60:e012:a00::21
Connecting to ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)|137.226.34.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 68024840 (65M) [application/octet-stream]
Saving to: ‘jenkins_2.7.1_all.deb’
...
2016-07-20 19:57:31 (515 KB/s) - ‘jenkins_2.7.1_all.deb’ saved [68024840/68024840]

# md5sum jenkins_2.7.1_all.deb
f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb

This time around, it appears that the manually downloaded version has the same checksum as it should have according to the repository index/manifest, but the one that gets downloaded through apt-get install jenkins clearly is not. This could indeed indicate that there is as serious problem with mirrors serving the wrong files.

I do not know how to investigate from which mirror apt downloads its manifest/index files each time on an apt-get update and neither how I can determine which mirror is used on an apt-get install. Any assistance there would be welcome.

Additional questions:
How or why is it even possible that there are mirrors with the same file, as in with the same package name and version, with different checksums?

Usually, a particular version of a package is supposed to be immutable from the moment it gets released. Even if mirrors end up out of cync, there should never be package files with the same name (and thus also version) but with different content. Or am I missing something?

I do not know enough about the situation/infrastructure to make any informed statements, but the signs so far do give me reason to worry.

I think I have found the source of the problem, or at least a crucial part of it.

EVENTS:

• Wed, 06 Jul 2016 12:44:04 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 26eb48db6c1b369c2033e54c5a9062e3
• Thu, 14 Jul 2016 07:58:06 -0700, Kohsuke Kawaguchi build jenkins_2.7.1_all.deb with MD5 f4377650305fbf43b6a0a96e19c0dc87

CONCLUSIONS:

• Apparently, there are (at least) two different versions of the same package file (jenkins_2.7.1_all.deb) and their corresponding repository manifest/index file in circulation.
• Apparently, the deb file that gets downloaded through apt can come (and apparently often does) from a different origin than the repository manifest/index, resulting in a mismatch.
• Because two package files got changed (/etc/default/jenkins & /etc/init.d/jenkins), outside of the versioning policy for Jenkins itself, rebuiling the package does appear to have been legit.
• The usual packaging version suffix for changes outside the scope of the versioning of an upstream package, like e.g. jenkins_2.7.1-1_all.deb and jenkins_2.7.1-2_all.deb, was for some reason not used here.

SUGGESTED SOLUTION:

• A swift update of all mirrors that still contain the outdated deb package file and their corresponding Debian repository index/manifest files.
• Never again rebuild packages with exactly the same version numbers (e.g. by using an extra packaging suffix), or at least ensure a very fast and reliable mirror sync process if this can not be avoided.

REMARKS:

• The combined result of badly synched mirrors and deb packages being rebuild (even if with no changes but just different checksums because of a new timestamp), can evidently turn out disasterous.
• The current situation was probably unforseen, but especially for a CI project, it could still easily discourage a new user like me. Especially a less technical savvy user, who might only notice that they just can't get Jenkins up and running at all.

PROOF:
# verify the checksums of the two deb packages we got:
$ md5sum jenkins_*
f4377650305fbf43b6a0a96e19c0dc87 jenkins_2.7.1_all.deb
26eb48db6c1b369c2033e54c5a9062e3 jenkins_2.7.1_all.deb.FAILED

# Unpack both package files:
$ mkdir jenkins jenkins.FAILED
$ cd jenkins
$ ar -x ../jenkins_2.7.1_all.deb
$ cd ../jenkins.FAILED/
$ ar -x ../jenkins_2.7.1_all.deb.FAILED
$ cd ..

# Unpack control and data archives of 1st deb package:
$ cd jenkins
$ mkdir control data
$ cd control
$ tar zxf ../control.tar.gz
$ cd ../data
$ tar Jxf ../data.tar.xz
$ cd ../..

# Unpack control and data archives of 2nd deb package:
$ cd jenkins.FAILED
$ mkdir control data
$ cd control
$ tar zxf ../control.tar.gz
$ cd ../data
$ tar Jxf ../data.tar.xz
$ cd ../..

# Inspect 1st package control files:
$ cd jenkins/control
$ md5sum *
c26b730354d8ae1c669c64a7420cff2d conffiles
935454337cb6099d6bc475f33f0b08d4 control
d42e0ffd4ce78afc0fd542a92809db8c md5sums
ddd85780e84b8cf07dc50aab8ef35fc4 postinst
452b5d5406a95b199ed4eb72a676fdd9 postrm
c728a327fc7c2ad8699043a4c0ea1e94 prerm
$ cd ../..

# Inspect 2nd package control files:
$ cd jenkins.FAILED/control
$ md5sum *
c26b730354d8ae1c669c64a7420cff2d conffiles
935454337cb6099d6bc475f33f0b08d4 control
21e3e095b7219ab11a1cca48f5839131 md5sums
ddd85780e84b8cf07dc50aab8ef35fc4 postinst
452b5d5406a95b199ed4eb72a676fdd9 postrm
c728a327fc7c2ad8699043a4c0ea1e94 prerm
$ cd ../..

# Compare the entries in both md5sums files:
$ find . -name md5sums | xargs cat | sort -k 2
78b54b39b16b4f49d1588a0b39bb1220 usr/share/doc/jenkins/changelog.gz
a8d87b06e94ded342e4e1902dbab37d6 usr/share/doc/jenkins/changelog.gz
5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright
5ea3b174918ea0373267db270b24f0da usr/share/doc/jenkins/copyright
207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war
207e5070822fa2a38ccf5e2c4c4139e9 usr/share/jenkins/jenkins.war

# Please note that jenkins.war is identical in both instances!

# Inspect the differences in both changelog.gz files:
$ zdiff jenkins/data/usr/share/doc/jenkins/changelog.gz jenkins.FAILED/data/usr/share/doc/jenkins/changelog.gz
5c5
< – Kohsuke Kawaguchi <kk@kohsuke.org> Thu, 14 Jul 2016 07:58:06 -0700
—
> – Kohsuke Kawaguchi <kk@kohsuke.org> Wed, 06 Jul 2016 12:44:04 -0700

# Compare all files to see if there are any other changes:
$ find . -type f | xargs md5sum | sed 's|\.FAILED||' | sort -k 2
26eb48db6c1b369c2033e54c5a9062e3 ./jenkins_2.7.1_all.deb
f4377650305fbf43b6a0a96e19c0dc87 ./jenkins_2.7.1_all.deb
c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles
c26b730354d8ae1c669c64a7420cff2d ./jenkins/control/conffiles
935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control
935454337cb6099d6bc475f33f0b08d4 ./jenkins/control/control
21e3e095b7219ab11a1cca48f5839131 ./jenkins/control/md5sums
d42e0ffd4ce78afc0fd542a92809db8c ./jenkins/control/md5sums
ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst
ddd85780e84b8cf07dc50aab8ef35fc4 ./jenkins/control/postinst
452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm
452b5d5406a95b199ed4eb72a676fdd9 ./jenkins/control/postrm
c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm
c728a327fc7c2ad8699043a4c0ea1e94 ./jenkins/control/prerm
1d32e21af04baf0f9747a9e84c592541 ./jenkins/control.tar.gz
c7fbeb4ef29058e54242b5088aa52d1e ./jenkins/control.tar.gz
7a7fd1d1e607ff49a0409780472c1ece ./jenkins/data/etc/default/jenkins
ffb23af22df84fa22e4c6cb02a06e460 ./jenkins/data/etc/default/jenkins
443c4e9cc5f8e24139270ed5a6cf9d22 ./jenkins/data/etc/init.d/jenkins
e7d1b32adba36743f4655e2a2e1e19d8 ./jenkins/data/etc/init.d/jenkins
9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins
9ccf66fc7b41f580d6c701e42d1bb794 ./jenkins/data/etc/logrotate.d/jenkins
0833d8e1b1af7b28f28810a75ba468a0 ./jenkins/data.tar.xz
b795121ef63336db3602be5e135c1cc6 ./jenkins/data.tar.xz
78b54b39b16b4f49d1588a0b39bb1220 ./jenkins/data/usr/share/doc/jenkins/changelog.gz
a8d87b06e94ded342e4e1902dbab37d6 ./jenkins/data/usr/share/doc/jenkins/changelog.gz
5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright
5ea3b174918ea0373267db270b24f0da ./jenkins/data/usr/share/doc/jenkins/copyright
207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war
207e5070822fa2a38ccf5e2c4c4139e9 ./jenkins/data/usr/share/jenkins/jenkins.war
3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary
3cf918272ffa5de195752d73f3da3e5e ./jenkins/debian-binary

# Inspect changes in /etc/default/jenkins:
$ diff jenkins/data/etc/default/jenkins jenkins.FAILED/data/etc/default/jenkins
58a59,60
> # port for AJP connector (disabled by default)
> AJP_PORT=-1
66a69
> # --ajp13Port=$AJP_PORT
72c75
< JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"
—
> JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"

# Inspect changes in /etc/init.d/jenkins
$ diff jenkins/data/etc/init.d/jenkins jenkins.FAILED/data/etc/init.d/jenkins
132c132
< return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java|daemon)' | grep -c . `
—
> return `ps -U $JENKINS_USER --no-headers -f | egrep -e '(java)' | grep -v defunct | grep -c . `

Context, in case you're unaware: https://jenkins.io/blog/2016/07/14/2-7-1-re-release/

TBH I'd rather have this problem than that one… still, we need to look into better handling of this case going forward. First time we ever needed something like this AFAIK.

Thank you for that extra context, Daniel. I was not aware of that, mostly because I just started using Jenkins. It does confirm my personal findings, so at least I'm glad about that.

I would not argue too quickly that this problem is less worse that the original problem though. Because currently, and this has apparently so far still not been fixed at all, depending on where you are located and what mirror gets selected for downloading the deb package (which appears to be a process that a user has no control over), Jenkins can simply not be installed at all (at least not through the preferred apt-get install method).

I would like to stress that such a show stopper (because that is really what it is if the official instruction fail to work) can be very damaging to the first impressions of Jenkins to potential new users. That should not be underestimated IMHO. I happen to be the type that does not like to give up that quickly, but if a particular percentage (I do not know how many mirrors are actually out of sync) of all Debian users who recently tried to install Jenkins have come to the conclusion that this is just impossible, then it could even end up damaging the reputation of Jenkins as a whole (which is of course unrelated in reality, but that might not matter).

From here on, I think the most important step would be to first make sure that all mirrors get synced ASAP. As for the future, I think that my suggestion to use a packaging version suffix the same way that Debian does for upstream maintained packages, might still not be such a bad idea. This way, changes in files that are not part of the upstream content can still be versioned. The convention here is to add a dash to the upstream version and then a packaging iteration number to it. So, the 2.7.1 package of Jenkins would have been generated as respectively:

• jenkins_2.7.1-1_all.deb
• jenkins_2.7.1-2_all.deb

With such a naming scheme, even with mirrors out of sync, the current problem would not exist and Jenkins could still be installed (even if outdated). Of course, the issue with mirrors being out of sync is still a problem that needs to be address for its own obvious reasons.

What so far worries me most though, is that apparently nobody is seriously taking up this problem. This problem has now existed for a week and I would not call this just a minor issue. The fact that it got unnoticed for a while is understandable (though it could be argued that proper CI would have prevented that as well). But after I filed this bug report, I still did not see much happening to have this issue swiftly resolved, by somebody with the required privileges to do so. This does in fact worry me. I would not mind putting in effort to resolve this issue, but I expect that I just don't have the required knowledge and neither the credentials for it.

Of course, I could just have stumbled on that single or just a few obscure mirrors that are out of sync and only causing me problems (I tried installs from both The Netherlands and from Serbia), but until that is established as a fact, I personally would treat this problem as potentially big and rather serious one.

elwin the mirrors in the netherlands and serbia were the ones that were so out of date that I disabled them a few days ago (see also our mirrors status page)

Since last night I have been provisioning, and destroying, new Jenkins masters on EC2 and I haven't yet seen any issues. This is mostly in us-west-2 so I will only be hitting the US mirrors however.

Unfortunately since the mirrors are pull-based and operated voluntarily the best I can do is identify mirrors which have not pulled updates properly and disable them from our mirror indexing service.

I have just tried several new installs from The Netherlands and they now all appear to work.

For the ones that did not appear to work yesterday, I may have only myself to blame for. It turns out that an apt-proxy server that I disabled/bypassed when this problem first appeared, got turned on and might have been serving the outdated packages (at least yesterday), while they no longer exist on any mirrors.

My sincere apologies for this mistake on my end.

No problem, I know how frustrating problems like this can be, sometimes it seems like everything breaks at the same time.

With our partnership with Azure we're going to be migrating from this mirrored setup to a direct CDN which will prevent issues like this in the future.