-
Bug
-
Resolution: Fixed
-
Major
-
Powered by SuggestiMate
The jenkins-1.565-1.1 RPM contains a post-install script with performs the following:
# Ensure the right ownership on files . /etc/sysconfig/jenkins chown -R ${JENKINS_USER:-jenkins} /var/log/jenkins chown -R ${JENKINS_USER:-jenkins} ${JENKINS_HOME:-/var/lib/jenkins}
These chowns can take a very long time on systems with lots of files. A couple of our Jenkins boxes have thousands of files in a the /var/lib/jenkins/workspace directory (which itself is a network mount), which means this chown takes many minutes to complete (like ~15 minutes) and actually changes the ownership of no files. This holds up package deployment unnecessarily.
I propose that either these chowns be removed entirely (the RPM spec sets the ownership of things the RPM puts down), or better yet that they're restricted to specific files.
- is related to
-
JENKINS-12231 RPM upgrade does not honor JENKINS_USER, and always resets files ownership to "jenkins"
-
- Resolved
-
-
JENKINS-26460 RPM upgrade/install should adapt ownership of /var/cache/jenkins
-
- Resolved
-
- links to
[JENKINS-23273] RPM post-install should not do a "chown -R ${JENKINS_USER:-jenkins} ${JENKINS_HOME:-/var/lib/jenkins}"
@Daniel Beck:
I think you are correct. Sorry for my misunderstanding.
~tommy
Updating jenkins on our environment took more than a hour because of this behaviour! This is very frustrating because everytime I need to explain the long downtime...
I disagree that the behaviour should be removed. In a setting where Jenkins is run by a different user than "jenkins", this behaviour is absolutely required.
Maybe it would make sense to first check whether any of the files need to be changed?
Can somebody check how fast this command would run:
find /var/lib/jenkins -not -user jenkins
Then the code can be changed to something like
if find ${JENKINS_HOME:-/var/lib/jenkins} -not -user ${JENKINS_USER:-jenkins}; then chown -R ${JENKINS_USER:-jenkins} ${JENKINS_HOME:-/var/lib/jenkins} fi if find /var/log/jenkins -not -user ${JENKINS_USER:-jenkins}; then chown -R ${JENKINS_USER:-jenkins} /var/log/jenkins fi
I disagree that the behaviour should be removed. In a setting where Jenkins is run by a different user than "jenkins", this behaviour is absolutely required.
Doesn't Jenkins fail to boot if it cannot write to its home directory?
A sane solution would be printing a warning if Jenkins is configured for a different user than owns the directory, or only chown-R'ing if the owner of JENKINS_HOME is different than expected – but not unconditionally. If only some files below JENKINS_HOME are owned by a different user, it's either intentional, or you broke it in some drunken craze, and I don't think the installer needs to fix it in either case.
If you are going to do the `find` you might as well `-exec` the chown instead of going back to a "chown -R" ... the only way that would make sense is if the find would stop immediately upon finding its first "match". You are still traversing the entire filesystem with potentially thousands of files (in our case), causing undue delay. Our workaround for this issue is to stop jenkins and umount the `/var/lib/jenkins` filesystem prior to patching. Otherwise it takes a very long time.
I like the idea of changing it to a "warning" message, perhaps wrapped in big square of asterisks or something :-/
The package has a list of files that it would be installing. Perhaps the "correct" thing to do would be to run a for loop over that list of files to chown just those files to $JENKINS_USER, only if the $JENKINS_USER is not "jenkins" ... that way it won't traverse the entire filesystem, and it will only do anything if you are not running jenkins as jenkins
~tommy
Tommy's point about only chown'ing files that are in the RPM seems like a bright idea to me. I'm not sure if you can do an "rpm -qvl" during a post-install script though (has the package been installed enough to get this during post-install?). However, this would be a good solution.
In the short term, Tommy's other point about only attempting any sort of chown if the JENKINS_USER is not "jenkins" also makes a lot of sense, and would be easier to implement. It still penalises people who don't run as 'jenkins', but I'm guessing they're a pretty small minority. It'd help everyone else while working out the better solution in the future.
Cheers,
...Ralph
Tommy's point about only chown'ing files that are in the RPM seems like a bright idea to me.
None of the files in /var/lib/jenkins are created during installation (for some reason, the plugins/ folder gets created, but nothing else).
only attempting any sort of chown if the JENKINS_USER is not "jenkins" also makes a lot of sense, and would be easier to implement.
Breaks on any system where the user used to be custom but was reset to jenkins later. I don't see any benefit either.
https://github.com/jenkinsci/jenkins/pull/1537
This exposes an /etc/sysconfig variable to skip the chown process. It seems to work for me, but this is my very first RPM package update so some eyes on this would be appreciated.
@jieryn - that's a nice solution. I haven't tested the change directly, but it looks okay to me. When it hits an RPM, I'll try it on a few of our CI boxes and see how it goes.
The trouble for non-jenkins users is worse than it looks:
if the chown is missing, upgrading the rpm will change the ownership of /var/*/jenkins to "jenkins". So, if you had it set up to work with another user before, it will be broken afterwards. So, unless the post-install script does change the ownership explicitely, every rpm upgrade breaks Jenkins for such users.
I think that jieryn's olution is probably a good one. Please include "/var/cache/jenkins" into the list of updated dirs!
As requested by Olaf Lenz:
# time find /var/lib/jenkins -not -user jenkins
real 50m15.587s
user 0m6.984s
sys 0m55.076s
So this is a NoGo!
What's the state on this issue? It is still a pain to update Jenkins on CentOS as the whole System is down for more than a hour now!
As a workaround, I login to another shell and kill the chown underneath the post-install script. That is the last step in the script anyway and it lets the upgrade finish. Of course, I just use the default jenkins user, so it doesn't really matter if the chown half-completes.
As a workaround...
We actually mount a filesystem on `/var/lib/jenkins` and umount it when doing updates. This actually happens because we upgrade jenkins on the inactive cluster node, but maybe its an option for others.
I still vote for only running chown if the JENKINS_USER is not "jenkins". If someone is "changing" from a non standard user to jenkins, then its on them to deal with the chown themselves. Quite frankly, anyone using a non-standard user to run jenkins while trying to use RPM jenkins and the "rpm provided" directories deserve what trouble they have.
~tommy
Code changed in jenkins
User: jieryn
Path:
rpm/build/SOURCES/jenkins.sysconfig.in
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/1e4744ef8ff3cf0f16ea03aabf4afe2478a3e32e
Log:
[FIXED JENKINS-23273] /etc/sysconfig option to skip (off default) long chown
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
rpm/build/SOURCES/jenkins.sysconfig.in
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/58084e56eee77341e580474a5dd1176a2e0148fa
Log:
Merge pull request #2 from jieryn/JENKINS-23273
[FIXED JENKINS-23273] /etc/sysconfig option to skip (off default) lon…
Compare: https://github.com/jenkinsci/packaging/compare/dab31b61b6e6...58084e56eee7
jieryn's PR has been merged, but it's a temporary pain reliever and not a fix. I discussed this in here but my suggestion is to do chmod non-recursively to just touch files & directories managed by RPM. IIUC, this is what danielbeck is proposing.
When that change is made, this bug should be marked as fixed.
Until that day, those who are affected by this issue can use JENKINS_INSTALL_SKIP_CHOWN switch to mitigate the pain. The switch should be considered temporary — it will be removed when the proper fix is made.
i'm currently updating to jenkins-1.638-1.1.noarch.rpm, and the chown is killing me... our jenkins root dir is a tick under 3T and i've been waiting for over an hour for the chown to finish.
perhaps instead of blindly chowning the entire jenkins root, just do everything except the jobs/ dir?
So as best as I can tell from my experience and some research, there's an assumption that RPMs don't support custom users for ownership of files, etc - or rather, they only support the user(s) defined in the RPM spec file, not users that could be changed dynamically post-install. So I have to think this is a won't-fix.
The NoGo-behaviour is still existing and as there seems to be no solution, we still use the hard workaround: If the update is running, open another console, find the chown process and kill it the hard way using kill -9 <pid>. The chown makes absolutely no sense on boxes, where jenkins is the one and only account.
Yeah, I misworded - I'd say the real fix here is to get rid of the chown completely, or switch it to only run when JENKINS_USER does not equal "jenkins".
I'm not sure what the problem is here.. just add the /etc/sysconfig/jenkins property named JENKINS_INSTALL_SKIP_CHOWN="true" and the chown operation will be skipped as advertised. This keeps the old behavior and has a trivial fix for those that don't want to be impacted by the chown.
Let's not lose our sense of history here. This entire chown process was put into place to migrate old hudson installs to jenkins. That's the reason it exists. I think migrations from pre-fork hudson are probably almost nonexistant at this point, but....
Hm, seems I missed this option completely! Will try this asap, thx for the hint.
This keeps the old behavior and has a trivial fix for those that don't want to be impacted by the chown.
Not trying to resurrect the old discussion, but requiring every user to customize what's essentially an invisible option so that updates finish in a reasonable time once they start using Jenkins seriously doesn't seem like a good long-term situation. Out of the box behavior should work for the 80% of users accepting the defaults, not the 20% (and I suspect I'm overly generous there) who tinker anyway (or who haven't updated their software in five years).
So while I haven't gotten around to fixing this, I still think this needs fixing.
As for the argument that we should support migration from hudson: the whole mechanism with the JENKINS_USER is somewhat broken, anyway.
To be able to use it, you first have to create /etc/sysconfig/jenkins, as the chown happens in the RPM postinstall script.
If you edit the file and change the JENKINS_USER only after the installation and then try to start jenkins (e.g. via sudo /etc/init.d/jenkins start), it won't start, and it will not give a useful error message, as it cannot write its log or anything, as the paths are not chowned.
So, from my POV, the whole mechanism is useful only for very few people, anyway, if it was useful at all. So I suggest: remove the mechanism, it will hurt nobody.
Since there don't seem to be an overall agreement on how to set the default I would propose the following solution (untested pseudocode):
# Ensure the right ownership on files . /etc/sysconfig/jenkins JENKINS_CHOWN_USER=${JENKINS_USER:-jenkins} JENKINS_CHOWN_HOME=${JENKINS_HOME:-/var/lib/jenkins} if [ ! -e /var/lib/jenkins/.ran-chown-for-${JENKINS_CHOWN_USER} ]; then # in case there where different users before rm /var/lib/jenkins/.ran-chown-for-* chown -R ${JENKINS_CHOWN_USER} /var/log/jenkins chown -R ${JENKINS_CHOWN_USER} ${JENKINS_CHOWN_HOME} echo $(date) >> /var/lib/jenkins/.chowned-to-${JENKINS_CHOWN_USER} fi
volker I like the approach of adding a marker file here, but have an even better idea: by default we chown -R (maintaining old behavior), but only if the top-level directory is not owned by the Jenkins user. This way, we only chown where needed (improving performance tremendously), and if the JENKINS_INSTALL_SKIP_CHOWN option is set, we never chown.
If a chown is performed, a marker file is created indicating the original user/group listing.
by default we chown -R (maintaining old behavior), but only if the top-level directory is not owned by the Jenkins user
svanoort, I can confirm, that is not true. JENKINS_HOME directory has jenkins owner and chown -R still runs on update.
Can confirm I have this issue also.
With yum update I can see:
root 8466 1.4 0.0 110128 992 pts/0 R+ 09:32 0:42 chown -R jenkins /var/lib/jenkins
It takes a very long time to run
CentOS Linux release 7.2.1511
svanoort how could we move forward here ? It seems to be a pain for all RedHat users.
Oops, shouldn't still be assigned to me. Would be a good candidate for someone in community to pick up and move forward.
I have linked a PR to solve this, any comments, suggestions or bugs are welcome
Code changed in jenkins
User: Raul
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/02d2c3d1eed76df9027af59a33923c7bef6a7734
Log:
JENKINS-23273 Change ownership only if needed
- Now /var/cache%
{name} /var/log/%{name}
and workdir files are chowned
only if parent folder is not already owned by appropriate user - Also added sticky bit to that folders so only the appropriate user can
create files there
Code changed in jenkins
User: Raul
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/6cbcaf9da22f1a0481d7baec069a2935e7e28a14
Log:
JENKINS-23273 Remove hudson default update site before migrating
If it is not removed a message may appear saying that a new version of
jenkins is available with a Hudson version and download url
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/5083f2ea1fbae613870cee96df1ce90f891b7202
Log:
JENKINS-23273 Removed the hudson -> jenkins migration
It does not make sense nowadays, and if we remove that we do not need to
chown the jenkins installation folders because they are created with the proper user
assigment
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/679fbfe0898aeaa142c2ef0b23f9561d788d5381
Log:
JENKINS-23273 Change ownership only if needed
- Now /var/cache%
{name} /var/log/%{name}
and workdir files are chowned
only if parent folder is not already owned by appropriate user - Also added sticky bit to that folders so only the appropriate user can
create files there
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/d7b16bc1f25c534572ef238a885bb28571455b99
Log:
JENKINS-23273 Revert sticky bit to not reopen JENKINS-12231 like
situations
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/15a2e6971b631601b55f77884827fdde19c33ad2
Log:
JENKINS-23273 Make sure instance is always running after upgrade
Changed the chown logic so is only skipped if the configured user and
the owner are the same of the name param (default jenkins)
This way the chown is skipped in the majority of the unneeded cases and
installations while we make sure upgrades still work
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/bd92f0a356b9aa3eac35d54e154365b05dc28f8b
Log:
JENKINS-23273 Update comments
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/d6f9ba991306e1b5f5e87315debd7dc0c66fc5b1
Log:
JENKINS-23273 Do a recursive chown only when needed
- When the owner of files and configured JENKINS_USER are not the same we
need a recursive chown - If the owner and the configured JENKINS_USER are the same but not the
default chown only the folders - In the default case (owner, JENKINS_USER and %name being the same) do
nothing
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/5ee2ae85ebc6daf89235e28965fcae65d5dd2f79
Log:
JENKINS-23273 Addressing feedback
- Added logging
- Removed unneeded touch
- Used stat --format=%U instead of ls -ld
- Created a bash function to avoid duplication
Code changed in jenkins
User: Vojtěch-Zweibrücken-Šafařík
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/d6732acca6e360e6a45c3a05d3db175c04697fb5
Log:
JENKINS-23273 Fix typo
Code changed in jenkins
User: Raul Arabaolaza
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/322d0ee11dd26e369c4590fb389b4a411ed63840
Log:
JENKINS-23273 Make some variables space safe by quoting them
Code changed in jenkins
User: Sam Van Oort
Path:
rpm/build/SPECS/jenkins.spec
http://jenkins-ci.org/commit/packaging/eb7579d4d39466ade20f0405ce1b0e2a28d764de
Log:
Merge pull request #87 from raul-arabaolaza/JENKINS-23273
JENKINS-23273 RPM post-install should not do a "chown R ${JENKINS_USER:-jenkins} ${JENKINS_HOME:/var/lib/jenkins}"
Compare: https://github.com/jenkinsci/packaging/compare/2206ee9f38e9...eb7579d4d394
benh57 It was (around 2.53). I will make sure changelog is updated. Packaging is asynchronous, so it is hard to track changes in the core. I also want to start a developer ML thread for it
Code changed in jenkins
User: Oleg Nenashev
Path:
content/_data/changelogs/weekly.yml
http://jenkins-ci.org/commit/jenkins.io/bd7e8598e4b85d84286ded5f2e6d71d08d8531ee
Log:
Add missing changelog entries for JENKINS-23273 and JENKINS-40693
Code changed in jenkins
User: Daniel Beck
Path:
content/_data/changelogs/weekly.yml
http://jenkins-ci.org/commit/jenkins.io/d6c1369d8ebcdff4a9f126097ecc6a7c30ea1d86
Log:
Merge pull request #921 from oleg-nenashev/changelog/missing-entries
Add missing changelog entries for JENKINS-23273 and JENKINS-40693
Compare: https://github.com/jenkins-infra/jenkins.io/compare/14212ab18dfd...d6c1369d8ebc
Code changed in jenkins
User: Daniel Beck
Path:
content/_data/changelogs/lts.yml
content/_data/changelogs/weekly.yml
content/doc/upgrade-guide/2.60.adoc
http://jenkins-ci.org/commit/jenkins.io/800435b6892e68f259f9c6ecbc54e79b1798c656
Log:
Mention JENKINS-23273, minor upgrade guide changes
tommythekid:
Are you sure about the symbolic links? My man chmod states (emphasis mine):