-
Improvement
-
Resolution: Fixed
-
Critical
-
None
-
Powered by SuggestiMate -
2.335, 2.332.1
Currently debian packages (and presumably redhat/suse/etc) ship with old style SysV init.d file.
Most distributions are now systemd based, and to be able to make use of it's additional features it would be useful have /lib/systemd/system/jenkins.service file.
This would allow features like systemd's restart functionality to work (useful when Jenkins occasionally crashes, etc). See discussion around this here: http://serverfault.com/questions/760552/how-to-use-automatic-restart-with-jenkins-on-systemd
Details of how provide both systemd unit file and SysV init file are detailed here:
https://wiki.debian.org/Teams/pkg-systemd/Packaging
This should be applicable to all distro, and systemd unit most likely the same across them all, as it's much more standardised than each distros SysV init infrastructure.
- is related to
-
JENKINS-65809 Jenkins fails to restart after plugin updates on Debian 11 (bullseye)
-
- Closed
-
-
JENKINS-67404 Jenkins Debian/Ubuntu Start Timeout Failure Mode
-
- Closed
-
- relates to
-
JENKINS-68156 Service (systemd) change to type=simple
-
- Closed
-
-
JENKINS-67405 Newbie Friendly Setup Improvement jenkinsci/packaging
-
- Resolved
-
- links to
[JENKINS-41218] Provide native systemd unit
Hello,
What can I/we do to have this fix applied? Systemd has become the defacto service manager across major / all distributions.
There are new releases in Debian and RHEL that use systemd. There are great benefits to have apps managed by systemd.
It would be great to have this upstream. Even if it is disabled and we need to `systemctl enable ` it manually.
Looking forward for your feedback.
Thanks
The above is for running Jenkins in Tomcat, isn't it? It also seems to be RedHat specific (/etc/sysconfig). When running apps in Tomcat, one usually just need to start Tomcat itself, which in turn takes care of starting all the webapps deployed to it. So setting up a "tomcat@.service" for single webapps doesn't seem to be necessary at all (and also feels wrong given that this would mean to start one Tomcat instance for each webapp configured this way).
OTOH, this ticket is about a "native" systemd unit, which I think means: One that starts Jenkins standalone, just like the current sys-v init script does.
dhs
The above is for running Jenkins in Tomcat, isn't it?
Yes. The example provided based on [Advanced Configuration - Multiple Tomcat Instances|https://tomcat.apache.org/tomcat-8.0-doc/RUNNING.txt] - e.g. single installation and "n" - Tomcat Server instances, which allow to run NOT multiple applications within single server(which is also possible by default), but multiple Tomcat Server instances.
This is widely used technique when you desire to split up different application servers but share binaries between them.
Unit template "tomcat@.service" follow the described flow above:
- Determines current default Tomcat binaries(configured via alternatives) directory and expose required variable
ExecStartPre=/bin/bash -c 'systemctl set-environment CATALINA_HOME=$(readlink -f /usr/bin/tomcat)' - Retrieves server instance configuration(RedHat or Debian makes no difference here, just replace 'sysconfig' to 'default' on deb packaging stage)
EnvironmentFile=-/etc/sysconfig/%i - Starts dedicated Tomcat Instance for Jenkins
This kind of configuration provides flexibility on Servlet container configuration, protocols configuration used for proxy communication, simple server and java versions updates, and simplicity for multiple server configuration(dev/int/stage/prod instances) when you are on bare-metal/virtual machines.
So setting up a "tomcat@.service" for single webapps doesn't seem to be necessary at all
For the configuration when the single app should have its own server.
One that starts Jenkins standalone, just like the current sys-v init script does.
I'd left this advanced java app deployment configuration here for someone who tried to find systemd solution from the maintainers. Feel free to switch this unit to the internal winstone(jetty) servlet container used at Jenkins war/jar binaries.
Regards.
axmetishe, but still, this way of starting Jenkins should not be part of the Jenkins package, but the Tomcat one, right? Also, the "EnvironmentFile" entry is there for providing some kind of backwards compatibility with system v. In a pure systemd setup they shouldn't be used anymore as environment variables can be added directly via "Environment" entries. Customizations should then go into /etc/systemd/system/myservice.service.d/*.conf. This completely eliminates the need for ANY distribution specific files.
I'll try to write a native service file for standalone Jenkins, I already have one for starting swarm agents which I can use as a starting point...
dhs
not be part of the Jenkins package, but the Tomcat one, right?
not Jenkins, nor Tomcat - environment specific configuration
Also, the "EnvironmentFile" entry is there for providing some kind of backwards compatibility with system v
wrong - this is not a backward compatibility level, but conscious way to pass basic deployment configuration instead of large unit file with tons of Environment definitions which will not be updated from place to place
In a pure systemd setup they shouldn't be used anymore as environment variables
wrong again - it's a just two different methods for environment configuration for 'pure' systemd setup
Customizations should then go into /etc/systemd/system/myservice.service.d/*.conf
customization will still be applied even using EnvironmentFile which is related to general configuration - https://www.freedesktop.org/software/systemd/man/systemd.exec.html
This completely eliminates the need for ANY distribution specific files
You will still have to do some customization with deb/rpm packaging at pre-inst/post-inst for example and string replacement will be the lesser problem
I'll try to write a native service file for standalone Jenkins, I already have one for starting swarm agents which I can use as a starting point
Glad to hear that.
Regards.
HI,
dhs : You are correct, I am talking about the standalone application.
axmetishe : Jenkins is not using tomcat installation from Debian. It's running on embedded Tomcat and it should stay like this. It's fine. I believe creating dependencies between packages is a bad practice that complicates a lot of things.
I am running Jenkins 2.222 from pkg.jenkins.io and it is running as a standalone application on Debian:
```
jenkins 1397 0.0 0.0 20244 172 ? S 07:35 0:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid – /usr/bin/java -Djava.awt.headless=true -Djava.util.logging=DEBUG -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
jenkins 1398 4.9 14.2 4762984 1138044 ? Sl 07:35 9:42 /usr/bin/java -Djava.awt.headless=true -Djava.util.logging=DEBUG -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
```
I am using Debian. I do believe we should keep things simple and leave out detection and automation stuff. We should avoid assumptions.
Having the following should solve the problem easily IMO
- systemd unit delivered as part of the package
- documentation that the user can enable it
- a `systemctl enable` command with the path see https://www.freedesktop.org/software/systemd/man/systemctl.html
The behavior of the package does not even have to change IMO. We can add the files, allow people to test them out and change the defaults a few months later.
Both Debian and RHEL systemd files can be provided in case one is not enough.
No fiddling with alternatives or other stuff.
As a final note:
I am willing to provide patches however I would like to do the work if it has a chance to get merged. Otherwise it makes no sense to spend time on this.
Waiting for an ok on the solution.
Here you are:
[Unit] Description=Standalone Jenkins Master server Documentation=https://www.jenkins.io/doc Wants=network-online.target After=network-online.target [Service] User=jenkins Group=jenkins Environment=HTTP_PORT=8080 Environment=JAVA_ARGS=-Djava.awt.headless=true Environment=JENKINS_HOME=/var/lib/jenkins Environment=JENKINS_WAR=/usr/share/jenkins/jenkins.war Environment=LISTEN_ADDRESS=0.0.0.0 Environment=WEBROOT=/var/cache/jenkins/war WorkingDirectory=/var/lib/jenkins LimitNOFILE=8192 ExecStart=/usr/bin/java ${JAVA_ARGS} -jar ${JENKINS_WAR} --webroot=${WEBROOT} --httpPort=${HTTP_PORT} --httpListenAddress=${LISTEN_ADDRESS} [Install] WantedBy=multi-user.target
An example overlay file (to overwrite the defaults) could look like this:
[Service] Environment=JENKINS_HOME=/srv/jenkins Environment=LISTEN_ADDRESS=127.0.0.1
and should be placed into /etc/systemd/system/jenkins.service.d/ as *.conf by the system admin.
I also use this for some services to grant them access to the OpenAFS network filesystem, which requires Kerberos authentication, by overwriting the ExecStart= line. Something that's not possible with an environment file in /etc/default.
Thanks dhs, I hope the files get added to the jenkins package and are distributed with it so people can use them instead of the scripts.
After some time, it can be installed by default instead of the scripts.
You're welcome. Forgot to mention: While this works for me, it doesn't mean it works for others, too. But it should make a good starting point (and should as well be distribution independent).
Good job dhs, thank you.
By the way, it will be better to set dependency on network-online.target in order to be sure that we can create network socket.
Since more than a year has passed and 2.303.1 STILL ships a SysV init script instead of a systemd service file, I'd like to ask what's the status here? All the major distributions default to systemd nowadays, so it's about time...
This is now a more pressing issue with the 2.303.3 LTS update. This added daemonize as a package dependency, which is not available in any RHEL package repository. It is, however, available in Fedora's EPEL package repository, which can be enabled on any RHEL system.
Jenkins itself is only available as an RPM through OpenShift in RHEL (link requires a Red Hat account)- one typically does not RPM install jenkins on a RHEL system, but instead deploys the Jenkins container image that Red Hat packages for OpenShift. My team is responsible for this packaging, and would prefer to not add daemonize to our set of packages to install. Ideally we can work with the packaging team to make this happen.
Related issue JENKINS-65809 where Jenkins can't restart anymore on Debian 11 due to a change to the daemon utility.
Changing to a systemd service file would workaround the issue.
Added a link.
BTW: I'd rather call this a "fix" instead of "workaround"
I find Debian daemon net-tools package dependencies distressing since AFAICT their use is limited to /etc/init.d/jenkins script alone.
I suggest replacing net-tools "netstat --listen --numeric-ports" command with package iproute2 command "ss --listening --numeric --tcp" instead.
I suggest replacing various daemon commands "daemon $DAEMON_ARGS ..." with package dpkg command "start-stop-daemon" equivalents.
The above presumes /etc/init.d/jenkins script is retained without daemon net-tools package dependencies.
I prefer and suggest providing systemd unit jenkins.service to replace /etc/init.d/jenkins.
I volunteer to promptly implement whatever consensus emerges.
Happy to see a pull request that:
- Provides a systemd package for deb
- Provides an upgrade path for existing users of deb
- Provides a documentation pull request to www.jenkins.io describing the upgrade
- Declares the distributions where the installation was tested and where the upgrade was tested. Should include at least Debian Bullseye, Debian Buster, Ubuntu 18.04, and Ubuntu 20.04
Happy to see a pull request that:
- Provides a systemd package for Red Hat rpm
- Provides an upgrade path for existing users of Red Hat rpm
- Provides a documentation pull request to www.jenkins.io describing the upgrade
- Declares the distributions where the installation was tested and where the upgrade was tested. Should include at least CentOS 7, Red Hat Enterprise Linux 7, Amazon Linux 2, Red Hat Enterprise Linux 8
Happy to see a pull request that:
- Provides a systemd package for SUSE rpm
- Provides an upgrade path for existing users of SUSE rpm
- Provides a documentation pull request to www.jenkins.io describing the upgrade
- Declares the distributions where the installation was tested and where the upgrade was tested. Should include at least OpenSUSE 15.3, SUSE Enterprise Linux 15.3, OpenSUSE Leap
I'm a happy user of the packaging and a willing tester of the packaging. I have access to many of those configurations. I'm not an expert in packaging scripts. I'm not an expert in systemd.
I'm accustomed to lots of angry, hostile feedback every time there is a packaging change. It is quite common for people that do not otherwise contribute to the Jenkins project to decide that they need to harshly rebuke the maintainers when they change the packaging or make a packaging mistake. Brace yourself for that type of feedback...
markewaite, given that ALL the distributions you mention above have long switched to systemd as their (default, Debian also supports others) init system, I doubt anyone (except Devuan users ) will get angry on you if Jenkins provides a proper way to start the service that fits so much better.
As for a pull request: There's a working systemd service configuration attached to this ticket already.
markewaite said, "I have access to many of those configurations." May I ask, are those configurations project or personal?
I see setting up Jenkins development environment and test configurations are a hurdle. Are shared platforms available?
conrad_t_pino the test configurations are hosted on Google Cloud Platform and are donated by my employer. I'm not allowed to grant access to others.
Shared platforms are available to run ci.jenkins.io Jenkins jobs on amd64, arm64, ppc64le, and s390x, but they do not provide root access. We run basic installation acceptance tests there today with Docker images for amd64, arm64, and s390x.
I could share access to platforms in my home network, but they are more limited than the GCP agents I'm using.
Is https://github.com/jenkinsci/jenkins/blob/master/CONTRIBUTING.md the best reference to launch development environment to work on this issue?
I think that https://github.com/jenkinsci/packaging/blob/master/README.md is the best reference and is the place where the packages are defined. I don't think you'll need to build Jenkins from source code in order to work on the packaging definitions.
You can download the Jenkins war file from your local Jenkins installation or from https://get.jenkins.io/war-stable/
markewaite is a genius I can ignore this Jenkins build error; Google Doc Jenkins Build Error 20211217-1230
I will try packaging work with suggested WAR download. Many thanks!!
- File packaging/setup.mk requires MSI and WAR be defined:
- export WAR?=$(error Required variable WAR must point to the jenkins.war file you are packaging)
- export MSI?=$(error Required variable MSI must point to the jenkins.msi file you are packaging)
- Download https://get.jenkins.io/windows-stable/2.319.1/jenkins.msi helps but...
- Ubuntu 20.04 LTS (focal) lacks createrepo package wanted by packaging/rpm/setup.sh file.
- sudo apt-get install -y rpm expect createrepo
- Ubuntu availability limited to 18.04 LTS https://packages.ubuntu.com/bionic/createrepo
Should 2 above become a separate issue?
It looks like the createrepo utility is available in Debian 10 and in Ubuntu 18.04 but is not available in Debian testing or in Ubuntu 20.04. Short term, that means you'll either need to use one of those two operating systems or use a Docker image from one of those two operating systems. The system that performs the weekly build is based on Ubuntu 18.04 as far as I know.
I think that reporting a separate issue for the absence of createrepo from the Ubuntu distribution won't help. It was removed intentionally because they decided to rewrite it in C rather than port the Python 2 code to Python 3.
Should 2 above become a separate Jenkins JIRA issue?
Debian package createrepo-c is added in same release dropping createrepo i.e. no function gap, just name discontinuity.
Ubuntu release sequence has the name discontinuity AND both packages absent gap.
My particular concerns so far include:
Where 1 fails (createrepo package) and 2 is incomplete (debuild docker vagrant ...) and vagrant wants "serverspec" which I guess is: vagrant-serverspec
My focus is driven by least effort to make useful contributions. I'd like to spare those who follow annoyances found so far.
My use case now is local virtual machine without Docker. I don't rule Docker out but ask is supporting both worthwhile?
IMO jenkinsci/packaging has vast improvement opportunity and I'm asking guidance about managing the process.
Since there hasn't been much improvement on jenkinsci/packaging, any area you feel to improve is worth considering.
My initial thoughts (with the caveat that I'm just guessing) were that there are at least three separate steps involved and that those steps could be separate from one another, delivered independently. Debian, Red Hat RPM, and SUSE RPM seem each like independent, separable steps.
I like the idea of the Docker image for development, but have not spent enough development time in this repository using the Docker image. The last changes I made were done from my local computer and had all the tools installed locally without a Docker image. If you find that the Docker image slows development or is unworkable, I suspect others are open to removing it.
If there is a change to add support for SystemD, please don't remove the existing sysvinit support. It is possible to keep both together so people using non-SystemD init systems are not forced to migrate off Jenkins. Debian also supports alternative init systems, so it is not just Devuan users that would be affected.
The artifacts from build 23 of jenkinsci/packaging#266 are available for testing:
- deb: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/23/artifact/target/debian/jenkins_2.333_all.deb
- rpm: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/23/artifact/target/rpm/jenkins-2.333-1.1.noarch.rpm
- suse: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/23/artifact/target/suse/jenkins-2.333-1.2.noarch.rpm
Please test these in a staging (non-production!) environment and report your results in jenkinsci/packaging#266. Thank you in advance.
I received a very low participation rate from testers the last time I posted, so I am posting again to beg for people to test the artifacts from build 35 of jenkinsci/packaging#266:
- deb: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/35/artifact/target/debian/jenkins_2.333_all.deb
- rpm: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/35/artifact/target/rpm/jenkins-2.333-1.1.noarch.rpm
- suse: https://ci.jenkins.io/job/Packaging/job/packaging/job/PR-266/35/artifact/target/suse/jenkins-2.333-1.2.noarch.rpm
If you are able, please test these in a staging (non-production!) environment and report your results in jenkinsci/packaging#266. Many thanks in advance.
I've tested packaging PR 266 and found no issues. The transition is looking good to me.
I've tested in Debian 10, Debian 11, and Red Hat Enterprise Linux 8. I won't be testing in a System V init environment like Devuan. Others will need to do that testing. wwuck, are you available to test the latest build of PR 266 in one of your Devuan environments?
markewaite I have run a test on build 36. Would you like me to test the artifact from build 37 as well?
basil Does this change the ability of Jenkins to run on ports 80 and/or 443? One needs root privilege to use those ports, the old init.d worked with https port set to 443, but I had no luck starting Jenkins 2.335 via systemd on port 443 (I ended up using NGINX as a proxy).
Does this change the ability of Jenkins to run on ports 80 and/or 443? One needs root privilege to use those ports, the old init.d worked with https port set to 443, but I had no luck starting Jenkins 2.335 via systemd on port 443 (I ended up using NGINX as a proxy).
zbynek Fixed in jenkinsci/packaging#285 toward Jenkins 2.336. In the future, please file a new Jira issue with details about the Linux distribution being used and steps to reproduce the problem.
We just upgraded LTS from 2.319.3 to 2.332.1 and this seemed to break our install. To fix we had to manually re-enable the service with:
systemctl enable jenkins
After running that, all is well.
We've recently upgraded to LTS 2.332.1 as well and have noticed that on RHEL-based distos, the shipped unit file does not respect the configuration which is still written to `/etc/sysconfig/jenkins`.
We had a custom `JENKINS_JAVA_CMD` statement which was ignored as a part of rebuilding on the new LTS version, which caused the service to fail to start up.
I think if the RHEL package is going to ship wit h `/etc/sysconfig/jenkins`, the unit file needs to respect it when it's altered.
hybby, that should probably be a new issue, linked to this one as this issue is closed.
lkisac01 Kindly refrain from commenting on closed tickets with requests for help that have nothing to do with the ticket you are commenting on. This is a bug tracker, not a help desk. A quick search of this bug tracker will show that you are looking for JENKINS-67165. For further requests for help please post to the user mailing list.
I'll leave the unit template here for Jenkins within multi-instance tomcat installation, maybe this will be helpful.
Only one thing - the internal restart commands like safeRestart or restart cause service to be unavailable since restart command is not correct in such case.
UPD:
Example config file for tomcat@jenkins instance:
Regards.