Is this a new installation, or an upgrade?

Could you try the beta which we released earlier today?

There is no new version available on the jenkins-rc repo:
[centos@jenkins ~]$ sudo yum list jenkins
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

• base: ftp.lysator.liu.se
• extras: ftp.lysator.liu.se
• updates: ftp.lysator.liu.se
  Installed Packages
  jenkins.noarch 2.0-1.1 @jenkins-rc

I can't see which build it corresponds to.

Probably just caused by our release process that does not support multiple alphas etc. and reuses/replaces versions. This is the first time we don't just increment a number on a weekly basis, and it shows

If you look at http://pkg.jenkins-ci.org/redhat-rc/ it was last updated this Thursday.

The sysconfig contains
JENKINS_AJP_PORT="8009"
JENKINS_AJP_LISTEN_ADDRESS=""
and that triggers the behaviour.

If I comment out those two lines, then it works much better.

If AJP is no longer supported (which is what java -jar jenkins.war --help indicates to me), then AJP should be removed from sysconfig and init-script. It is a debatable point whether jenkins should ignore those parameters rather than crash, but it is a rather heavy-fisted way of telling people to update their installation. Probably the code has not been completely removed yet.

Anyways, the fix seems simple - suggestion attached as beta1.patch

Thanks for reaching out!

Native packaging needs to be adapted to no longer use the AJP connector. Ideally, Winstone will not fail so hard but just log a warning otherwise.

https://github.com/jenkinsci/packaging/blob/master/rpm/build/SOURCES/jenkins.sysconfig.in#L110
https://github.com/jenkinsci/packaging/blob/master/rpm/build/SOURCES/jenkins.init.in#L91...L92

https://github.com/jenkinsci/packaging/blob/master/suse/build/SOURCES/jenkins.sysconfig.in#L77
https://github.com/jenkinsci/packaging/blob/master/suse/build/SOURCES/jenkins.init.in#L82

https://github.com/jenkinsci/packaging/blob/master/deb/build/debian/jenkins.default#L60
https://github.com/jenkinsci/packaging/blob/master/deb/build/debian/jenkins.default#L69
https://github.com/jenkinsci/packaging/blob/master/deb/build/debian/jenkins.default#L75

Suggest: remove options for AJP from packaging and consider making winstone to log a warning and fallback to a working behavior.
Consider: When upgrading from a older Jenkins that was previously using AJP, Jenkins shouldn't fail hard.

Breaking AJP will be a hard deal-breaker for some people, even though it is an older technology. This would be a significant compatibility break for some users.

If possible we should handle this WAR-side: the best option would be to use some library to translate AJP requests to proxied HTTP, failing that we should ignore arguments and log a warning (so existing configurations do not break on upgrade).

Consider: When upgrading from a older Jenkins that was previously using AJP, Jenkins shouldn't fail hard.

Agreed. This would be a Big Deal for some.

Edit: to be clear, while we can comment out the AJP arguments in the default configuration, it won't fix users getting their AJP systems broken.

Worth noting that if you look at the Jetty issue, there are a number of people who were reliant on AJP, but they are pretty emphatic that support for AJP isn't coming there: https://bugs.eclipse.org/bugs/show_bug.cgi?id=425244

FWIW given the popularity of Jetty, I don't think an issue with 3 votes and 10 watchers indicates a widely used feature.

And we're not really able to back this out anyway anymore, we need the Servlet 3 support. This will need to be addressed in the release notes, hoping someone reads them.

However, what we need to address in Jenkins is the brokenness of packaging, and, depending on how upgrading those works, whether we need to downgrade from exception to warning being logged in winstone-jetty (so those who have it enabled unnecessarily can still use Jenkins)

Code changed in jenkins
User: Sam Van Oort
Path:
deb/build/debian/jenkins.default
osx/Library/Application Support/Jenkins/jenkins-runner.sh
osx/Library/Documentation/Jenkins/command-line-preferences.html
rpm/build/SOURCES/jenkins.init.in
rpm/build/SOURCES/jenkins.sysconfig.in
suse/build/SOURCES/jenkins.init.in
suse/build/SOURCES/jenkins.sysconfig.in
http://jenkins-ci.org/commit/packaging/581cc9f8f3d86fbc744a796d3129b5f40ebdb228
Log:
Remove AJP per JENKINS-33776 because Jetty 9 dropped support

Code changed in jenkins
User: Sam Van Oort
Path:
deb/build/debian/jenkins.default
osx/Library/Application Support/Jenkins/jenkins-runner.sh
osx/Library/Documentation/Jenkins/command-line-preferences.html
rpm/build/SOURCES/jenkins.init.in
rpm/build/SOURCES/jenkins.sysconfig.in
suse/build/SOURCES/jenkins.init.in
suse/build/SOURCES/jenkins.sysconfig.in
http://jenkins-ci.org/commit/packaging/fd3abc6bae3575829a2c42609f3257b90408e471
Log:
Merge pull request #51 from jenkinsci/remove-ajp

Remove AJP per JENKINS-33776 because Jetty 9 dropped support

Compare: https://github.com/jenkinsci/packaging/compare/b3ce03afe1af...fd3abc6bae35

So, in operations terms, where is it documented how to address this change in 2.0 coming from 1.6?

We're not using any reverse proxy setup in front of our Jenkins instance, yet it completely fails to start since 2.0 as released overnight.

{{Running from: /usr/lib/jenkins/jenkins.war
Apr 21, 2016 9:06:28 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Logging initialized @652ms
Apr 21, 2016 9:06:29 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Apr 21, 2016 9:06:29 AM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Empty contextPath
Apr 21, 2016 9:06:29 AM winstone.Logger logInternal
INFO: Winstone shutdown successfully
Apr 21, 2016 9:06:29 AM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.IOException: Failed to start a listener: winstone.Ajp13ConnectorFactory
at winstone.Launcher.spawnListener(Launcher.java:207)
at winstone.Launcher.<init>(Launcher.java:148)
at winstone.Launcher.main(Launcher.java:352)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Main._main(Main.java:246)
at Main.main(Main.java:91)
Caused by: java.lang.UnsupportedOperationException: AJP support is removed in Winstone 3.0 due to Jetty 9 not supporting AJP. For reverse proxying, please use HTTP instead of AJP.
at winstone.Ajp13ConnectorFactory.start(Ajp13ConnectorFactory.java:31)
at winstone.Launcher.spawnListener(Launcher.java:205)
... 8 more
}}

Fix for us was to configure /etc/sysconfig/jenkins and set the AJP port setting to "-1". Since we'd never touched that file before, I hadn't looked there.

EDIT: I am wrong above. We'd touched that file before. We drop it in place as part of the 'jenkins' Chef cookbook.

I realize this is open source, and this outage for us is fully our responsibility since we were allowing unattended RPM updates for any version of Jenkins pushed to the Jenkins RHEL/CentOS yum repository ... but it would have been ideal to have some sort of operational guidance in the 2.0 docs about where to look for AJP issues, etc. We effectively had this "totally backward compatible!" upgrade break our Jenkins overnight and with no real operational guidance as to where to look to solve the issue mentioned in the log.

jblaine This should not have happened, since changes were made to the packaging in order to address this and remove the offending AJP arguments in the config. It seems that the changes did not replace the existing configuration in-place. I will took a look at how to address this in packaging, and provide guidance in release notes.

Looking into upgrade cases that may have been missed

svanoort I think you did the right/common thing. The new settings were dropped into /etc/sysconfig/jenkins.rpmnew as is typical on a RHEL box where someone has modified the previous default file (/etc/sysconfig/jenkins). We had modified ours.

I think maybe the most-right solution is to make the exception message clearer for operations folks?

"<some-ajp-propertyname-here> is enabled in the current runtime environment and is set to the value "<value-here>". This means you are trying to use AJP with Jenkins 2+. AJP support is removed in Winstone 3.0 due to Jetty 9 not supporting AJP. It is therefore removed from Jenkins as well. For reverse proxying, please use HTTP instead of AJP."

If not that, maybe something other than "bunch of Java terms"

jblaine Had you guys also customized the init scripts for jenkins? The change strips out AJP arguments being passed in via the init script too, since I knew about RPM's default behavior being to retain existing config files (which would have caused your scenario). However if the init script is modified as well... then all bets are off.

2.0 docs about where to look for AJP issues, etc. We effectively had this "totally backward compatible!" upgrade break our Jenkins overnight

We warn users about this change on https://jenkins.io/2.0/ and on https://jenkins.io/changelog/ , so we tried

I'll wait for a response to Sam's latest comment before deciding how to proceed here. We can't just ignore when that option is set in Winstone-Jetty because of those installs that actually use it. And we cannot really include a message that contains Jenkins-specific (and platform specific) advice as it's technically an independent component.

No response in more than a week, so resolving as Not a Defect – customized config files passed now invalid parameters and need to be adapted.

I ran into this problem myself going from 1.656 to 2.1 via yum. First I attempted to create and apply patches from /etc/sysconfig/jenkins, /etc/sysconfig/jenkins.rpmnew, /etc/init.d/jenkins and /etc/init.d/jenkins.rpmnew. I validated that the AJP settings were returned into the configuration however I was still getting the stacktrace reported with this issue.

My resolution was to:

1. Set JENKINS_AJP_PORT="-1"
2. Validate that Jenkins would start (meaning I wasn't utilizing AJP, which I knew)
3. Upgrade from 1.656 to 2.1 via yum

So ultimately it would appear that you have to disable AJP and then the settings in /etc/init.d/jenkins mean nothing. My example shows that I had to disable AJP before I even did the upgrade.

Food for thought.

I hit this problem going from 1.562 to 2.2 via update of war file.
I did not have customisations in /etc/sysconfig/jenkins
Setting JENKINS_AJP_PORT="-1" in /etc/sysconfig/jenkins solved problem.

jamesco Did you customize /etc/sysconfig/jenkins previously? Was a /etc/sysconfig/jenkins.rpmnew created by the 2.x install?

Hi Daniel,

I'm pretty sure I didn't customize /etc/sysconfig/jenkins . . I think maybe my problem was because of .war install and not rpm install.
Good to find this bug as could see how to solve problem.

I didn't get a /etc/sysconfig/jenkins.rpmnew.
I didn't do an rpm upgrade, when I clicked upgrade the .war file was downloaded.
So I swapped that in.

It was old enough version of jenkins I was going from as well.

Thanks.
James.

Okay. So this should be an unusual enough situation as unmodified RPM installs don't allow replacing the war this way – Jenkins has no write access to its own war file by default.

Hi, I think this might still be an issue. My chef tests started failing recently - they spin up a Centos 7 VM, install Jenkins (currently jenkins-2.7.1-1.1.noarch) from the http://pkg.jenkins-ci.org/redhat-stable repo, and then tries to start it. The /etc/sysconfig/jenkins config file included in the rpm has a line that tries to run AJP on port 8009:

## Type:        integer(0:65535)
## Default:     8009
## ServiceRestart: jenkins
#
# Ajp13 Port Jenkins is listening on.
# Set to -1 to disable
#
JENKINS_AJP_PORT="8009"

This causes jenkins to fail at startup as described by https://issues.jenkins-ci.org/browse/JENKINS-33776?focusedCommentId=255250&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-255250

It would be better if the /etc/sysconfig/jenkins file included in the rpm didn't try to configure AJP so jenkins would run right out of the box without requiring the users to modify it.

caboteria - I agree that this needs to be remedied from a base install. Have you tried installing with the latest version (2.13) to test? There's nothing that points to that being removed or changed for the RPM, but it never hurts to try. Since there is a workaround in place (and since it's marked as resolved) it's not a high priority for the devs.

This was marked resolved because we fixed it. I'm currently trying to get information on the release process, maybe something went wrong there (specifically, master branch of https://github.com/jenkinsci/packaging used rather than the jenkins-2.0 branch).

Thanks for your attention on this.

Based on the source I think it's likely that 2.13 behaves the same as 2.7.1. According to the PR linked from this report, the fix was merged to the jenkins-2.0 branch but there's a comment that might indicate that was a mistake: https://github.com/jenkinsci/packaging/pull/51#issue-144292301 says "A separate jenkins-2.0 packaging branch has been created, because pre-Jenkins 2 users may rely on AJP." which to me indicates that the fix should have been applied to master but not jenkins-2.0 (since the fix disables AJP). The fix was applied to jenkins-2.0, though.

It also looks as if the 2.7.1 RPM's are built with the master branch of jenkinsci/packaging instead of the jenkins-2.0 branch, as Daniel indicates. I supposed if both of these things were fixed then the result would be no change to the current behavior. Unfortunately, seomone's going to feel some pain, I guess, either the set of those using AJP, or the set of all new and upgrading users.

Would it make sense to re-open this report to track the effort to get the fix to end-users, or is there a different report to track that?

All other 2.x releases do not have this problem, as it's caused by different build/release processes for weeklies/LTS, and 2.7.1 being the first 2.x based LTS – and still using the master branch of packaging.

KK is on it, we'll probably just re-release 2.7.1 packages with the correct scripts. I will keep updating this issue in lack of a better issue. No need to reopen.

For what it's worth I found this because I'm part of that 'set of all new and upgrading users' that just got bitten by deciding to go for the 'more stable' LTS version. I guess there's always someone, however small the problem window is...

I'll disable it in sysconfig for now, and look forward to a fixed RPM.

Re-released with today's date: http://pkg.jenkins-ci.org/redhat-stable/ – not sure how that'll affect existing caches etc., but it's out at least.

The fresh install use case now works for me. Thank you!

Code changed in jenkins
User: Sam Van Oort
Path:
README.md
http://jenkins-ci.org/commit/packaging/61d57d8b885b19e071c1a434db20a8bbe95889ab
Log:
Mark that this is just for pre-2.x due to removing AJP for JENKINS-33776