Upgrading Artifactory plugin from 3.6.2 to 3.10.6 fixed this for us

brendanh that's a good result. That supports the instructions in the upgrade guide that users should update their plugins before the upgrade to Jenkins 2.277.1 and then update their plugins again after they upgrade to Jenkins 2.277.1. Unfortunately, as far as I can tell, the artifactory plugin is not involved in this bug report. clsmith4 has provided the list of installed plugins from his system, and the artifactory plugin is not in the list.

Performance Plugin also results in a similar problem IIRC.

In our case, broken plugins were warnings and static analysis utilities

Which were replaced by single warnings-ng plugin

I have the same issue since upgrading from 2.777.2 to 2.777. (ofc I have updated all plugins after that too).
It occurs only on jobs which have many lines in the pipeline or shell execute (i.e 10 lines no problem, having 100 and more: fails)

I got the above error then and found nothing in the logs (but I may look at the wrong places ofc)

I had to rollback to 2.777.2 and it works as before.

steadfasterx the failure behavior depends on the specific plugins and versions installed on your Jenkins.

Please open a new issue using the instructions at "How to report an issue" with very careful attention to include the precise list of plugins and their versions as listed in "What information to provide for Environment and Description".

The failure you're reporting might also be related to an incorrect configuration of a reverse proxy (like nginx or Apache) between your web browser and Jenkins.

Hi Guys,

Why this kind of a breaking change is included in the LTS version on the first place?

I'm sure that I'm not the only one using the same Jenkins instance for several years now.

Which can mount to ten's if not hundreds of installed plugins.

This update is basically putting us all on hold...I'm actually starting to realize that we may need to look an alternative to Jenkins.

If you ask me, the next LTS version should rollback this change. At least till the vast majority of plugins support this.

Thanks,

Gilad

glevy

You're advocating for Jenkins to never change, because there is no realistic way to implement any change of this scope without impacting tons of plugins, many of them unmaintained. That said, we did our best to reduce the impact.

Why this kind of a breaking change is included in the LTS version on the first place?

We specifically timed merging this change until the weekly release after the previous LTS baseline to give maintainers the most time before this went into LTS. It was in a weekly release in late October, meaning plugin developers had more than four months to fix their plugins before an LTS user was impacted. Core maintainers identified many potentially affected plugins and notified their maintainers in advance (last June!), provided relevant developer documentation, and even fixed dozens of plugins themselves. For admins, there's the upgrade guide with an overview of what we know to be impacted so they can prepare for the upgrade.

We could have put a "3" in front of the version number to make this even more clear, but that's about it – other than abandoning any efforts to modernize Jenkins altogether.

Hi danielbeck,

I get your point and understand the challenge in implementing a change of this scope.

But try to look at this from a user/customer perspective, I find it hard to justify that my "Jenkins as a whole" production environment is compromised by a "nice to have" feature. 

Hi guys,

Looks like I am entrapped in the same situation. After following the upgrade procedure from LTS 2.263.3 to 2.277.3, there are some options in the Job->Configure that are not getting saved (although the green label "saved" shows up at the top). If I retract to the last working config.xml which contains my settings of the "General" tab and "Build Triggers" (like Poll SCM etc..) and I click "save" (again, the label confirms all supposed to be well) - the settings are gone. These ones:

• Multiple SCMs plugin
• Icon Shim
• Pipeline: Declarative Agent API
• jQuery UI plugin

I got disabled and uninstalled. I've got 183 plugins installed and enabled but can't really find the culprit.

I got all plugins updated but the two, which contain breaking changes and I don't want to break 100+ jobs depending on them: 

Copy Artifact Plugin 1.43.1 and TEAM FOUNDATION SERVER PLUG-IN 5.157.1 

best wishes, I'd really appreciate a hint, 12h spent on it and counting..

domi_nik if you cannot run without Team Foundation Server plugin (and its two security issues), then you need to return to Jenkins 2.263.4. Alternately, you could consider becoming the maintainer of the TFS plugin. There are many TFS plugin users that would be very grateful to you.

This sounds somewhat similar to an issue I ran into with saving jobs configurations https://issues.jenkins.io/browse/JENKINS-65276 

In my case we did not actually need the tfs 5.157.1 plugin (we were just experimenting with it) and uninstalling that plugin resolved my problem saving maven jobs. I note that it seems really strange that the tfs plugin caused problems when trying to save jobs where it was not even used.

FTR INFRA-2751 for the discussion about republishing the TFS plugin. At the moment we do not plan to provide a fix until a new maintainer steps up

In our case the culprit was also the TFS plugin. Fortunately we were not actually using it and we could uninstall it to solve the issue.

Hello guys!

Many thanks markewaite and jmmccullough , indeed the TFS plugin is the culprit. It's just sufficient to disable it to get everything working as expected. Thank you for inviting me .. but I am not a developer .  

It's yet another issue with Jenkins, where the answer is "maintain this or fix that yourself" - well sorry, but no. Development of Jenkins in recent months is kind of a joke really. Stuck on Java 8, with Java 11 (which is old already) version leaking memory like crazy, and no newer versions support.   Half of the plugins haven't been updated for years. And then you bring issue like this (which should be reversed and considered for v.3.0 release as mentioned above) which not in any universe should be brought to regular LTS update. 

It's no wonder more and more people are moving to other software - Jenkins is becoming really hard to be taken seriously anymore.

oleg_nenashev

I was also able to pinpoint the issue to the TFS plugin as well.

But it is mandatory on our setup...

If someone could suggest the changes needed in order to fix the the table/div issues it would be greatly appreciated.

people can decide if they are willing to take the risk and build it by themselves.

An unofficial fix can buy us some time...while moving everyone to git.

Hello,

I detected that jobs that have the join trigger plugin cannot be saved.

I detected that jobs that have the join trigger plugin cannot be saved.

JENKINS-64639

I'm having the same issues saving changes to jobs. I tried creating a new job and started adding plugins one at a time to see if I could reproduce it. In my case the plugin causing my issue was the Publish Over CIFS plugin. When I added it I was able to save the job. I went back and tried to modify it and then it would not let me save/apply any changes.

On Jenkins Jenkins 2.277.4 and Publish Over CIFS 0.12.

In my case the plugin causing my issue was the Publish Over CIFS plugin

JENKINS-64236 was fixed in 0.15, released in November.

Hello,

I detected this problem with saving job in  Groovy pipeline editor. (See console log in picture - this error is appeared if I do changes in pipeline editor => page is crashed if I click on save button)

Saving configuration is working correctly if I am doing changes directly in chrome browser in my JENKINS server.

I had the similar issue, in my case it was solved after uninstall the plugin "OWASP Markup Formatter".

See the problem is - there are literally hundreds of plugins on official update center and no comprehensive list of the troublemakers. And it's not so easy to just try them all just like that. 

Because of this (and non-existent effort to make Jenkins work with up-to-date Java versions), we're looking for alternative software suite (and there are some, thankfully), as this is just silly.

I don't believe it is a plugin problem. I have a number of Jenkins servers displaying the same problem with the inability to save (Running on Windows Server 2012R2 and Windows 10).

I too tried removing various plugins and thought that I may have found the problem because I could then occasionally save, but then realised that it was actually an intermittent problem. Sometimes it does save straight away and other times not, so I reenabled the plugins and went back to the drawing board.

On 5 of the 6 Jenkins servers I am running, I am using the latest LTS version and have seen this problem on all of them, the only one that was not displaying this problem was the one using the weekly release version. So to cut a long story short, I changed one of the LTS versions (2.277.4.) to 2.295 and the problem was solved. I then tried it on a second server with exactly the same success.

So in conclusion I an convinced it is a problem with the LTS core and not any plugins. I would prefer to use the LTS versions on my productive machines, but this problem is very frustrating and I may have to switch the rest of my servers to the weekly release instead of the so called "stable" release if this bug is not fixed.

I changed one of the LTS versions (2.277.4.) to 2.295 and the problem was solved

More likely than not the 2.289.1 scheduled for release tomorrow will solve it then. Otherwise I would ask you file a new issue and provide useful information what exactly goes wrong, any log and browser console output, etc. — these catch-all issues where a dozen people experience similar but probably not the same problem and provide contradictory information are not really useful in addressing the problems encountered.

I found that going back to 2.277.1 greatly reduced the amount of saving errors (didn't have any while saving job configs and only had some error popups while saving global settings, but the settings themselves would save correctly). 

Hi peejay,

Do you have the TFS plugin installed on one of the environment by any chance?

danielbeck

"More likely than not the 2.289.1 scheduled for release tomorrow will solve it then."

We upgraded to v2.289.1 on Windows 2012 Servers and the issue still persists.

Changing some portions of job configurations will allow Apply or Save process to work, but changing others, such as Git plugin section causing no action when Apply is clicked, and error shown when Save is clicked:

Thanks,

John Burrows

My experience with 2.289.1 seems to be exactly the same as John Burrows'. I found that I could generally save the general settings, but the random first job on my test instance would fail with the same error as above...

Replay and Upload file also affected

Updated to 2.289.1, but still the same issue.

One more information: The problem also appears when using "replay for Pipelines" or "upload a file for Job run". So basically always when there is a HTTP POST operation involved.

Update: We've rolled back to 2.277.1 for now. Problem is not existing there.

jburrows

We upgraded to v2.289.1 on Windows 2012 Servers and the issue still persists.

Can you confirm you have no problems like this in 2.295, like the person I was responding to?

danielbeck my issue with saving configuration of a pipeline is finally gone in v2.297 !

Before that I was able to narrow it down between 2 versions:

2.277.2: no issues
2.277.3 and later: issues occurs

so maybe introduced by https://www.jenkins.io/security/advisory/2021-04-20/ .
no idea if that helps someone and I will test further if v2.297 fixes all my related issues

Thanks for the response steadfasterx and I'm happy it works for you again. Based on the versions provided and the specific error you showed earlier, that seems likely. Might be useful if you're able to narrow down the version that fixed it, starting with the releases updating Winstone/Jetty (2.290, 2.296) and their immediate predecessors.

FWIW this nicely illustrates the difficulty in investigating such reports from the maintainer side – the last known good version for you is later than the first bad version in the original report, i.e. different people report (likely) different problems in the same issue. Without direct access to your infra or other ways to reliably reproduce the problem from scratch, we have nothing, because what information is provided is contradictory. This isn't intended as a slight to anyone involved here, just explaining how this looks from the other side.

While troubleshooted an instance impacted by this when upgrading from 2.277.2 to 2.277.3, we found the following issues Encrypted buffer max length exceeded when enabling FINE level for org.eclipse.jetty:

2021-06-08 19:52:47.334+0000 [id=1667607]	FINE	o.e.j.i.s.SslConnection$DecryptedEndPoint#handleException: DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=412/5000} stored fill exception
javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:735)
	at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:342)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(Thread.java:748)
2021-06-08 19:52:47.335+0000 [id=1667607]	FINE	o.e.j.i.s.SslConnection$DecryptedEndPoint#fill: <fill f=-2 uf=false SslConnection@72088a32::SocketChannelEndPoint@73a710b0{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=0/5000}{io=0/0,kio=0,kro=1}->SslConnection@72088a32{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=412/5000}=>HttpConnection@27093e32[p=HttpParser{s=CONTENT,0 of 21594},g=HttpGenerator@6c5eb404{s=START}]=>HttpChannelOverHttp@35bf1d41{s=HttpChannelState@3125348b{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=48,c=false/false,a=IDLE,uri=//jenkins.example.com/job/-freestyle-/configSubmit,age=2}
2021-06-08 19:52:47.335+0000 [id=1667607]	FINE	o.e.jetty.io.AbstractEndPoint#close: close(javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded) DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP

Could those problems starting from 2.277.3 be actually caused by https://issues.jenkins.io/browse/JENKINS-65280 / https://github.com/eclipse/jetty.project/pull/6073 and not table to div ?

see https://issues.jenkins.io/browse/JENKINS-65624

steadfasterx as you can easily reproduce.  maybe (not sure) you could try using `-Djsse.SSLEngine.acceptLargeFragments=true`  and tell us if it fix the issue? (but Jenkins weekly should definitely fix it)

olamy We tried this Java option Djsse.SSLEngine.acceptLargeFragments=true with version 2.289.1, but unfortunately we still have our initial problem. For us this shows as "secure connection failed", but I guess that's browser specific.

I can confirm this issue has been solved in the new LTS version 2.289.2.

After upgrading to 2.289.2 the save button works as expected, as 2.277.1 did. Versions between these two versions had the problem from this issue.

We are using 2.289.2 and suffering this problem.  Save/Apply fails to work.  Hitting "Apply" in FireFox throws up a blank "Error" box, whereas all other browsers do nothing.

Have uninstalled any suspect plugins from the comments here and from this list:

https://issues.jenkins.io/secure/Dashboard.jspa?selectPageId=20741

But still not able save/apply on any jobs.  This is a pretty big deal.

garen I suspect that you're describing a different set of conditions than the conditions in the original bug report. I suspect you're describing a different set of conditions than those in the comments from others in this issue report.

Usually, the best approach is to perform the steps described in the Jenkins 2.277.1 upgrade guide and in the Jenkins 2.277.1 upgrade webinar so that you can identify the plugin or plugins causing the issue yourself. Those steps are:

• Upgrade all plugins to latest releases
• Remove plugins with security issues (like TFS)
• Remove deprecated plugins (like warnings, checkstyle, pmd, etc.)
• Remove unused plugins
• Remove plugins with known issues in tables to divs

If those steps are not enough to resolve the issue, see the instructions at https://www.jenkins.io/doc/developer/views/table-to-div-migration/#identifying-the-broken-plugin .

If the steps on that page are not enough to resolve the issue, then open a new issue report with the precise list of plugin names and versions, as generated by https://www.jenkins.io/doc/book/system-administration/diagnosing-errors/#how-to-report-a-bug

garen that plugin is one of the plugins with known issues in tables to divs. The issue is described in JENKINS-65955. That's why the guidelines recommend removing plugins with known issues in tables to divs.

I too am struggling with Jenkins instance to Save some settings to my existing projects. The SAVE / Apply is not working on editing adjusting any of the Project setup elements
What should be done to get the project edit/update work for my jenkins instance.

Please revert with resolution.

Thanks in Advance

Same problem with release 2.277.4.
Does not systematicaly occurs. But when facing the problem, reloading the page with CTRL+F5 allows to be again able to save modifications. Just a workaround, but maybe a clue...

nitinqa and lviolon please refer to my earlier comment for the steps you need to take to investigate your specific issue. When you declare that you see the same problem, but don't provide the details that confirm your exact plugin names and versions, you're not providing enough information for others to assist.

You're right Mark. The aim of my comment was just for the moment to share the workaround.
I read your comment and looked to the dashboard about divs issues.
Although I have a large amount of plugins (more than 100), only job-dsl-plugin and publish-over-ssh-plugin are in the dashboard. Nevertheless, I have too much plugins to investigate a non systematic problem.
So I will start with removing as much plugins as possible.

same problem for us as well..WE have installed Jnekins 2.303.1 as the changelog for the versions said that this issue is fixed, however it is not fixed. I am attaching(Installed Plugins SJ.docx) our list of installed plugins here. I went through the suggested steps over different tickets in regards to this issue and followed suggestion but no resolution so far. This save/apply issue only occurs to our Maven projects but other freestyle pipelines are working fine. I also disabled Job Mail Direct Plugin and other plugins suggested in the known issues table but no luck.

sj you said:

WE have installed Jnekins 2.303.1 as the changelog for the versions said that this issue is fixed, however it is not fixed

Please provide the details of the location in the 2.303.1 changelog that claims "this issue is fixed". There is no reference to this issue in the changelog or in the upgrade guide.

The list of plugins that you uploaded ( Installed Plugins. SJ.docx ) is unusable for others who might want to help you. If you'd like help from others, please follow the instructions at "How to report an issue", especially the part that describes how to provide a precise list of the plugins you have installed and their versions. The list you provided uses a format that is difficult to read, does not have version numbers, and uses the display name of the plugin instead of the plugin identifier.

Based on the list, I assume that "Build Time Blame" plugin in your list will have tables to divs issues, since https://github.com/jenkinsci/build-time-blame-plugin/search?q=%3Ctable shows that it uses table markup for at least one of its actions. Remove that plugin and see if it helps .

The checkstyle plugin has reached end of life. Remove it.

The job direct mail plugin is known to have tables to divs issues. Remove it.

You said:

I went through the suggested steps over different tickets in regards to this issue and followed suggestion but no resolution so far.

As far as I can tell, your installed list of plugins indicates you did not follow the instructions in other issue reports or the instructions in this issue report.

To reiterate, those instructions are:

Perform the steps described in the Jenkins 2.277.1 upgrade guide and in the Jenkins 2.277.1 upgrade webinar so that you can identify the plugin or plugins causing the issue yourself. Those steps are:

• Upgrade all plugins to latest releases
• Remove plugins with security issues (like TFS)
• Remove deprecated plugins (like checkstyle, mulitple scms, pmd, etc.)
• Remove unused plugins (like CVS, Subversion, Ivy, External Monitor Job Type, etc.)
• Remove plugins with known issues in tables to divs (like job direct mail plugin)

If those steps are not enough to resolve the issue, see the instructions at https://www.jenkins.io/doc/developer/views/table-to-div-migration/#identifying-the-broken-plugin .

If the steps on that page are not enough to resolve the issue, then open a new issue report with the precise list of plugin names and versions, as generated by https://www.jenkins.io/doc/book/system-administration/diagnosing-errors/#how-to-report-a-bug

markewaite - As a quick update, we managed to get past this issue successfully by completely removing all references/traces to the coding-webhook plugin and by also upgrading to Jenkins version 2.303.2. I would've replied earlier, but I wanted to confirm that the upgrade from 2.263.4 to 2.303.2 fixed the Global configuration save/apply binding issues for us. (i.e. - Global Security/Configure System/etc.)

Once we were able to generate a complete list of plugins that were unique only to the problematic Jenkins master, we installed those same plugins on a separate test Jenkins master and starting uninstalling/removing those specific plugins one by one. By completely removing all references of the coding-webhook plugin, we were able to successfully save/apply new binding changes to all Jenkins jobs again.

I should note that the coding-webhook plugin was previously uninstalled during the earlier plugin troubleshooting efforts, but the plugin's homedir under ${JENKINS_HOME/plugins} was still present along with other recursive references to the plugin's .JPI file under ${JENKINS_HOME}. The strange part is that the actual .JPI file was completely removed during the initial plugin uninstall and wasn't found anywhere under the ${JENKINS_HOME} which was a red flag. Another giveaway for us was intermittent error messages about the coding-webhook plugin were observed from time-to-time while refreshing the installed plugins list in the Jenkins UI after a restart of Jenkins. It wasn't a continuous error message and would go away after a certain amount of time, but would show up at the bottom of page after restarting Jenkins and then immediately refreshing the installed plugins list afterwards.

While removing all references of the coding-webhook plugin resolved the save/apply Jenkins jobs issue, we had to wait until upgrading to version 2.303.2 to officially confirm that the global save/apply binding issues that we encountered had been resolved for us.

Thank you markewaite and everyone else that took the time to reply for helping to guide us with the direction on troubleshooting, diagnosing and ultimately getting us past these issues!

I'm glad to hear it. The coding webhook plugin has been deprecated in favor of the coding.net Jenkins integration. Since that site is in a language I do not read, I assume that it describes something about Jenkins and their integration with Jenkins.

this problem might be similar to this one dealing with vsphere plugin:
https://issues.jenkins.io/browse/JENKINS-67234