markewaite - After reviewing the dependency issues it looks like most of the issues stem from the Folders Plugin being disabled ,however, when I check in Plugin manager it shows as being installed and not disabled or uninstalled(see attached screenshot). Why is it complaining that the Folders Plugin is disabled when it shows up as being installed in Plugin manager? I also see the same for a few other plugins showing as disabled.

markewaite - We have another Jenkins master in a lower environment that was stood up with almost exactly the same plugins and updated to the 2.277.1 release and we're not seeing the issue on this master. Not sure why the other master is seeing this issue but below are the plugins that were installed on this master:

BlazeMeter plugin (BlazeMeterJenkinsPlugin): 4.7
JiraTestResultReporter plugin (JiraTestResultReporter): 2.0.7
Office 365 Connector (Office-365-Connector): 4.14.0
Parameterized Remote Trigger Plugin (Parameterized-Remote-Trigger): 3.1.5.1
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Additional Metrics Plugin (additional-metrics): 1.3
All changes plugin (all-changes): 1.5
Amazon Elastic Container Service (ECS) / Fargate plugin (amazon-ecs): 1.37
Ansible plugin (ansible): 1.1
Ant Plugin (ant): 1.11
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 2.1
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.13-1.0
AppDynamics Dashboard Plugin for Jenkins (appdynamics-dashboard): 1.0.15
Artifactory Plugin (artifactory): 3.10.5
Associated Files Plugin (associated-files): 0.2.1
Async Http Client (async-http-client): 1.9.40.0
Audit Trail (audit-trail): 3.8
Authentication Tokens API Plugin (authentication-tokens): 1.4
Amazon S3 Bucket Credentials Plugin (aws-bucket-credentials): 1.0.0
CloudBees AWS Credentials Plugin (aws-credentials): 1.28
AWS Global Configuration Plugin (aws-global-configuration): 1.6
Amazon Web Services SDK (aws-java-sdk): 1.11.973
AWS Lambda Plugin (aws-lambda): 0.5.10
AWS Parameter Store Build Wrapper (aws-parameter-store): 1.2.2
AWS Secrets Manager Credentials Provider (aws-secrets-manager-credentials-provider): 0.5.3
S3 Yum repo parameter plugin (aws-yum-parameter): 1.5
Azure Commons Plugin (azure-commons): 1.1.1
Badge (badge): 1.8
Common API for Blue Ocean (blueocean-commons): 1.24.5
REST API for Blue Ocean (blueocean-rest): 1.24.5
Bootstrap 4 API Plugin (bootstrap4-api): 4.6.0-2
bouncycastle API Plugin (bouncycastle-api): 2.20
Branch API Plugin (branch-api): 2.6.3
Build Environment Plugin (build-environment): 1.7
Build History Metrics plugin (build-history-metrics-plugin): 1.2
build-metrics (build-metrics): 1.3
Build Monitor View (build-monitor-plugin): 1.12+build.201809061734
Build Name and Description Setter (build-name-setter): 2.1.0
Build Notifications Plugin (build-notifications): 1.5.0
Build Pipeline Plugin (build-pipeline-plugin): 1.5.8
Build-Publisher plugin (build-publisher): 1.22
Build Time Blame Plugin (build-time-blame): 2.0.3
Build Timeout (build-timeout): 1.20
build user vars plugin (build-user-vars-plugin): 1.7
Build With Parameters (build-with-parameters): 1.5
buildgraph-view (buildgraph-view): 1.8
built-on-column (built-on-column): 1.1
Calendar View Plugin (calendar-view): 0.3.1
categorized-view (categorized-view): 1.11
Checks API plugin (checks-api): 1.6.0
Claim Plugin (claim): 2.18.2
Cloud Statistics Plugin (cloud-stats): 0.26
Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 2.9.7
CloudBees Credentials Plugin (cloudbees-credentials): 3.3
Folders Plugin (cloudbees-folder): 6.15
CodeBuilder: AWS CodeBuild Cloud Agents (codebuilder-cloud): 1.0.1
Command Agent Launcher Plugin (command-launcher): 1.5
Compact Columns (compact-columns): 1.12
Concurrent Step (concurrent-step): 1.0.0
Conditional BuildStep (conditional-buildstep): 1.4.1
Config File Provider Plugin (config-file-provider): 3.7.0
Configuration as Code Plugin (configuration-as-code): 1.47
Configuration as Code Plugin - Groovy Scripting Extension (configuration-as-code-groovy): 1.1
Configure Job Column Plugin (configure-job-column-plugin): 1.0
Confluence Publisher (confluence-publisher): 2.0.6
Console Badge (console-badge): 1.1
Console Column Plugin (console-column-plugin): 1.5
Copy Artifact Plugin (copyartifact): 1.46
Create Job Advanced (createjobadvanced): 1.9.0
Credentials Plugin (credentials): 2.3.15
Credentials Binding Plugin (credentials-binding): 1.24
cron-shelve (cron-shelve): 1.3
Cron Column Plugin (cron_column): 1.4
Crowd 2 Integration (crowd2): 2.0.2
Custom Job Icon plugin (custom-job-icon): 0.2
Custom View Tabs Plugin (custom-view-tabs): 1.3
Customized Build Message Plugin (customized-build-message): 1.1
Dark Theme (dark-theme): 0.0.12
Dashboard View (dashboard-view): 2.15
database (database): 1.7
MySQL Database Plugin (database-mysql): 1.4
PostgreSQL Database Plugin (database-postgresql): 1.2
Delivery Pipeline Plugin (delivery-pipeline-plugin): 1.4.2
Dependency Graph Viewer Plugin (depgraph-view): 1.0.5
Description Column Plugin (description-column-plugin): 1.3
Display URL API (display-url-api): 2.3.4
docker-build-step (docker-build-step): 2.6
Docker Commons Plugin (docker-commons): 1.17
Docker API Plugin (docker-java-api): 3.1.5.2
Docker Pipeline (docker-workflow): 1.26
Doktor (doktor): 0.4.1
Downstream Build Cache Plugin (downstream-build-cache): 1.6
Downstream build view (downstream-buildview): 1.9
Drop Down ViewsTabBar Plugin (dropdown-viewstabbar-plugin): 1.7
Durable Task Plugin (durable-task): 1.35
Dynamic Search View Plugin (dynamic-search-view): 0.3.0
Dynamic Extended Choice Parameter Plug-In (dynamic_extended_choice_parameter): 1.0.1
Amazon EC2 plugin (ec2): 1.56
ECharts API Plugin (echarts-api): 5.0.1-1
Email Extension Plugin (email-ext): 2.82
Email Extension Template Plugin (emailext-template): 1.2
Embeddable Build Status Plugin (embeddable-build-status): 2.0.3
Environment Injector Plugin (envinject): 2.4.0
EnvInject API Plugin (envinject-api): 1.7
Environment Script Plugin (environment-script): 1.2.5
Extended Choice Parameter Plug-In (extended-choice-parameter): 0.82
Extended Read Permission Plugin (extended-read-permission): 3.2
Extensible Choice Parameter plugin (extensible-choice-parameter): 1.7.0
External Monitor Job Type Plugin (external-monitor-job): 1.7
Extra Columns Plugin (extra-columns): 1.22
Favorite (favorite): 2.3.3
Favorite View Plugin (favorite-view): 1.0
Font Awesome API Plugin (font-awesome-api): 5.15.2-2
GitHub Pull Request Builder (ghprb): 1.42.2
Git plugin (git): 4.6.0
Git Changelog (git-changelog): 3.1
Alternative build chooser (git-chooser-alternative): 1.1
Git client plugin (git-client): 3.6.0
Git Parameter Plug-In (git-parameter): 0.9.13
GIT server Plugin (git-server): 1.9
Git Tag Message Plugin (git-tag-message): 1.7.1
GitHub plugin (github): 1.33.1
GitHub API Plugin (github-api): 1.123
GitHub Branch Source Plugin (github-branch-source): 2.10.2
GitHub Issues Plugin (github-issues): 1.2.4
GitHub Authentication plugin (github-oauth): 0.33
GitHub Pull Request Coverage Status (github-pr-coverage-status): 2.1.1
Hudson global-build-stats plugin (global-build-stats): 1.5
Global Post Script Plugin (global-post-script): 1.1.4
Global Slack Notifier Plugin (global-slack-notifier): 1.5
Global Variable String Parameter (global-variable-string-parameter): 1.2
Google OAuth Credentials plugin (google-oauth-plugin): 1.0.4
Gradle Plugin (gradle): 1.36
Groovy (groovy): 2.3
Groovy Label Assignment plugin (groovy-label-assignment): 1.2.0
Groovy Postbuild (groovy-postbuild): 2.5
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.8-1.0
housekeeper (housekeeper): 1.1
HTML Publisher plugin (htmlpublisher): 1.25
HTTP POST Plugin (http-post): 1.2
Hudson Personal View (hudson-pview-plugin): 1.8
Icon Shim (icon-shim): 3.0.0
Ivy Plugin (ivy): 2.1
Jackson 2 API Plugin (jackson2-api): 2.12.1
Javadoc Plugin (javadoc): 1.6
JAXB plugin (jaxb): 2.3.0.1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.5
Design Language (jenkins-design-language): 1.24.5
Jira Issue Updater (jenkins-jira-issue-updater): 1.18
Multijob plugin (jenkins-multijob-plugin): 1.36
Jira plugin (jira): 3.2
jira-ext Plugin (jira-ext): 0.9
JIRA Trigger Plugin (jira-trigger): 1.0.1
Java JSON Web Token (JJWT) Plugin (jjwt-api): 0.11.2-9.c8b45b8bb173
Job Direct Mail Plugin (job-direct-mail): 1.5
Job Configuration History Plugin (jobConfigHistory): 2.26
Job/Queue/Slaves Monitoring Plugin (jqs-monitoring): 1.4
jQuery plugin (jquery): 1.12.4-1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
jQuery UI plugin (jquery-ui): 1.0.2
JQuery3 API Plugin (jquery3-api): 3.5.1-3
JSch dependency plugin (jsch): 0.1.55.2
JUnit Plugin (junit): 1.49
Kafka Logs Plugin (kafkalogs): 0.1.6
Kanboard Plugin (kanboard): 1.5.11
Kubernetes plugin (kubernetes): 1.29.2
Kubernetes Client API Plugin (kubernetes-client-api): 4.13.2-1
Kubernetes Credentials Plugin (kubernetes-credentials): 0.8.0
Kubernetes Credentials Provider (kubernetes-credentials-provider): 0.16
AWS Lambda Test Runner (lambda-test-runner): 0.1.0
LDAP Plugin (ldap): 2.4
Live Screenshot (livescreenshot): 1.4.5
Lockable Resources plugin (lockable-resources): 2.10
Mail Watcher Plugin (mail-watcher-plugin): 1.16
Mailer Plugin (mailer): 1.33
Managed Scripts (managed-scripts): 1.5.4
MapDB API Plugin (mapdb-api): 1.0.9.0
Mask Passwords Plugin (mask-passwords): 3.0
Matrix Authorization Strategy Plugin (matrix-auth): 2.6.5
Matrix Combinations Plugin (matrix-combinations-parameter): 1.3.1
Matrix Groovy Execution Strategy Plugin (matrix-groovy-execution-strategy): 1.0.7
Matrix Project Plugin (matrix-project): 1.18
Maven Integration plugin (maven-plugin): 3.10
Mercurial plugin (mercurial): 2.14
Metrics Plugin (metrics): 4.0.2.7
Mission Control Plugin (mission-control-view): 0.9.16
Modern Status (modernstatus): 1.3
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
mean time to repair plugin (mttr): 1.1
Multi-Branch Project Plugin (DEPRECATED) (multi-branch-project-plugin): 0.7
Multiple SCMs plugin (multiple-scms): 0.6
myst-plugin (myst-plugin): 2.5.0.0
Nested View Plugin (nested-view): 1.17
next-executions (next-executions): 1.0.15
Nexus Platform Plugin (nexus-jenkins-plugin): 3.11.20210308-082521.0d183ff
Node Iterator API Plugin (node-iterator-api): 1.5.0
NodeJS Plugin (nodejs): 1.4.0
Node and Label parameter plugin (nodelabelparameter): 1.7.2
Notify.Events (notify-events): 1.3.0
OAuth Credentials plugin (oauth-credentials): 0.4
OkHttp Plugin (okhttp-api): 3.14.9
OpenShift Client Jenkins Plugin (openshift-client): 1.0.34
OpenShift Sync (openshift-sync): 1.0.45
PAM Authentication plugin (pam-auth): 1.6
Parallel Test Executor Plugin (parallel-test-executor): 1.13
Parameter Pool Plugin (parameter-pool): 1.0.3
Parameterized Scheduler (parameterized-scheduler): 0.9.2
Parameterized Trigger plugin (parameterized-trigger): 2.40
Pending Changes plugin (pending-changes): 0.3.0
Performance Plugin (performance): 3.19
Pipeline Aggregator View (pipeline-aggregator-view): 1.11
Pipeline: AWS Steps (pipeline-aws): 1.43
Pipeline: Build Step (pipeline-build-step): 2.13
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0
Pipeline GitHub Notify Step Plugin (pipeline-githubnotify-step): 1.0.5
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10
Pipeline: Input Step (pipeline-input-step): 2.12
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.2
Pipeline: Model API (pipeline-model-api): 1.8.4
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: Declarative (pipeline-model-definition): 1.8.4
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.8.4
Pipeline: Multibranch with defaults (pipeline-multibranch-defaults): 2.1
Pipeline: REST API Plugin (pipeline-rest-api): 2.19
Pipeline: Stage Step (pipeline-stage-step): 2.5
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.8.4
Pipeline: Stage View Plugin (pipeline-stage-view): 2.19
Plain Credentials Plugin (plain-credentials): 1.7
Plot plugin (plot): 2.1.9
Plugin Utilities API Plugin (plugin-util-api): 2.0.0
Poll Mailbox Trigger Plugin (poll-mailbox-trigger-plugin): 1.026
Popper.js API Plugin (popper-api): 1.16.1-2
Post build task (postbuild-task): 1.9
PostBuildScript Plugin (postbuildscript): 2.11.0
PowerShell plugin (powershell): 1.4
Prerequisite build step plugin (prereq-buildstep): 1.1
Project Description Setter (project-description-setter): 1.2
Project statistics Plugin (project-stats-plugin): 0.4
promoted builds plugin (promoted-builds): 3.9
Infrastructure plugin for Publish Over X (publish-over): 0.22
Publish Over SSH (publish-over-ssh): 1.22
Pub-Sub "light" Bus (pubsub-light): 1.13
Python Plugin (python): 1.3
Python Wrapper Plugin (python-wrapper): 1.0.3
R Plugin (r): 0.6
Radiator View Plugin (radiatorviewplugin): 1.29
Remoting Kafka Plugin (remoting-kafka): 2.0.1
Resource Disposer Plugin (resource-disposer): 0.15
Role-based Authorization Strategy (role-strategy): 3.1
ruby-runtime (ruby-runtime): 0.12
Run Condition Plugin (run-condition): 1.5
S3 publisher plugin (s3): 0.11.6
SAML Plugin (saml): 2.0.2
SASUnit Plugin (sasunit-plugin): 1.024
SCM API Plugin (scm-api): 2.6.4
Script Security Plugin (script-security): 1.76
scripted Cloud Plugin (scripted-cloud-plugin): 0.12
Selection tasks plugin (selection-tasks-plugin): 1.0
Selenium Auto Exec Server(AES) plugin (selenium-aes): 0.5
Selenium Builder plugin (selenium-builder): 1.14
Shared Workspace (shared-workspace): 1.0.2
Shelve Project Plugin (shelve-project-plugin): 3.1
ShiningPanda Plugin (shiningpanda): 0.24
Show Build Parameters plugin (show-build-parameters): 1.0
Slack Notification Plugin (slack): 2.46
Slack Upload Plugin (slack-uploader): 1.7
Snakeyaml API Plugin (snakeyaml-api): 1.27.0
Amazon SNS Build Notifier (snsnotify): 1.13
SQLPlus Script Runner (sqlplus-script-runner): 2.0.15
Server Sent Events (SSE) Gateway Plugin (sse-gateway): 1.24
SSH plugin (ssh): 2.6.1
SSH Credentials Plugin (ssh-credentials): 1.18.1
SSH Build Agents plugin (ssh-slaves): 1.31.5
SSH2 Easy Plugin (ssh2easy): 1.4
Statistics Gatherer Plugin (statistics-gatherer): 2.0.3
Status Monitor Plugin (statusmonitor): 1.3
Structs Plugin (structs): 1.22
Subversion Plug-in (subversion): 2.14.0
Team Views (team-views): 0.9.0
Theme Manager (Incubating) (theme-manager): 0.6
Timestamper (timestamper): 1.12
Token Macro Plugin (token-macro): 2.15
Tool Environment plugin (toolenv): 1.2
Trilead API Plugin (trilead-api): 1.0.13
Active Choices Plug-in (uno-choice): 2.5.5
Variables Replace (variables-replace-plugin): 1.0.0
Variant Plugin (variant): 1.4
View Job Filters (view-job-filters): 2.3
WMI Windows Agents Plugin (windows-slaves): 1.7
Pipeline (workflow-aggregator): 2.6
Pipeline: API (workflow-api): 2.41
Pipeline: Basic Steps (workflow-basic-steps): 2.23
Pipeline: Groovy (workflow-cps): 2.90
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.18
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.38
Pipeline: Job (workflow-job): 2.40
Pipeline: Multibranch (workflow-multibranch): 2.22
Pipeline: SCM Step (workflow-scm-step): 2.12
Pipeline: Step API (workflow-step-api): 2.23
Pipeline: Supporting APIs (workflow-support): 3.8
Workspace Cleanup Plugin (ws-cleanup): 0.39
cross-platform shell plugin (xshell): 0.10

I've also resolved dependency issues on both so that they mirror each other, however, the problem master is still showing the issue when trying to save/apply any configuration to it except for credentials.

A plugin can be disabled by an administrator unchecking the Enabled checkbox in the plugin manager.

Your screen shot shows that the folders plugin is installed but disabled. The checkbox on the left needs to be clicked to enable the plugin:

markewaite - Yes I noticed that right when I sent the screenshot that it wasn't checked. After re-checking the icon and restarting Jenkins the dependency issue went away. However, we're still seeing the same issue when attempting to save changes to the master. The only thing that seems to work is saving new credentials.

clsmith4 I'm not sure what could be the difference between the working and the non-working case. You said that the same plugins and versions are used in both cases. Might be worth comparing the global configuration of the two systems (the config.xml file in \$JENKINS_HOME) in case that hints at a relevant difference.

markewaite -  I've noticed that the jobs on the problematic Jenkins master that there's a different Job UI being presented than the working Jenkins master. On the problematic master, we're presented with a limited Job section:
General > Office 365 Connector > Source Code Management > Build Triggers

There is also a Bindings section which specifies that we must select a valid file in order to complete the Job configuration(see attached screenshot).

On the working Jenkins master, we don't have a Bindings section present at all and are presented with a full Job section:
General > Office 365 Connector > Source Code Management > Build Triggers > Build Environment > Build > Post-build Actions

Do you know which plugin(s) are responsible for these differences? I've also attached screenshots of both Job differences.

clsmith4 I don't recognize that. Often you can identify the plugin that is providing something by adding one of those things to a disposable job, then click the '?' icon on the right side of one of the added items. Inside the online help it will say "from ... plugin"

clsmith4 I was able to make the save work from "Configure System" by updating to latest plugins and removing several status plugins. List of plugins I was using is in plugins-working.txt

$ wget https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/2.9.0/jenkins-plugin-manager-2.9.0.jar
$ wget https://updates.jenkins.io/stable/latest/jenkins.war
$ mv jenkins.war jenkins-2.277.1.war
$ mkdir JENKINS-65142
$ cd JENKINS-65142
$ wget https://issues.jenkins.io/secure/attachment/54298/plugins-working.txt
$ mv plugins-working.txt plugins.txt
$ java -jar ../jenkins-plugin-manager-2.9.0.jar --plugin-download-directory plugins --plugin-file plugins.txt --war ../jenkins-2.277.1.war --latest false
$ JENKINS_HOME=$(pwd) java -jar ../jenkins-2.277.1.war

I can't explain why one system works for you and another does not, but the differences in the two plugins.txt files may help your investigation.

Upgrading Artifactory plugin from 3.6.2 to 3.10.6 fixed this for us

brendanh that's a good result. That supports the instructions in the upgrade guide that users should update their plugins before the upgrade to Jenkins 2.277.1 and then update their plugins again after they upgrade to Jenkins 2.277.1. Unfortunately, as far as I can tell, the artifactory plugin is not involved in this bug report. clsmith4 has provided the list of installed plugins from his system, and the artifactory plugin is not in the list.

Performance Plugin also results in a similar problem IIRC.

In our case, broken plugins were warnings and static analysis utilities

Which were replaced by single warnings-ng plugin

I have the same issue since upgrading from 2.777.2 to 2.777. (ofc I have updated all plugins after that too).
It occurs only on jobs which have many lines in the pipeline or shell execute (i.e 10 lines no problem, having 100 and more: fails)

I got the above error then and found nothing in the logs (but I may look at the wrong places ofc)

I had to rollback to 2.777.2 and it works as before.

steadfasterx the failure behavior depends on the specific plugins and versions installed on your Jenkins.

Please open a new issue using the instructions at "How to report an issue" with very careful attention to include the precise list of plugins and their versions as listed in "What information to provide for Environment and Description".

The failure you're reporting might also be related to an incorrect configuration of a reverse proxy (like nginx or Apache) between your web browser and Jenkins.

Hi Guys,

Why this kind of a breaking change is included in the LTS version on the first place?

I'm sure that I'm not the only one using the same Jenkins instance for several years now.

Which can mount to ten's if not hundreds of installed plugins.

This update is basically putting us all on hold...I'm actually starting to realize that we may need to look an alternative to Jenkins.

If you ask me, the next LTS version should rollback this change. At least till the vast majority of plugins support this.

Thanks,

Gilad

glevy

You're advocating for Jenkins to never change, because there is no realistic way to implement any change of this scope without impacting tons of plugins, many of them unmaintained. That said, we did our best to reduce the impact.

Why this kind of a breaking change is included in the LTS version on the first place?

We specifically timed merging this change until the weekly release after the previous LTS baseline to give maintainers the most time before this went into LTS. It was in a weekly release in late October, meaning plugin developers had more than four months to fix their plugins before an LTS user was impacted. Core maintainers identified many potentially affected plugins and notified their maintainers in advance (last June!), provided relevant developer documentation, and even fixed dozens of plugins themselves. For admins, there's the upgrade guide with an overview of what we know to be impacted so they can prepare for the upgrade.

We could have put a "3" in front of the version number to make this even more clear, but that's about it – other than abandoning any efforts to modernize Jenkins altogether.

Hi danielbeck,

I get your point and understand the challenge in implementing a change of this scope.

But try to look at this from a user/customer perspective, I find it hard to justify that my "Jenkins as a whole" production environment is compromised by a "nice to have" feature. 

Hi guys,

Looks like I am entrapped in the same situation. After following the upgrade procedure from LTS 2.263.3 to 2.277.3, there are some options in the Job->Configure that are not getting saved (although the green label "saved" shows up at the top). If I retract to the last working config.xml which contains my settings of the "General" tab and "Build Triggers" (like Poll SCM etc..) and I click "save" (again, the label confirms all supposed to be well) - the settings are gone. These ones:

• Multiple SCMs plugin
• Icon Shim
• Pipeline: Declarative Agent API
• jQuery UI plugin

I got disabled and uninstalled. I've got 183 plugins installed and enabled but can't really find the culprit.

I got all plugins updated but the two, which contain breaking changes and I don't want to break 100+ jobs depending on them: 

Copy Artifact Plugin 1.43.1 and TEAM FOUNDATION SERVER PLUG-IN 5.157.1 

best wishes, I'd really appreciate a hint, 12h spent on it and counting..

domi_nik if you cannot run without Team Foundation Server plugin (and its two security issues), then you need to return to Jenkins 2.263.4. Alternately, you could consider becoming the maintainer of the TFS plugin. There are many TFS plugin users that would be very grateful to you.

This sounds somewhat similar to an issue I ran into with saving jobs configurations https://issues.jenkins.io/browse/JENKINS-65276 

In my case we did not actually need the tfs 5.157.1 plugin (we were just experimenting with it) and uninstalling that plugin resolved my problem saving maven jobs. I note that it seems really strange that the tfs plugin caused problems when trying to save jobs where it was not even used.

FTR INFRA-2751 for the discussion about republishing the TFS plugin. At the moment we do not plan to provide a fix until a new maintainer steps up

In our case the culprit was also the TFS plugin. Fortunately we were not actually using it and we could uninstall it to solve the issue.

Hello guys!

Many thanks markewaite and jmmccullough , indeed the TFS plugin is the culprit. It's just sufficient to disable it to get everything working as expected. Thank you for inviting me .. but I am not a developer .  

It's yet another issue with Jenkins, where the answer is "maintain this or fix that yourself" - well sorry, but no. Development of Jenkins in recent months is kind of a joke really. Stuck on Java 8, with Java 11 (which is old already) version leaking memory like crazy, and no newer versions support.   Half of the plugins haven't been updated for years. And then you bring issue like this (which should be reversed and considered for v.3.0 release as mentioned above) which not in any universe should be brought to regular LTS update. 

It's no wonder more and more people are moving to other software - Jenkins is becoming really hard to be taken seriously anymore.

oleg_nenashev

I was also able to pinpoint the issue to the TFS plugin as well.

But it is mandatory on our setup...

If someone could suggest the changes needed in order to fix the the table/div issues it would be greatly appreciated.

people can decide if they are willing to take the risk and build it by themselves.

An unofficial fix can buy us some time...while moving everyone to git.

Hello,

I detected that jobs that have the join trigger plugin cannot be saved.

I detected that jobs that have the join trigger plugin cannot be saved.

JENKINS-64639

I'm having the same issues saving changes to jobs. I tried creating a new job and started adding plugins one at a time to see if I could reproduce it. In my case the plugin causing my issue was the Publish Over CIFS plugin. When I added it I was able to save the job. I went back and tried to modify it and then it would not let me save/apply any changes.

On Jenkins Jenkins 2.277.4 and Publish Over CIFS 0.12.

In my case the plugin causing my issue was the Publish Over CIFS plugin

JENKINS-64236 was fixed in 0.15, released in November.

Hello,

I detected this problem with saving job in  Groovy pipeline editor. (See console log in picture - this error is appeared if I do changes in pipeline editor => page is crashed if I click on save button)

Saving configuration is working correctly if I am doing changes directly in chrome browser in my JENKINS server.

I had the similar issue, in my case it was solved after uninstall the plugin "OWASP Markup Formatter".

See the problem is - there are literally hundreds of plugins on official update center and no comprehensive list of the troublemakers. And it's not so easy to just try them all just like that. 

Because of this (and non-existent effort to make Jenkins work with up-to-date Java versions), we're looking for alternative software suite (and there are some, thankfully), as this is just silly.

I don't believe it is a plugin problem. I have a number of Jenkins servers displaying the same problem with the inability to save (Running on Windows Server 2012R2 and Windows 10).

I too tried removing various plugins and thought that I may have found the problem because I could then occasionally save, but then realised that it was actually an intermittent problem. Sometimes it does save straight away and other times not, so I reenabled the plugins and went back to the drawing board.

On 5 of the 6 Jenkins servers I am running, I am using the latest LTS version and have seen this problem on all of them, the only one that was not displaying this problem was the one using the weekly release version. So to cut a long story short, I changed one of the LTS versions (2.277.4.) to 2.295 and the problem was solved. I then tried it on a second server with exactly the same success.

So in conclusion I an convinced it is a problem with the LTS core and not any plugins. I would prefer to use the LTS versions on my productive machines, but this problem is very frustrating and I may have to switch the rest of my servers to the weekly release instead of the so called "stable" release if this bug is not fixed.

I changed one of the LTS versions (2.277.4.) to 2.295 and the problem was solved

More likely than not the 2.289.1 scheduled for release tomorrow will solve it then. Otherwise I would ask you file a new issue and provide useful information what exactly goes wrong, any log and browser console output, etc. — these catch-all issues where a dozen people experience similar but probably not the same problem and provide contradictory information are not really useful in addressing the problems encountered.

I found that going back to 2.277.1 greatly reduced the amount of saving errors (didn't have any while saving job configs and only had some error popups while saving global settings, but the settings themselves would save correctly). 

Hi peejay,

Do you have the TFS plugin installed on one of the environment by any chance?

danielbeck

"More likely than not the 2.289.1 scheduled for release tomorrow will solve it then."

We upgraded to v2.289.1 on Windows 2012 Servers and the issue still persists.

Changing some portions of job configurations will allow Apply or Save process to work, but changing others, such as Git plugin section causing no action when Apply is clicked, and error shown when Save is clicked:

Thanks,

John Burrows

My experience with 2.289.1 seems to be exactly the same as John Burrows'. I found that I could generally save the general settings, but the random first job on my test instance would fail with the same error as above...

Replay and Upload file also affected

Updated to 2.289.1, but still the same issue.

One more information: The problem also appears when using "replay for Pipelines" or "upload a file for Job run". So basically always when there is a HTTP POST operation involved.

Update: We've rolled back to 2.277.1 for now. Problem is not existing there.

jburrows

We upgraded to v2.289.1 on Windows 2012 Servers and the issue still persists.

Can you confirm you have no problems like this in 2.295, like the person I was responding to?

danielbeck my issue with saving configuration of a pipeline is finally gone in v2.297 !

Before that I was able to narrow it down between 2 versions:

2.277.2: no issues
2.277.3 and later: issues occurs

so maybe introduced by https://www.jenkins.io/security/advisory/2021-04-20/ .
no idea if that helps someone and I will test further if v2.297 fixes all my related issues

Thanks for the response steadfasterx and I'm happy it works for you again. Based on the versions provided and the specific error you showed earlier, that seems likely. Might be useful if you're able to narrow down the version that fixed it, starting with the releases updating Winstone/Jetty (2.290, 2.296) and their immediate predecessors.

FWIW this nicely illustrates the difficulty in investigating such reports from the maintainer side – the last known good version for you is later than the first bad version in the original report, i.e. different people report (likely) different problems in the same issue. Without direct access to your infra or other ways to reliably reproduce the problem from scratch, we have nothing, because what information is provided is contradictory. This isn't intended as a slight to anyone involved here, just explaining how this looks from the other side.

While troubleshooted an instance impacted by this when upgrading from 2.277.2 to 2.277.3, we found the following issues Encrypted buffer max length exceeded when enabling FINE level for org.eclipse.jetty:

2021-06-08 19:52:47.334+0000 [id=1667607]	FINE	o.e.j.i.s.SslConnection$DecryptedEndPoint#handleException: DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=412/5000} stored fill exception
javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:735)
	at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:342)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(Thread.java:748)
2021-06-08 19:52:47.335+0000 [id=1667607]	FINE	o.e.j.i.s.SslConnection$DecryptedEndPoint#fill: <fill f=-2 uf=false SslConnection@72088a32::SocketChannelEndPoint@73a710b0{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=0/5000}{io=0/0,kio=0,kro=1}->SslConnection@72088a32{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP:SCRUBBED_PORT,OPEN,fill=-,flush=-,to=412/5000}=>HttpConnection@27093e32[p=HttpParser{s=CONTENT,0 of 21594},g=HttpGenerator@6c5eb404{s=START}]=>HttpChannelOverHttp@35bf1d41{s=HttpChannelState@3125348b{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=48,c=false/false,a=IDLE,uri=//jenkins.example.com/job/-freestyle-/configSubmit,age=2}
2021-06-08 19:52:47.335+0000 [id=1667607]	FINE	o.e.jetty.io.AbstractEndPoint#close: close(javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded) DecryptedEndPoint@7acc808f{l=/SCRUBBED_IP:443,r=/SCRUBBED_IP

Could those problems starting from 2.277.3 be actually caused by https://issues.jenkins.io/browse/JENKINS-65280 / https://github.com/eclipse/jetty.project/pull/6073 and not table to div ?

see https://issues.jenkins.io/browse/JENKINS-65624

steadfasterx as you can easily reproduce.  maybe (not sure) you could try using `-Djsse.SSLEngine.acceptLargeFragments=true`  and tell us if it fix the issue? (but Jenkins weekly should definitely fix it)

olamy We tried this Java option Djsse.SSLEngine.acceptLargeFragments=true with version 2.289.1, but unfortunately we still have our initial problem. For us this shows as "secure connection failed", but I guess that's browser specific.

I can confirm this issue has been solved in the new LTS version 2.289.2.

After upgrading to 2.289.2 the save button works as expected, as 2.277.1 did. Versions between these two versions had the problem from this issue.

We are using 2.289.2 and suffering this problem.  Save/Apply fails to work.  Hitting "Apply" in FireFox throws up a blank "Error" box, whereas all other browsers do nothing.

Have uninstalled any suspect plugins from the comments here and from this list:

https://issues.jenkins.io/secure/Dashboard.jspa?selectPageId=20741

But still not able save/apply on any jobs.  This is a pretty big deal.

garen I suspect that you're describing a different set of conditions than the conditions in the original bug report. I suspect you're describing a different set of conditions than those in the comments from others in this issue report.

Usually, the best approach is to perform the steps described in the Jenkins 2.277.1 upgrade guide and in the Jenkins 2.277.1 upgrade webinar so that you can identify the plugin or plugins causing the issue yourself. Those steps are:

• Upgrade all plugins to latest releases
• Remove plugins with security issues (like TFS)
• Remove deprecated plugins (like warnings, checkstyle, pmd, etc.)
• Remove unused plugins
• Remove plugins with known issues in tables to divs

If those steps are not enough to resolve the issue, see the instructions at https://www.jenkins.io/doc/developer/views/table-to-div-migration/#identifying-the-broken-plugin .

If the steps on that page are not enough to resolve the issue, then open a new issue report with the precise list of plugin names and versions, as generated by https://www.jenkins.io/doc/book/system-administration/diagnosing-errors/#how-to-report-a-bug

garen that plugin is one of the plugins with known issues in tables to divs. The issue is described in JENKINS-65955. That's why the guidelines recommend removing plugins with known issues in tables to divs.

Same problem with release 2.277.4.
Does not systematicaly occurs. But when facing the problem, reloading the page with CTRL+F5 allows to be again able to save modifications. Just a workaround, but maybe a clue...

nitinqa and lviolon please refer to my earlier comment for the steps you need to take to investigate your specific issue. When you declare that you see the same problem, but don't provide the details that confirm your exact plugin names and versions, you're not providing enough information for others to assist.

You're right Mark. The aim of my comment was just for the moment to share the workaround.
I read your comment and looked to the dashboard about divs issues.
Although I have a large amount of plugins (more than 100), only job-dsl-plugin and publish-over-ssh-plugin are in the dashboard. Nevertheless, I have too much plugins to investigate a non systematic problem.
So I will start with removing as much plugins as possible.

same problem for us as well..WE have installed Jnekins 2.303.1 as the changelog for the versions said that this issue is fixed, however it is not fixed. I am attaching(Installed Plugins SJ.docx) our list of installed plugins here. I went through the suggested steps over different tickets in regards to this issue and followed suggestion but no resolution so far. This save/apply issue only occurs to our Maven projects but other freestyle pipelines are working fine. I also disabled Job Mail Direct Plugin and other plugins suggested in the known issues table but no luck.

sj you said:

WE have installed Jnekins 2.303.1 as the changelog for the versions said that this issue is fixed, however it is not fixed

Please provide the details of the location in the 2.303.1 changelog that claims "this issue is fixed". There is no reference to this issue in the changelog or in the upgrade guide.

The list of plugins that you uploaded ( Installed Plugins. SJ.docx ) is unusable for others who might want to help you. If you'd like help from others, please follow the instructions at "How to report an issue", especially the part that describes how to provide a precise list of the plugins you have installed and their versions. The list you provided uses a format that is difficult to read, does not have version numbers, and uses the display name of the plugin instead of the plugin identifier.

Based on the list, I assume that "Build Time Blame" plugin in your list will have tables to divs issues, since https://github.com/jenkinsci/build-time-blame-plugin/search?q=%3Ctable shows that it uses table markup for at least one of its actions. Remove that plugin and see if it helps .

The checkstyle plugin has reached end of life. Remove it.

The job direct mail plugin is known to have tables to divs issues. Remove it.

You said:

I went through the suggested steps over different tickets in regards to this issue and followed suggestion but no resolution so far.

As far as I can tell, your installed list of plugins indicates you did not follow the instructions in other issue reports or the instructions in this issue report.

To reiterate, those instructions are:

Perform the steps described in the Jenkins 2.277.1 upgrade guide and in the Jenkins 2.277.1 upgrade webinar so that you can identify the plugin or plugins causing the issue yourself. Those steps are:

• Upgrade all plugins to latest releases
• Remove plugins with security issues (like TFS)
• Remove deprecated plugins (like checkstyle, mulitple scms, pmd, etc.)
• Remove unused plugins (like CVS, Subversion, Ivy, External Monitor Job Type, etc.)
• Remove plugins with known issues in tables to divs (like job direct mail plugin)

If those steps are not enough to resolve the issue, see the instructions at https://www.jenkins.io/doc/developer/views/table-to-div-migration/#identifying-the-broken-plugin .

If the steps on that page are not enough to resolve the issue, then open a new issue report with the precise list of plugin names and versions, as generated by https://www.jenkins.io/doc/book/system-administration/diagnosing-errors/#how-to-report-a-bug

markewaite - As a quick update, we managed to get past this issue successfully by completely removing all references/traces to the coding-webhook plugin and by also upgrading to Jenkins version 2.303.2. I would've replied earlier, but I wanted to confirm that the upgrade from 2.263.4 to 2.303.2 fixed the Global configuration save/apply binding issues for us. (i.e. - Global Security/Configure System/etc.)

Once we were able to generate a complete list of plugins that were unique only to the problematic Jenkins master, we installed those same plugins on a separate test Jenkins master and starting uninstalling/removing those specific plugins one by one. By completely removing all references of the coding-webhook plugin, we were able to successfully save/apply new binding changes to all Jenkins jobs again.

I should note that the coding-webhook plugin was previously uninstalled during the earlier plugin troubleshooting efforts, but the plugin's homedir under ${JENKINS_HOME/plugins} was still present along with other recursive references to the plugin's .JPI file under ${JENKINS_HOME}. The strange part is that the actual .JPI file was completely removed during the initial plugin uninstall and wasn't found anywhere under the ${JENKINS_HOME} which was a red flag. Another giveaway for us was intermittent error messages about the coding-webhook plugin were observed from time-to-time while refreshing the installed plugins list in the Jenkins UI after a restart of Jenkins. It wasn't a continuous error message and would go away after a certain amount of time, but would show up at the bottom of page after restarting Jenkins and then immediately refreshing the installed plugins list afterwards.

While removing all references of the coding-webhook plugin resolved the save/apply Jenkins jobs issue, we had to wait until upgrading to version 2.303.2 to officially confirm that the global save/apply binding issues that we encountered had been resolved for us.

Thank you markewaite and everyone else that took the time to reply for helping to guide us with the direction on troubleshooting, diagnosing and ultimately getting us past these issues!

I'm glad to hear it. The coding webhook plugin has been deprecated in favor of the coding.net Jenkins integration. Since that site is in a language I do not read, I assume that it describes something about Jenkins and their integration with Jenkins.

Closing as "won't fix" because the coding-webhook plugin is deprecated.

this problem might be similar to this one dealing with vsphere plugin:
https://issues.jenkins.io/browse/JENKINS-67234