Hi rocha_stratovan, please provide the list of plugins installed on your system as suggested at "How to report a bug".

Please confirm that you followed the instructions in the Jenkins 2.277.1 upgrade guide, especially that you updated your plugins after installing Jenkins 2.277.1. If you did not upgrade your plugins, please do so, then report if the issue remains.

Please check the list of known broken tables-to-divs plugins dashboard. If you're using any of these plugins, please disable them (or wait to upgrade) and upvote the issue.

If you are not using any of the known broken plugins from that list, you can minimize your Jenkins installation to identify which plugin is the problem following the instructions here to disable plugins in groups of 10 until you determine which plugin is at fault. Then, file a new Jira issue, including steps to reproduce and screenshots and label it with 'tables-to-divs-regression'.

Hello basil.

I identified the matrix plugin version in the environment section. Are you saying you want a list of all of the plugins?

Assuming you want the entire list, this is the the plugin list (reverted back to the 2.363.4 stage. This is a highly used system and I couldn't leave it in a broken state. The only differrence between this list now and when the problem occurred was that I fell back to plugin versions that work with 2.363.4

Jenkins: 2.263.4
OS: Windows 8.1 - 6.3
—

1. ace-editor:1.1
2. ant:1.11
3. antisamy-markup-formatter:2.1
4. apache-httpcomponents-client-4-api:4.5.13-1.0
5. authentication-tokens:1.4
6. bootstrap4-api:4.6.0-2
7. bouncycastle-api:2.20
8. branch-api:2.6.2
9. build-blocker-plugin:1.7.7
10. build-pipeline-plugin:1.5.8
11. build-publisher:1.22
12. build-timeout:1.20
13. build-with-parameters:1.5
14. buildresult-trigger:0.17
15. built-on-column:1.1
16. call-remote-job-plugin:1.0.21
17. checks-api:1.6.1
18. cloudbees-folder:6.15
19. command-launcher:1.5
20. conditional-buildstep:1.4.1
21. credentials-binding:1.24
22. credentials:2.3.15
23. cvs:2.18
24. description-setter:1.10
25. display-url-api:2.3.4
26. docker-commons:1.17
27. docker-workflow:1.26
28. doclinks:0.6.1
29. doxygen:0.18
30. durable-task:1.35
31. echarts-api:5.0.1-1
32. email-ext:2.82
33. envfile:1.2
34. envinject-api:1.7
35. envinject:2.3.0
36. extended-choice-parameter:0.82
37. extended-read-permission:3.2
38. external-monitor-job:1.7
39. folder-properties:1.2.1
40. font-awesome-api:5.15.2-2
41. git-client:3.7.0
42. git-server:1.9
43. git:4.7.0
44. greenballs:1.15.1
45. handlebars:3.0.8
46. icon-shim:2.0.3
47. jackson2-api:2.12.1
48. javadoc:1.6
49. jdk-tool:1.5
50. jenkins-multijob-plugin:1.36
51. job-import-plugin:3.4
52. job-restrictions:0.8
53. jquery-detached:1.2.1
54. jquery:1.12.4-1
55. jquery3-api:3.5.1-3
56. jsch:0.1.55.2
57. junit:1.49
58. ldap:1.26
59. mailer:1.34
60. mapdb-api:1.0.9.0
61. matrix-auth:2.6.6
62. matrix-combinations-parameter:1.3.1
63. matrix-project:1.18
64. maven-plugin:3.8
65. momentjs:1.1.1
66. nodelabelparameter:1.7.2
67. p4:1.11.3
68. pam-auth:1.6
69. Parameterized-Remote-Trigger:3.1.5.1
70. parameterized-trigger:2.39
71. perforce:1.3.36
72. pipeline-input-step:2.12
73. pipeline-model-api:1.8.4
74. pipeline-model-definition:1.8.4
75. pipeline-model-extensions:1.8.4
76. pipeline-stage-step:2.5
77. pipeline-stage-tags-metadata:1.8.4
78. plain-credentials:1.7
79. plugin-usage-plugin:1.2
80. plugin-util-api:2.1.0
81. popper-api:1.16.1-2
82. rebuild:1.32
83. remote-jobs-view-plugin:0.0.3
84. resource-disposer:0.15
85. role-strategy:3.1.1
86. run-condition:1.5
87. scm-api:2.6.4
88. script-security:1.76
89. shelve-project-plugin:3.1
90. snakeyaml-api:1.27.0
91. ssh-credentials:1.18.1
92. ssh-slaves:1.31.5
93. ssh:2.6.1
94. statusmonitor:1.3
95. strawboss:1.3
96. structs:1.22
97. subversion:2.14.0
98. text-finder:1.15
99. timestamper:1.12
100. token-macro:2.13
101. translation:1.16
102. trilead-api:1.0.13
103. variant:1.4
104. windows-slaves:1.7
105. workflow-api:2.42
106. workflow-basic-steps:2.23
107. workflow-cps-global-lib:2.18
108. workflow-cps:2.90
109. workflow-durable-task-step:2.38
110. workflow-job:2.40
111. workflow-multibranch:2.22
112. workflow-scm-step:2.12
113. workflow-step-api:2.23
114. workflow-support:3.8
115. ws-cleanup:0.39

     ---

The upgrade steps I followed where:

1. upgrade Jenkins from 2.363.4 to 2.277.1
2. upgraded all compatible plugins

After that I attempted to edit a Matrix Configuration job and observed the problem that was reported.

I looked at the issues reported in the list of known broken tables-to-divs plugins dashboard. I don't use any of these reported plugins.

I will look into the detailed debugging steps that you've recommended.

basil,

I tried the minimal Jenkins steps from "Some plugins are known to be broken"

I have the problems on a Windows server, and those instructions are only relevant to a Linux server. Regardless, I did the actions on a Linux server but the problem doesn't reproduce.

Is there a similar method that can be done for a minimal windows server?

basil,

 I figured out a way to do it for windows.

1. mkdir <desktop>\JenkinsIssue
2. Copy the .war files downloaded for Linux to Windows::<desktop>\JenkinsIssue
3. start command prompt in <desktop>\JenkinsIssue
4. set JENKINS_HOME=%CD%\HOME
5. java -jar jenkins-plugin-manager-2.2.0.jar -f plugins.txt -d %JENKINS_HOME%/plugins --war jenkins.war
6. java -jar jenkins.war

Unfortunately the problem does not occur when I do this on Windows either. Is there some other way to debug/drill deeper into this issue?

Hi rocha_stratovan, I'm glad you were able to figure out how to create a minimal Jenkins installation on Windows. In order to solve this problem, we're going to first have to figure out a way to reproduce the issue from scratch on a minimal Jenkins installation. So let's make sure your minimal Jenkins installation is running exactly the same version of Jenkins and plugins as the broken production instance was. Make sure you're using the 2.277.1 WAR from https://get.jenkins.io/war-stable/2.277.1/jenkins.war and that you've upgraded all plugins to the latest version within your minimal Jenkins install.

At that point, try to come up with a series of steps that reproduce the problem. This is key. Start by creating a simple job and then start adding configuration from your production installation until the minimal installation reproduces the problem. At that point please report the steps to reproduce and then begin the exercise of disabling/enabling plugins until the problematic one is discovered.

basil

Thank you for you patience/guidance/tutelage.

I found the failing plugin: Perforce plugin

I reproduced my problem with a copy of the production. The steps for this as well as the method for zeroing in on the failing plugin are described below.

1. Copy production Jenkins to temporary machine
   
   We have a master/slave scenario, but the slaves are started with SSH and so the temporary setup won't actually connect to the slave machine because it doesn't have the SH keys.
   
   
   1. Start git bash on windows machine
      
      
   2. navigate to my jenkins root
      
      
   3. $ tar -cvf /c/temp/jenkins.tar --exclude="/builds/" --exclude="/.git/" --exclude="/workspace/" --exclude="-Doc/" --exclude="shelvedProjects/" *
      
      
   4. Copy tarball to test machine
      
      
   5. Detar
      
      
2. Start jenkins 2.277.1
   java -jar jenkins.war
   
   
3. Upgrade all available/compatible plugins
   
   
4. Attempt to create freestyle project – FAILS
   
   
5. Disable all plugins by touching <plugin>,jpi.disabled
   
   
6. Attempt to create freestyle project – SUCCESS
   
   
7. Enable plugins that begin with "t-z" – SUCCESS
   
   
8. Enable plugins that begin with "p-s" – FAIL
   
   
9. Disable plugins that begin with "p" – SUCCESS
   
   

   ./p4.jpi.disabled
   ./pam-auth.jpi.disabled
   ./Parameterized-Remote-Trigger.jpi.disabled
   ./parameterized-trigger.jpi.disabled
   ./perforce.jpi.disabled
   ./pipeline-input-step.jpi.disabled
   ./pipeline-model-api.jpi.disabled
   ./pipeline-model-definition.jpi.disabled
   ./pipeline-model-extensions.jpi.disabled
   ./pipeline-stage-step.jpi.disabled
   ./pipeline-stage-tags-metadata.jpi.disabled
   ./plain-credentials.jpi.disabled
   ./plugin-usage-plugin.jpi.disabled
   ./plugin-util-api.jpi.disabled
   ./popper-api.jpi.disabled
   

   
   
   

10. Enable plugins that begin with "pipeline" – SUCCESS

    ./p4.jpi.disabled
    ./pam-auth.jpi.disabled
    ./Parameterized-Remote-Trigger.jpi.disabled
    ./parameterized-trigger.jpi.disabled
    ./perforce.jpi.disabled
    ./plain-credentials.jpi.disabled
    ./plugin-usage-plugin.jpi.disabled
    ./plugin-util-api.jpi.disabled
    ./popper-api.jpi.disabled
    

    
    
    

11. Enable plugins that begin with "pl" – SUCCESS

    ./p4.jpi.disabled
    ./pam-auth.jpi.disabled
    ./Parameterized-Remote-Trigger.jpi.disabled
    ./parameterized-trigger.jpi.disabled
    ./perforce.jpi.disabled
    ./popper-api.jpi.disabled
    

    
    
    

12. Enable plugins that begin with "param" – SUCCESS

    ./p4.jpi.disabled
    ./pam-auth.jpi.disabled
    ./perforce.jpi.disabled
    ./popper-api.jpi.disabled
    

    
    
    

13. Enable plugins that begin with "p4" – SUCCESS

    ./pam-auth.jpi.disabled
    ./perforce.jpi.disabled
    ./popper-api.jpi.disabled
    

    
    
    

14. Enable plugins that begin with "perforce" – fail FOUND ONE

    ./pam-auth.jpi.disabled
    ./popper-api.jpi.disabled
    

    
    
    

15. Disable plugins that begin with "perforce"
    Enable the other plugins that start with "p" – SUCCESS
    
    
16. Enable all plugins except for "perforce.jpi" – SUCCESS

Great result! So we know that perforce-plugin is broken with tables-to-divs. The GitHub page says

Note: This has been deprecated in favor of the pure-java p4-plugin

so I would recommend simply disabling/uninstalling this plugin from your Jenkins installation. I can't find a component for perforce-plugin but I have moved this issue to the p4-plugin component as it is the closest component I can think of and added the tables-to-divs-regression label in the hope that this will help other users in the future.

Thank you basil.

After investigation I found that we no longer use the Perforce plugin. We had switched to p4 plugin awhile back – but the legacy plugin hadn't been deleted.

We are now able to proceed with the upgrade.

Cheers!