-
Bug
-
Resolution: Unresolved
-
Blocker
-
None
-
Jenkins version: 2.387.1 running on Amazon Linux 2
Hi everyone,
I'm opening this issue because I have the same problem described in this issue: https://issues.jenkins.io/browse/JENKINS-59145.
As in the above issue, the problem started when we updated both Jenkins to the latest version and all the plugins. Unfortunately, I don't have the list of plugins updated.
The effect is the same: the SCM configuration is always empty. Everything is working correctly under the hood, but the UI shows the SCM configuration as empty.
The root of the issue JENKINS-59145 was the plugin "permissive script security" (https://plugins.jenkins.io/permissive-script-security/), but checking the installed plugins, we don't use it.
As an attachment, I'm adding a config.xml ok to one of the pipelines that report this problem. Checking a lot of other jobs, I assume that the issue is present to all the pipelines. [^config.xml]
To replicate the issue I think is enough to deploy a new Jenkins server with the same plugin installed.
Jenkins has been updated to the latest LTS version: 2.387.1, and all plugins have been updated to the latest version.
Here is the list of plugins, the Jenkins, OS and Java version created as reported here
Jenkins: 2.387.1 OS: Linux - 4.14.294-220.533.amzn2.x86_64 Java: 11.0.18 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- amazon-ecr:1.114.vfd22430621f5 ansicolor:1.0.2 ant:481.v7b_09e538fcca antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 atlassian-jira-software-cloud:2.0.7 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.406-370.v8f993c987059 aws-java-sdk-cloudformation:1.12.406-370.v8f993c987059 aws-java-sdk-codebuild:1.12.406-370.v8f993c987059 aws-java-sdk-ec2:1.12.406-370.v8f993c987059 aws-java-sdk-ecr:1.12.406-370.v8f993c987059 aws-java-sdk-ecs:1.12.406-370.v8f993c987059 aws-java-sdk-efs:1.12.406-370.v8f993c987059 aws-java-sdk-elasticbeanstalk:1.12.406-370.v8f993c987059 aws-java-sdk-iam:1.12.406-370.v8f993c987059 aws-java-sdk-logs:1.12.406-370.v8f993c987059 aws-java-sdk-minimal:1.12.406-370.v8f993c987059 aws-java-sdk-sns:1.12.406-370.v8f993c987059 aws-java-sdk-sqs:1.12.406-370.v8f993c987059 aws-java-sdk-ssm:1.12.406-370.v8f993c987059 aws-secrets-manager-credentials-provider:1.202.ve0ec0c17611c aws-secrets-manager-secret-source:1.62.v120c8e5058ee bitbucket-build-status-notifier:1.4.2 bitbucket-oauth:0.13 bitbucket-push-and-pull-request:2.8.3 blueocean-autofavorite:1.2.5 blueocean-commons:1.27.3 blueocean-config:1.27.3 blueocean-core-js:1.27.3 blueocean-dashboard:1.27.3 blueocean-events:1.27.3 blueocean-git-pipeline:1.27.3 blueocean-i18n:1.27.3 blueocean-jwt:1.27.3 blueocean-personalization:1.27.3 blueocean-pipeline-api-impl:1.27.3 blueocean-pipeline-editor:1.27.3 blueocean-pipeline-scm-api:1.27.3 blueocean-rest:1.27.3 blueocean-rest-impl:1.27.3 blueocean-web:1.27.3 bootstrap5-api:5.2.2-1 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 build-blocker-plugin:1.7.8 build-monitor-plugin:1.14-653.va_1c684a_30b_ff build-name-setter:2.2.0 build-notifications:1.5.0 build-pipeline-plugin:1.5.8 build-timeout:1.28 build-user-vars-plugin:1.9 build-with-parameters:76.v9382db_f78962 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:2.0.0 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.815.v0dd5a_cb_40e0e code-coverage-api:4.0.2 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1613.vb_e54b_445ed6f configurationslicing:430.v966357576543 credentials:1224.vc23ca_a_9a_2cb_0 credentials-binding:523.vd859a_4b_122e6 dark-theme:315.va_22e7d692ea_a dashboard-view:2.472.v9ff2a_e6a_c529 data-tables-api:1.13.3-2 dependency-check-jenkins-plugin:5.4.0 disk-usage:0.28 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c docker-java-api:3.2.13-68.va_875df25a_b_45 docker-plugin:1.3.0 docker-workflow:563.vd5d2e5c4007f durable-task:504.vb10d1ae5ba2f echarts-api:5.4.0-2 email-ext:2.95 envinject:2.901.v0038b_6471582 envinject-api:1.199.v3ce31253ed13 environment-script:1.2.6 extended-choice-parameter:359.v35dcfdd0c20d extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:203.v683c09d993b_9 favorite:2.4.1 font-awesome-api:6.3.0-1 forensics-api:2.0.1 git:5.0.0 git-client:4.2.0 git-parameter:0.9.18 git-server:99.va_0826a_b_cdfa_d github:1.37.0 github-api:1.303-417.ve35d9dd78549 github-branch-source:1701.v00cc8184df93 global-build-stats:244.v27c8a_2e50a_34 google-login:1.7 gradle:2.3.2 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 ivy:2.4 jackson2-api:2.14.2-319.v37853346a_229 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.27.3 jersey2-api:2.38-1 jira:3.9 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-2 job-dsl:1.83 jobConfigHistory:1207.vd28a_54732f92 jquery:1.12.4-1 jquery3-api:3.6.3-1 jsch:0.1.55.61.va_e9ee26616e7 junit:1189.v1b_e593637fa_e ldap:671.v2a_9192a_7419d lockable-resources:1131.vb_7c3d377e723 mailer:448.v5b_97805e3767 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:150.vf80d33113e80 matrix-auth:3.1.6 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.21 mercurial:1260.vdfb_723cdcc81 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a multiple-scms:0.8 nodelabelparameter:1.11.0 okhttp-api:4.10.0-132.v7a_7b_91cef39c pam-auth:1.10 parameterized-scheduler:1.2 parameterized-trigger:2.45 pipeline-aggregator-view:1.15 pipeline-aws:1.43 pipeline-build-step:487.va_823138eee8b_ pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:629.vb_5627b_ee2104 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2123.va_31cb_3b_80ef8 pipeline-model-definition:2.2123.va_31cb_3b_80ef8 pipeline-model-extensions:2.2123.va_31cb_3b_80ef8 pipeline-rest-api:2.31 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2123.va_31cb_3b_80ef8 pipeline-stage-view:2.31 pipeline-utility-steps:2.15.1 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.1.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-3 project-stats-plugin:23.v47fee1f77b_84 pubsub-light:1.17 python:1.3 resource-disposer:0.21 role-strategy:587.588.v850a_20a_30162 run-condition:1.5 saml:4.385.v4dea_91565e9d scm-api:631.v9143df5b_e4a_a scmskip:1.0.3 script-security:1229.v4880b_b_e905a_6 shiningpanda:0.24 simple-theme-plugin:146.v0e67db_a_9052e slack:631.v40deea_40323b snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 spotinst:2.2.11 sse-gateway:1.26 ssh-agent:327.v230ecd01f86f ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.1 text-finder:1.23 theme-manager:1.6 throttle-concurrents:2.12 timestamper:1.22 token-macro:321.vd7cc1f2a_52c8 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb veracode-jenkins-plugin:20.2.6.1 view-job-filters:364.v48a_33389553d workflow-aggregator:596.v8c21c963d92d workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1010.vf7a_b_98e847c1 workflow-cps:3641.vf58904a_b_b_5d8 workflow-durable-task-step:1234.v019404b_3832a workflow-job:1284.v2fe8ed4573d4 workflow-multibranch:733.v109046189126 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.44
[JENKINS-70862] SCM configuration no longer visible due to spotinst plugin
Giacomo
added a comment - Hi markewaite, thank you for your response! I added the information requested. Do you think more info are necessary?
Thank you again
Giacomo
added a comment - Hi markewaite , thank you for your response! I added the information requested. Do you think more info are necessary?
Thank you again The job definition [^config.xml] that you uploaded has more than one flow-definition in the file and ends with an extra '[' character. I've assumed that the content of that file as uploaded is not the content of the file that you are actually using in Jenkins. I've removed the trailing '[' character and retained only the first copy of the file in my experiments.
I've duplicated the problem based on your plugins and have made a few steps reducing the number of plugins without resolving the issue. I'll leave it to you to identify exactly which plugin is causing the issue. You know better than others which plugins can safely be removed from your system.
JENKINS-70862.git-bundle
is a git bundle of the git repository that I used to duplicate the problem. You're welcome to use the same technique to create a temporary Jenkins instance so that you can locate the specific plugin that causes the issue. I removed all the plugins with security warnings and all the deprecated plugins without resolving the issue. I removed the blue ocean plugins without resolving the issue. I'm out of time for this exploration, so I'll leave it to you to continue the search for the plugin.
I confirmed that the set of plugins I install on my system does not show the issue. You could compare my plugins.txt and your plugin list as another way of identifying plugins to remove as you search for the specific plugin that causes the issue.
Giacomo
added a comment - Hi markewaite, thank you for working hard on this; appreciated! I don't understand if you could replicate the issue in another Jenkins installation and, if so, which is the minimum set of plugins you tried that replicated the issue.
Thank you again for your hard work
Giacomo
added a comment - Hi markewaite , thank you for working hard on this; appreciated! I don't understand if you could replicate the issue in another Jenkins installation and, if so, which is the minimum set of plugins you tried that replicated the issue.
Thank you again for your hard work Yes, I replicated the issue in a fresh copy of Jenkins 2.387.1 as described in the git repository stored in the JENKINS-70862.git-bundle . You can download the JENKINS-70862.git-bundle to a local computer, create a local git repository, and import that bundle into your local git repository.
To duplicate the problem, I ran the following command from inside that git repository:
$ bash ./README # Yes, it is a little strange to have an executable README
That script will download Jenkins 2.387.1, download the plugin installation manager tool, download the plugins listed in the plugins.txt file, and start JENKINS with those plugins installed.
Answer the setup prompts (temporary password), then choose to install no plugins (since they are all installed from plugins.txt already), and you can see the two sample jobs already defined (yours and mine).;
Giacomo
added a comment - Hi markewaite, it seems the bundle is empty... How should I use it?
This is what I have done:
# Download the JENKINS-70862.git-bundle file in the ~/Desktop/jenkins/bundle folder cd ~/Desktop/jenkins/bundle git init Initialized empty Git repository in /Users/giacomo/Desktop/jenkins/bundle/.git/ git bundle unbundle JENKINS-70862.git-bundle Unbundling objects: 100% (25/25), 7.41 KiB | 7.42 MiB/s, done. Resolving deltas: 100% (8/8), done. 6bdf4671b0b8d81a1c31abf851a68cad9a4fdad8 refs/heads/master 6bdf4671b0b8d81a1c31abf851a68cad9a4fdad8 HEAD ls -a . .. .git JENKINS-70862.git-bundle
Where the README file it is locate?
Giacomo
added a comment - Hi markewaite , it seems the bundle is empty... How should I use it?
This is what I have done:
# Download the JENKINS-70862.git-bundle file in the ~/Desktop/jenkins/bundle folder
cd ~/Desktop/jenkins/bundle
git init
Initialized empty Git repository in /Users/giacomo/Desktop/jenkins/bundle/.git/
git bundle unbundle JENKINS-70862.git-bundle
Unbundling objects: 100% (25/25), 7.41 KiB | 7.42 MiB/s, done.
Resolving deltas: 100% (8/8), done.
6bdf4671b0b8d81a1c31abf851a68cad9a4fdad8 refs/heads/master
6bdf4671b0b8d81a1c31abf851a68cad9a4fdad8 HEAD
ls -a
. .. .git JENKINS-70862.git-bundle
Where the README file it is locate?
The git bundle directions seem to indicate that the easiest way to unpack and see the contents of that bundle is with the command:
$ git clone -b master ~/JENKINS-70862.git-bundle JENKINS-70862-experiments-dir
That will use the bundle to create a new directory named JENKINS-70862-experiments-dir with the README, the plugins.txt, the jobs, and the history of those files
Hi markewaite,
I made a few changes in the README file:
- Java installation folder
- Fix a typo in the download endpoint of the jenkins .war file
After those were fixed, I was able to start the Jenkins instance and disable plugins one by one, I found the one that was causing the issue. The plugin https://plugins.jenkins.io/spotinst/releases/, release 2.2.10, introduced something that caused the issue. Rolling back to version 2.2.9, everything started working fine!
Now I'm reporting the issue to the plugin team to understand how they could fix it.
I'm not sure how it works in this case, but from my point of view, we can close this issue.
Thank you again for the help
Giacomo
added a comment - - edited Hi markewaite ,
I made a few changes in the README file:
Java installation folder
Fix a typo in the download endpoint of the jenkins .war file
After those were fixed, I was able to start the Jenkins instance and disable plugins one by one, I found the one that was causing the issue. The plugin https://plugins.jenkins.io/spotinst/releases/ , release 2.2.10, introduced something that caused the issue. Rolling back to version 2.2.9, everything started working fine!
Now I'm reporting the issue to the plugin team to understand how they could fix it.
I'm not sure how it works in this case, but from my point of view, we can close this issue.
Thank you again for the help 
Thanks for providing the list of plugins and their versions. You'll be more likely to receive help from others if you provide the list of plugins and their versions in a format that can be reused with the plugin installation manager tool. Instructions to generate that list are available in "How to report an issue"
That report will also provide some additional information about the Java version being used and the operating system kernel version.
Can you also provide steps that will allow others to duplicate the problem from a fresh Jenkins installation of 2.387.1? The job definition (config.xml) of one of the jobs that no longer shows its SCM configuration may be helpful for investigations as well.