Interesting that the QueueItemAuthenticatorConfiguration was last stored with authorize project 1.3.0 (released 5 years ago) yet your installed version of authorize project is 1.4.0 (release less than 1 year ago). You might try saving the authorize project configuration to see if saving from a newer version resolves the issue.

You could bisect the changes between 2.6.1 and 2.6.2 to identify which change caused the new behavior. Bisecting would allow you to decide which commit needs to be investigated more deeply.

Excellent suggestion:

You might try saving the authorize project configuration to see if saving from a newer version resolves the issue.

However it presumes the steps necessary to accomplish such a saving action are already known which is not so now.

For simplicity's sake I defer addressing bisecting suggestion until first suggestion is completed.

Bread crumb trail followed: Manage Jenkins > Security > Configure Global Security > Save

Changed "jenkins.security.QueueItemAuthenticatorConfiguration.xml" line 4 to:

<org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator plugin="authorize-project@1.4.0">

However Credentials 2.6.2 failed as before with normal operation restored with 2.6.1 downgrade.

GitHub URL: https://github.com/jenkinsci/credentials-plugin/compare/credentials-2.6.1...credentials-2.6.2

47 commits and 130 changed files from version 2.6.1 to  2.6.2 appears to be a lot of bisecting.

"credentials-plugin-2.6.2"  has 133 Java source files in these packages:

egrep --no-filename "^package" $( find credentials-plugin-2.6.2/src -type f -iname "*.java" ) | sort -u

package com.cloudbees.plugins.credentials;
package com.cloudbees.plugins.credentials.builds;
package com.cloudbees.plugins.credentials.casc;
package com.cloudbees.plugins.credentials.cli;
package com.cloudbees.plugins.credentials.common;
package com.cloudbees.plugins.credentials.domains;
package com.cloudbees.plugins.credentials.fingerpints;
package com.cloudbees.plugins.credentials.fingerprints;
package com.cloudbees.plugins.credentials.impl;
package com.cloudbees.plugins.credentials.matchers;
package jenkins.security;

What's quite odd is no "credentials-plugin-2.6.2" Java class appears in "jenkins.log" stack traces.

The "jenkins.security" package appears in stack traces:

*at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
{{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
*at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
{{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
*at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
{{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)

in obvious recursion but without "jenkins.security.ConfidentialStoreRule" class from "credentials-plugin-2.6.2" plugin.

Best is a way to find a way to reproduce the error in a fresh installation—a minimal test case.

If that does not seem feasible, but you can consistently reproduce the error in your installation—which sounds like it might be true based on

Downgrade to version 2.6.1 restores operation.

then, given the large number of changes here with no immediately apparent relationship to the stack trace, you will need to bisect. Having installed Git, Java 8, and Maven, something like

git clone https://github.com/jenkinsci/credentials-plugin
cd credentials-plugin
git checkout credentials-2.6.2^
mvn -Pquick-build package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, verify that problem occurs
git bisect start
git bisect bad
git checkout fd56f466de08642f4b4db567668a6ab4c2f814dc # “prepare for next development iteration”
mvn -Pquick-build clean package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, verify that problem does not occur
git bisect good
# repeat from here:
mvn -Pquick-build clean package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, git bisect good/bad accordingly

Since stack traces include no plugin class that should exclude plugin Java code bugs which leaves some other plugin file type processed by "jenkins.security.*" class tree where recursion is seen. How can we exploit this proposition to our advantage?

A fresh installation into a different virtual machine with Ubuntu instead of Debian is possible; would that still help us?

jglick thank you, "git bisect" example incredibly well targeted. How much time would you set aside to perform this task?

How much time would you set aside to perform this task?

An hour perhaps?

jglick An hour ... I can do that on current system after business hours. Thank you!

"java.lang.StackOverflowError" occurs with working version 2.6.1 making it a red herring for the purposes of this plugin.

Git bisect steps:

git bisect start credentials-2.6.2 credentials-2.6.1
git bisect good
git bisect good
git bisect bad
git bisect good
git bisect bad
git bisect good

Git bisect results:

23d24bd53f69c96d68b1fe5135d02eb542b93606 is the first bad commit
commit 23d24bd53f69c96d68b1fe5135d02eb542b93606
Author: Tim Jacomb <timjacomb1+github@gmail.com>
Date: Sun Oct 3 13:32:33 2021 +0100Use descriptor ID:040000 040000 f8b077237912d4d0e6874e9732179ebfd8c1c83c f3ed379200c5f3e894bb325d429faa226914cd18 M src

How should we follow up with "java.lang.StackOverflowError" events?

Ah the StackOverflowError is JENKINS-61990. Uninstall that plugin I guess; it seems to be broken without a fix.

So if that is unrelated to your issue, what exactly is your issue? It is unclear from the description what is “broken”; I was under the impression you meant that every credentials operation caused a StackOverflowError.

timja FYI

I'll take a look if there's clear steps to reproduce, (after authorize-project is removed given that plugin looks to be in a bad state)

After 2.6.1 restore there are no "java.lang.StackOverflowError" events which puts that back into scope but it's not the bug causing credentials access to fail. IMO we have multiple issues. I will bisect again to determine where "java.lang.StackOverflowError" appears.

git bisect log
# bad: [0d55107ff7d3cae06c4472ade55f59d76017dd66] [maven-release-plugin] prepare release credentials-2.6.2
# good: [6a78e9fa99acb8b0ec9f54b7afe0817ef39197c3] [maven-release-plugin] prepare release credentials-2.6.1
git bisect start 'credentials-2.6.2' 'credentials-2.6.1'
# good: [45232d553ae7b1fdc4bf9ee55145c760312470a9] Merge pull request #238 from aneveux/bee-8992-java11-support
git bisect good 45232d553ae7b1fdc4bf9ee55145c760312470a9
# good: [3a61456d43219ecf21f84cbccc40e1ebd9fffc08] Merge branch 'master' into offa_dev
git bisect good 3a61456d43219ecf21f84cbccc40e1ebd9fffc08
# bad: [db2bc6ca148ace9223581d2fd7363bcab67f55b3] Test adjustments
git bisect bad db2bc6ca148ace9223581d2fd7363bcab67f55b3
# good: [19d2dbca86911bf72369b3894bdb2b18976ec05c] Update pom.xml
git bisect good 19d2dbca86911bf72369b3894bdb2b18976ec05c
# bad: [23d24bd53f69c96d68b1fe5135d02eb542b93606] Use descriptor ID
git bisect bad 23d24bd53f69c96d68b1fe5135d02eb542b93606
# good: [b9cb5f9d1b6365fc3cc5641d7a7f6267e84c358a] Merge branch 'master' into fix-casc-for-cred-provider-filter
git bisect good b9cb5f9d1b6365fc3cc5641d7a7f6267e84c358a
# first bad commit: [23d24bd53f69c96d68b1fe5135d02eb542b93606] Use descriptor ID

This Git bisect isolates "java.lang.StackOverflowError" events:

user@host:~/jenkins/credentials-plugin$ git bisect log
# bad: [0d55107ff7d3cae06c4472ade55f59d76017dd66] [maven-release-plugin] prepare release credentials-2.6.2
# good: [6a78e9fa99acb8b0ec9f54b7afe0817ef39197c3] [maven-release-plugin] prepare release credentials-2.6.1
git bisect start 'credentials-2.6.2' 'credentials-2.6.1'
# good: [45232d553ae7b1fdc4bf9ee55145c760312470a9] Merge pull request #238 from aneveux/bee-8992-java11-support
git bisect good 45232d553ae7b1fdc4bf9ee55145c760312470a9
# good: [3a61456d43219ecf21f84cbccc40e1ebd9fffc08] Merge branch 'master' into offa_dev
git bisect good 3a61456d43219ecf21f84cbccc40e1ebd9fffc08
# bad: [db2bc6ca148ace9223581d2fd7363bcab67f55b3] Test adjustments
git bisect bad db2bc6ca148ace9223581d2fd7363bcab67f55b3
# bad: [19d2dbca86911bf72369b3894bdb2b18976ec05c] Update pom.xml
git bisect bad 19d2dbca86911bf72369b3894bdb2b18976ec05c
# bad: [a674595c109ff866249ea2293e136417fbe9e79e] Merge branch 'master' into fix-casc-for-cred-provider-filter
git bisect bad a674595c109ff866249ea2293e136417fbe9e79e
# good: [d421071de64bc500276c991deefd2ed34100f00f] Add config as code support for credentials configuration
git bisect good d421071de64bc500276c991deefd2ed34100f00f
# first bad commit: [a674595c109ff866249ea2293e136417fbe9e79e] Merge branch 'master' into fix-casc-for-cred-provider-filter

I now believe StackOverflow and credentials unavailable are distinct bugs committed separately.

jglick timja

Now that we have evidence implicating StackOverflowError to a commit for this plugin, how should we proceed now?

Can you uninstall / disable the authorize project plugin?

It's been unmaintained for quite awhile now (over 2 years):
https://groups.google.com/g/jenkinsci-dev/c/c1z4C3mQ2-k/m/tJhVSLa3BAAJ

There was a report of missing credentials in JENKINS-67170 that was resolved by updating the global configuration of the credentials. It is not clear how the change happens that is described there, but it may be worth a quick check based on the description in that bug.

timja
Nobody remembers why "Authorize Project" was installed.
Our workflow depends on "Role-based Authorization Strategy".
I find plugin Dependencies documentation fairly opaque.
How can I determine what I lose by trying?

You likely have the same issue that mark posted above try that. Sounds like there’s a migration bug somewhere

markewaite timja

Successfully rewrote file, note "credentials@2.6.1" text; output follows:

cpino@ibis:~/jenkins/credentials-plugin$ cat -n /var/lib/jenkins/credentials-configuration.xml; echo
1 <?xml version='1.1' encoding='UTF-8'?>
2 <com.cloudbees.plugins.credentials.CredentialsProviderManager plugin="credentials@2.6.1">
3 <providerFilter class="com.cloudbees.plugins.credentials.CredentialsProviderFilter$Includes">
4 <classNames class="linked-hash-set">
5 <string>com.cloudbees.plugins.credentials.SystemCredentialsProvider$ProviderImpl</string>
6 <string>com.cloudbees.plugins.credentials.UserCredentialsProvider</string>
7 </classNames>
8 </providerFilter>
9 <typeFilter class="com.cloudbees.plugins.credentials.CredentialsTypeFilter$Includes">
10 <classNames class="linked-hash-set">
11 <string>com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl$DescriptorImpl</string>
12 <string>com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DescriptorImpl</string>
13 <string>com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl$DescriptorImpl</string>
14 </classNames>
15 </typeFilter>
16 <restrictions/>
17 </com.cloudbees.plugins.credentials.CredentialsProviderManager>

Still broken after 2.6.2 upgrade.

Did you update to latest version and then view that page in the UI? Can you take a screenshot and then saved the config?

timja No, not at first as shown in "credentials-configuration.xml" above.

After version 2.6.2 upgrade UI changes to "All Available" to produce output:

1 <?xml version='1.1' encoding='UTF-8'?>
2 <com.cloudbees.plugins.credentials.CredentialsProviderManager plugin="credentials@2.6.2">
3 <providerFilter class="com.cloudbees.plugins.credentials.CredentialsProviderFilter$None"/>
4 <typeFilter class="com.cloudbees.plugins.credentials.CredentialsTypeFilter$None"/>
5 <restrictions/>
6 </com.cloudbees.plugins.credentials.CredentialsProviderManager>

After restart UI updated back to "Only Selected" with all checked to product output:

1 <?xml version='1.1' encoding='UTF-8'?>
2 <com.cloudbees.plugins.credentials.CredentialsProviderManager plugin="credentials@2.6.2">
3 <providerFilter class="com.cloudbees.plugins.credentials.CredentialsProviderFilter$Includes">
4 <classNames class="linked-hash-set">
5 <string>com.cloudbees.plugins.credentials.SystemCredentialsProvider$ProviderImpl</string>
6 <string>com.cloudbees.plugins.credentials.UserCredentialsProvider</string>
7 </classNames>
8 </providerFilter>
9 <typeFilter class="com.cloudbees.plugins.credentials.CredentialsTypeFilter$Includes">
10 <classNames class="linked-hash-set">
11 <string>com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl</string>
12 <string>com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey</string>
13 <string>com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl</string>
14 </classNames>
15 </typeFilter>
16 <restrictions/>
17 </com.cloudbees.plugins.credentials.CredentialsProviderManager>

After restart with 2.6.2 credentials work and "java.lang.StackOverflowError" logged. The "Authorize Project" is still installed.

Removing "Authorize Project" plugin stops "java.lang.StackOverflowError" events but I get red security warning to install it; the "Learn More" link is https://www.jenkins.io/doc/book/security/build-authorization/ which ends with, "The most notable plugin in this space is Authorize Project Plugin, which allows flexible configuration of global and per-project build authorization." which are likely the reason originally installed.

What's the modern way to secure fine grained permissions setup using "Role-based Authorization Strategy"?

jglick, my apologies please; I overlooked your questions:

The recurring UI visible errors are:

1. Remote SSH nodes don't start.
2. Projects using Subversion fail.
3. Project Configure doesn't show Subversion credentials.
4. "Manage Credentials" left alone to prevent corruption.

StackOverflowError events logged copiously at startup and then no more.

conrad_t_pino did you check the user interface that was mentioned in JENKINS 67170 comment ? It suggests that something may cause a change of the credentials configuration during the upgrade from 2.6.1 to 2.6.2. I can't duplicate the problem, but it sounds similar to the problem you're reporting.

I built and installed "Authorize Project" version 1.4.0 from source; it started without "StackOverflow" events logged.

I uninstalled, restarted then reinstalled from "Manage Plugins" pages and it started without "StackOverflow" events.

I am back to where I was before "credentiials-plugin" version 2.6.2 issues with version 2.6.2 working.

timja said, "Sounds like there’s a migration bug somewhere" which IMO has merit.

My remaining concerns is an overall secure Jenkins installation using "fine grained permissions" defined with "Role-based Authorization Strategy" which satisfies https://www.jenkins.io/doc/book/security/build-authorization/ admonitions and those thereto https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/jenkins/security/QueueItemAuthenticatorMonitor/message.properties

Please use JENKINS-61990 for any discussion of build authentication. Off topic here.

If I understand then, this is just a duplicate of JENKINS-67170.

IMO "resolving" this issue depends upon definitions:

1. No, Credentials 2.6.2 exposed two (2) yet unfixed bugs.
   1. "Authorize Project" related StackOverflowError
   2. credentials-configuration.xml related credentials access fails
2. Yes, the symptoms our system experienced are now mitigated

Running "git bisect" twice maps each bug to separate commits.  However neither proves "Credentials" has either code defect.  Both bug mitigations share common elements:

1. User Interface operations rewrite configuration:
   1. "Configure Credential Providers" "Only selected" JENKINS-67170
   2. "Manage Plugins" to uninstall then reinstall "Authorize Project"
2. Zero coding changes made in both cases

coupled with our July 2016  Jenkins launch IMO supporting timja proposition: Sounds like there’s a migration bug somewhere

Since related configuration files knowledge is uncertain and prior state wasn't captured further tests here seem infeasible. However if incorrect please follow up as I am grateful for the support shown and wish to support your work in kind.

JENKINS-61990 is almost surely unrelated; it longs predates this credentials plugin release. The lack of credentials is an acknowledged regression in the credentials plugin which has been analyzed in JENKINS-67170.

jglick there is a relationship but how causal remains unclear. The confirmed evidence:

A recent LTS update of many plugins included Credentials 2.6.1 and all was well afterwards:

1. Credentials worked as expected
2. StackOverflowError events absent

A day or two later update of Credentials 2.6.2 alone and both issues presented themselves.

Doing "git bisect start credentials-2.6.2 credentials-2.6.1" twice shows each issue appears at different Credentials commits.

Both issues were mitigated with UI actions leading to configuration files rewritten to latest versions and Credentials 2.6.2 well behaved.

I suspect both issues repeatability are plugin update sequence sensitive which may vary wildly between installations.

I conclude:

1. at least two configuration migration bugs are real
2. and Credentials 2.6.2 can trigger their appearance

but nothing done so far clearly identifies which components have code defects.

I have not studied JENKINS-61990 much but most likely it is a bug in the authorize-project plugin, which might be triggered under various circumstances by other code which is not at fault. As I said, that was occurring for a year prior to the appearance of credentials 2.6.2, so if there is any causal relationship (and not just random occurrence masquerading as a bisection success) then it is probably uninteresting.

JENKINS-67170 is clearly a bug in credentials with a (now) clearly understood cause and a proposed fix.

I created "Authorize Project" class name list from source file names and used it as fgrep pattern file to prove "Authorize Project" classes are in stack traces however without prior configuration files replication for debugging isn't possible here.

I include class name list here for help with a future occurrence.

org.jenkinsci.plugins.authorizeproject.AuthorizeProjectProperty
org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy
org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategyDescriptor
org.jenkinsci.plugins.authorizeproject.AuthorizeProjectUtil
org.jenkinsci.plugins.authorizeproject.ConfigurationPermissionEnforcer
org.jenkinsci.plugins.authorizeproject.Constants
org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator
org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticatorTest
org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator
org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticatorTest
org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategy
org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategyTest
org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategy
org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategyTest
org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategy
org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategyTest
org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategy
org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategyTest
org.jenkinsci.plugins.authorizeproject.testutil.AuthorizationCheckBuilder
org.jenkinsci.plugins.authorizeproject.testutil.AuthorizeProjectJenkinsRule
org.jenkinsci.plugins.authorizeproject.testutil.SecurityRealmWithUserFilter