• Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • credentials-plugin

      Credentials 2.6.2 upgrade breaks all credentials access operations:

      • Jenkins UI credentials controls
      • Job Subversion updates fail
      • Remote nodes don't start

      Downgrade to version 2.6.1 restores operation.

      java.lang.StackOverflowError stack traces are long and repeat twice.

      Attached ZIP file contains "QueueItemAuthenticatorConfiguration.xml" file cited in log and log file with exception stack traces, 30,264,301 bytes and 347,710 lines which I excerpt briefly:

       2021-12-06 16:54:00.620+0000 [id=11] WARNING hudson.model.Descriptor#load: Failed to load /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xml2021-12-06 16:54:00.620+0000 [id=11] WARNING hudson.model.Descriptor#load: Failed to load /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xmljava.lang.StackOverflowErrorCaused: java.io.IOException: Unable to read /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xml2021-12-06 16:54:00.648+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this componentjava.lang.IllegalStateException: Singleton is called recursively returning different results2021-12-06 16:54:00.660+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this componentjava.lang.IllegalStateException: Singleton is called recursively returning different results2021-12-06 16:54:00.694+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.s2m.AdminWhitelistRule, annotation=[none]]; skipping this componentjava.lang.NoClassDefFoundError: Could not initialize class jenkins.security.s2m.CallableWhitelistConfigCaused: com.google.inject.ProvisionException: Unable to provision, see the following errors:
      1) Error injecting constructor, java.lang.NoClassDefFoundError: Could not initialize class jenkins.security.s2m.CallableWhitelistConfig  at jenkins.security.s2m.AdminWhitelistRule.<init>(AdminWhitelistRule.java:59)
      1 error2021-12-06 16:54:00.712+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.s2m.MasterKillSwitchConfiguration, annotation=[none]]; skipping this componentcom.google.inject.ProvisionException: Unable to provision, see the following errors:
      1) null returned by binding at hudson.ExtensionFinder$GuiceFinder$SezpozModule.configure(ExtensionFinder.java:528) but jenkins.security.s2m.MasterKillSwitchConfiguration.rule is not @Nullable  while locating jenkins.security.s2m.AdminWhitelistRule    for field at jenkins.security.s2m.MasterKillSwitchConfiguration.rule(MasterKillSwitchConfiguration.java:19)
      1 error2021-12-06 16:54:00.897+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=hudson.tools.JDKInstaller$DescriptorImpl, annotation=[none]]; skipping this componentjava.lang.StackOverflowError

          [JENKINS-67308] Credentials 2.6.2 regression

          markewaite Yes, I visited "Configure Credential Providers" page and produced: comment-416861 which led to version 2.6.2 working.

          Conrad T. Pino added a comment - markewaite  Yes, I visited "Configure Credential Providers" page and produced:  comment-416861 which led to version 2.6.2 working.

          Conrad T. Pino added a comment - - edited

          I built and installed "Authorize Project" version 1.4.0 from source; it started without "StackOverflow" events logged.

          I uninstalled, restarted then reinstalled from "Manage Plugins" pages and it started without "StackOverflow" events.

          I am back to where I was before "credentiials-plugin" version 2.6.2 issues with version 2.6.2 working.

          timja said, "Sounds like there’s a migration bug somewhere" which IMO has merit.

          Conrad T. Pino added a comment - - edited I built and installed "Authorize Project" version 1.4.0 from source; it started without "StackOverflow" events logged. I uninstalled, restarted then reinstalled from "Manage Plugins" pages and it started without "StackOverflow" events. I am back to where I was before "credentiials-plugin" version 2.6.2 issues with version 2.6.2 working. timja  said, "Sounds like there’s a migration bug somewhere" which IMO has merit.

          My remaining concerns is an overall secure Jenkins installation using "fine grained permissions" defined with "Role-based Authorization Strategy" which satisfies https://www.jenkins.io/doc/book/security/build-authorization/ admonitions and those thereto https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/jenkins/security/QueueItemAuthenticatorMonitor/message.properties

           

          Conrad T. Pino added a comment - My remaining concerns is an overall secure Jenkins installation using "fine grained permissions" defined with "Role-based Authorization Strategy" which satisfies https://www.jenkins.io/doc/book/security/build-authorization/  admonitions and those thereto  https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/jenkins/security/QueueItemAuthenticatorMonitor/message.properties  

          Jesse Glick added a comment -

          Please use JENKINS-61990 for any discussion of build authentication. Off topic here.

          Jesse Glick added a comment - Please use JENKINS-61990 for any discussion of build authentication. Off topic here.

          Jesse Glick added a comment -

          If I understand then, this is just a duplicate of JENKINS-67170.

          Jesse Glick added a comment - If I understand then, this is just a duplicate of JENKINS-67170 .

          Conrad T. Pino added a comment - - edited

          IMO "resolving" this issue depends upon definitions:

          1. No, Credentials 2.6.2 exposed two (2) yet unfixed bugs.
            1. "Authorize Project" related StackOverflowError
            2. credentials-configuration.xml related credentials access fails
          2. Yes, the symptoms our system experienced are now mitigated

          Running "git bisect" twice maps each bug to separate commits.  However neither proves "Credentials" has either code defect.  Both bug mitigations share common elements:

          1. User Interface operations rewrite configuration:
            1. "Configure Credential Providers" "Only selected" JENKINS-67170
            2. "Manage Plugins" to uninstall then reinstall "Authorize Project"
          2. Zero coding changes made in both cases

          coupled with our July 2016  Jenkins launch IMO supporting timja proposition: Sounds like there’s a migration bug somewhere

          Since related configuration files knowledge is uncertain and prior state wasn't captured further tests here seem infeasible. However if incorrect please follow up as I am grateful for the support shown and wish to support your work in kind.

          Conrad T. Pino added a comment - - edited IMO "resolving" this issue depends upon definitions: No, Credentials 2.6.2 exposed two (2) yet unfixed bugs. "Authorize Project" related StackOverflowError credentials-configuration.xml related credentials access fails Yes, the symptoms our system experienced are now mitigated Running "git bisect" twice maps each bug to separate commits.  However neither proves "Credentials" has either code defect.  Both bug mitigations share common elements: User Interface operations rewrite configuration: "Configure Credential Providers" "Only selected"  JENKINS-67170 "Manage Plugins" to uninstall then reinstall "Authorize Project" Zero coding changes made in both cases coupled with our July 2016  Jenkins launch IMO supporting timja proposition: Sounds like there’s a migration bug somewhere Since related configuration files knowledge is uncertain and prior state wasn't captured further tests here seem infeasible. However if incorrect please follow up as I am grateful for the support shown and wish to support your work in kind.

          Jesse Glick added a comment -

          JENKINS-61990 is almost surely unrelated; it longs predates this credentials plugin release. The lack of credentials is an acknowledged regression in the credentials plugin which has been analyzed in JENKINS-67170.

          Jesse Glick added a comment - JENKINS-61990 is almost surely unrelated; it longs predates this credentials plugin release. The lack of credentials is an acknowledged regression in the credentials plugin which has been analyzed in JENKINS-67170 .

          jglick there is a relationship but how causal remains unclear. The confirmed evidence:

          A recent LTS update of many plugins included Credentials 2.6.1 and all was well afterwards:

          1. Credentials worked as expected
          2. StackOverflowError events absent

          A day or two later update of Credentials 2.6.2 alone and both issues presented themselves.

          Doing "git bisect start credentials-2.6.2 credentials-2.6.1" twice shows each issue appears at different Credentials commits.

          Both issues were mitigated with UI actions leading to configuration files rewritten to latest versions and Credentials 2.6.2 well behaved.

          I suspect both issues repeatability are plugin update sequence sensitive which may vary wildly between installations.

          I conclude:

          1. at least two configuration migration bugs are real
          2. and Credentials 2.6.2 can trigger their appearance

          but nothing done so far clearly identifies which components have code defects.

           

          Conrad T. Pino added a comment - jglick  there is a relationship but how causal remains unclear. The confirmed evidence: A recent LTS update of many plugins included Credentials 2.6.1 and all was well afterwards: Credentials worked as expected StackOverflowError events absent A day or two later update of Credentials 2.6.2 alone and both issues presented themselves. Doing " git bisect start credentials-2.6.2 credentials-2.6.1 " twice shows each issue appears at different Credentials commits. Both issues were mitigated with UI actions leading to configuration files rewritten to latest versions and Credentials 2.6.2 well behaved. I suspect both issues repeatability are plugin update sequence sensitive which may vary wildly between installations. I conclude: at least two configuration migration bugs are real and Credentials 2.6.2 can trigger their appearance but nothing done so far clearly identifies which components have code defects.  

          Jesse Glick added a comment -

          I have not studied JENKINS-61990 much but most likely it is a bug in the authorize-project plugin, which might be triggered under various circumstances by other code which is not at fault. As I said, that was occurring for a year prior to the appearance of credentials 2.6.2, so if there is any causal relationship (and not just random occurrence masquerading as a bisection success) then it is probably uninteresting.

          JENKINS-67170 is clearly a bug in credentials with a (now) clearly understood cause and a proposed fix.

          Jesse Glick added a comment - I have not studied JENKINS-61990 much but most likely it is a bug in the authorize-project plugin, which might be triggered under various circumstances by other code which is not at fault. As I said, that was occurring for a year prior to the appearance of credentials 2.6.2, so if there is any causal relationship (and not just random occurrence masquerading as a bisection success) then it is probably uninteresting. JENKINS-67170 is clearly a bug in credentials with a (now) clearly understood cause and a proposed fix.

          I created "Authorize Project" class name list from source file names and used it as fgrep pattern file to prove "Authorize Project" classes are in stack traces however without prior configuration files replication for debugging isn't possible here.

          I include class name list here for help with a future occurrence.

          org.jenkinsci.plugins.authorizeproject.AuthorizeProjectProperty
          org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy
          org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategyDescriptor
          org.jenkinsci.plugins.authorizeproject.AuthorizeProjectUtil
          org.jenkinsci.plugins.authorizeproject.ConfigurationPermissionEnforcer
          org.jenkinsci.plugins.authorizeproject.Constants
          org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator
          org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticatorTest
          org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator
          org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticatorTest
          org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategy
          org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategyTest
          org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategy
          org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategyTest
          org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategy
          org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategyTest
          org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategy
          org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategyTest
          org.jenkinsci.plugins.authorizeproject.testutil.AuthorizationCheckBuilder
          org.jenkinsci.plugins.authorizeproject.testutil.AuthorizeProjectJenkinsRule
          org.jenkinsci.plugins.authorizeproject.testutil.SecurityRealmWithUserFilter

          Conrad T. Pino added a comment - I created "Authorize Project" class name list from source file names and used it as fgrep pattern file to prove "Authorize Project" classes are in stack traces however without prior configuration files replication for debugging isn't possible here. I include class name list here for help with a future occurrence. org.jenkinsci.plugins.authorizeproject.AuthorizeProjectProperty org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategy org.jenkinsci.plugins.authorizeproject.AuthorizeProjectStrategyDescriptor org.jenkinsci.plugins.authorizeproject.AuthorizeProjectUtil org.jenkinsci.plugins.authorizeproject.ConfigurationPermissionEnforcer org.jenkinsci.plugins.authorizeproject.Constants org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticatorTest org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticator org.jenkinsci.plugins.authorizeproject.ProjectQueueItemAuthenticatorTest org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategy org.jenkinsci.plugins.authorizeproject.strategy.AnonymousAuthorizationStrategyTest org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategy org.jenkinsci.plugins.authorizeproject.strategy.SpecificUsersAuthorizationStrategyTest org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategy org.jenkinsci.plugins.authorizeproject.strategy.SystemAuthorizationStrategyTest org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategy org.jenkinsci.plugins.authorizeproject.strategy.TriggeringUsersAuthorizationStrategyTest org.jenkinsci.plugins.authorizeproject.testutil.AuthorizationCheckBuilder org.jenkinsci.plugins.authorizeproject.testutil.AuthorizeProjectJenkinsRule org.jenkinsci.plugins.authorizeproject.testutil.SecurityRealmWithUserFilter

            Unassigned Unassigned
            conrad_t_pino Conrad T. Pino
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: