Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67308

Credentials 2.6.2 regression

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • credentials-plugin

      Credentials 2.6.2 upgrade breaks all credentials access operations:

      • Jenkins UI credentials controls
      • Job Subversion updates fail
      • Remote nodes don't start

      Downgrade to version 2.6.1 restores operation.

      java.lang.StackOverflowError stack traces are long and repeat twice.

      Attached ZIP file contains "QueueItemAuthenticatorConfiguration.xml" file cited in log and log file with exception stack traces, 30,264,301 bytes and 347,710 lines which I excerpt briefly:

       2021-12-06 16:54:00.620+0000 [id=11] WARNING hudson.model.Descriptor#load: Failed to load /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xml2021-12-06 16:54:00.620+0000 [id=11] WARNING hudson.model.Descriptor#load: Failed to load /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xmljava.lang.StackOverflowErrorCaused: java.io.IOException: Unable to read /var/lib/jenkins/jenkins.security.QueueItemAuthenticatorConfiguration.xml2021-12-06 16:54:00.648+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this componentjava.lang.IllegalStateException: Singleton is called recursively returning different results2021-12-06 16:54:00.660+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.QueueItemAuthenticatorConfiguration, annotation=[none]]; skipping this componentjava.lang.IllegalStateException: Singleton is called recursively returning different results2021-12-06 16:54:00.694+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.s2m.AdminWhitelistRule, annotation=[none]]; skipping this componentjava.lang.NoClassDefFoundError: Could not initialize class jenkins.security.s2m.CallableWhitelistConfigCaused: com.google.inject.ProvisionException: Unable to provision, see the following errors:
      1) Error injecting constructor, java.lang.NoClassDefFoundError: Could not initialize class jenkins.security.s2m.CallableWhitelistConfig  at jenkins.security.s2m.AdminWhitelistRule.<init>(AdminWhitelistRule.java:59)
      1 error2021-12-06 16:54:00.712+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=jenkins.security.s2m.MasterKillSwitchConfiguration, annotation=[none]]; skipping this componentcom.google.inject.ProvisionException: Unable to provision, see the following errors:
      1) null returned by binding at hudson.ExtensionFinder$GuiceFinder$SezpozModule.configure(ExtensionFinder.java:528) but jenkins.security.s2m.MasterKillSwitchConfiguration.rule is not @Nullable  while locating jenkins.security.s2m.AdminWhitelistRule    for field at jenkins.security.s2m.MasterKillSwitchConfiguration.rule(MasterKillSwitchConfiguration.java:19)
      1 error2021-12-06 16:54:00.897+0000 [id=11] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=hudson.tools.JDKInstaller$DescriptorImpl, annotation=[none]]; skipping this componentjava.lang.StackOverflowError

        1. Jenkins-Configure-Credentials-Provider.png
          Jenkins-Configure-Credentials-Provider.png
          24 kB
        2. jenkins-fail-20211206.zip
          522 kB

          [JENKINS-67308] Credentials 2.6.2 regression

          Conrad T. Pino created issue -

          Mark Waite added a comment -

          Interesting that the QueueItemAuthenticatorConfiguration was last stored with authorize project 1.3.0 (released 5 years ago) yet your installed version of authorize project is 1.4.0 (release less than 1 year ago). You might try saving the authorize project configuration to see if saving from a newer version resolves the issue.

          Mark Waite added a comment - Interesting that the QueueItemAuthenticatorConfiguration was last stored with authorize project 1.3.0 (released 5 years ago) yet your installed version of authorize project is 1.4.0 (release less than 1 year ago). You might try saving the authorize project configuration to see if saving from a newer version resolves the issue.

          Mark Waite added a comment -

          You could bisect the changes between 2.6.1 and 2.6.2 to identify which change caused the new behavior. Bisecting would allow you to decide which commit needs to be investigated more deeply.

          Mark Waite added a comment - You could bisect the changes between 2.6.1 and 2.6.2 to identify which change caused the new behavior. Bisecting would allow you to decide which commit needs to be investigated more deeply.

          Conrad T. Pino added a comment -

          Excellent suggestion:

          You might try saving the authorize project configuration to see if saving from a newer version resolves the issue.

          However it presumes the steps necessary to accomplish such a saving action are already known which is not so now.

          For simplicity's sake I defer addressing bisecting suggestion until first suggestion is completed.

          Conrad T. Pino added a comment - Excellent suggestion: You might try saving the authorize project configuration to see if saving from a newer version resolves the issue. However it presumes the steps necessary to accomplish such a saving action are already known which is not so now. For simplicity's sake I defer addressing bisecting suggestion until first suggestion is completed.

          Conrad T. Pino added a comment -

          Bread crumb trail followed: Manage Jenkins > Security > Configure Global Security > Save

          Changed "jenkins.security.QueueItemAuthenticatorConfiguration.xml" line 4 to:

          <org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator plugin="authorize-project@1.4.0">

          However Credentials 2.6.2 failed as before with normal operation restored with 2.6.1 downgrade.

          Conrad T. Pino added a comment - Bread crumb trail followed: Manage Jenkins > Security > Configure Global Security > Save Changed " jenkins.security.QueueItemAuthenticatorConfiguration.xml " line 4 to: <org.jenkinsci.plugins.authorizeproject.GlobalQueueItemAuthenticator plugin="authorize-project@1.4.0" > However Credentials 2.6.2 failed as before with normal operation restored with 2.6.1 downgrade.

          Conrad T. Pino added a comment - - edited

          GitHub URL: https://github.com/jenkinsci/credentials-plugin/compare/credentials-2.6.1...credentials-2.6.2

          47 commits and 130 changed files from version 2.6.1 to  2.6.2 appears to be a lot of bisecting.

          Conrad T. Pino added a comment - - edited GitHub URL: https://github.com/jenkinsci/credentials-plugin/compare/credentials-2.6.1...credentials-2.6.2 47 commits and 130 changed files from version 2.6.1 to  2.6.2 appears to be a lot of bisecting.

          Conrad T. Pino added a comment -

          "credentials-plugin-2.6.2"  has 133 Java source files in these packages:

          egrep --no-filename "^package" $( find credentials-plugin-2.6.2/src -type f -iname "*.java" ) | sort -u

          package com.cloudbees.plugins.credentials;
          package com.cloudbees.plugins.credentials.builds;
          package com.cloudbees.plugins.credentials.casc;
          package com.cloudbees.plugins.credentials.cli;
          package com.cloudbees.plugins.credentials.common;
          package com.cloudbees.plugins.credentials.domains;
          package com.cloudbees.plugins.credentials.fingerpints;
          package com.cloudbees.plugins.credentials.fingerprints;
          package com.cloudbees.plugins.credentials.impl;
          package com.cloudbees.plugins.credentials.matchers;
          package jenkins.security;

          What's quite odd is no "credentials-plugin-2.6.2" Java class appears in "jenkins.log" stack traces.

          The "jenkins.security" package appears in stack traces:

          *at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
          {{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
          at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
          at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
          at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
          at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
          *at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
          {{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
          at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
          at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
          at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
          at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
          *at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)}}*
          {{at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60)
          at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95)
          at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66)
          at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80)
          at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
          at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39)

          in obvious recursion but without "jenkins.security.ConfidentialStoreRule" class from "credentials-plugin-2.6.2" plugin.

          Conrad T. Pino added a comment - " credentials-plugin-2.6.2 "  has 133 Java source files in these packages: egrep --no-filename "^package" $( find credentials-plugin-2.6.2/src -type f -iname "*.java" ) | sort -u package com.cloudbees.plugins.credentials; package com.cloudbees.plugins.credentials.builds; package com.cloudbees.plugins.credentials.casc; package com.cloudbees.plugins.credentials.cli; package com.cloudbees.plugins.credentials.common; package com.cloudbees.plugins.credentials.domains; package com.cloudbees.plugins.credentials.fingerpints; package com.cloudbees.plugins.credentials.fingerprints; package com.cloudbees.plugins.credentials.impl; package com.cloudbees.plugins.credentials.matchers; package jenkins.security; What's quite odd is no " credentials-plugin-2.6.2 " Java class appears in " jenkins.log " stack traces. The " jenkins.security " package appears in stack traces: * at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39) }}* {{ at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60) at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95) at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) * at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39) }}* {{ at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60) at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95) at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) * at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39) }}* {{ at jenkins.security.QueueItemAuthenticatorConfiguration.get(QueueItemAuthenticatorConfiguration.java:60) at jenkins.security.QueueItemAuthenticatorMonitor.isQueueItemAuthenticatorConfigured(QueueItemAuthenticatorMonitor.java:95) at jenkins.security.QueueItemAuthenticatorMonitor.isActivated(QueueItemAuthenticatorMonitor.java:66) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:80) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:39) in obvious recursion but without " jenkins.security.ConfidentialStoreRule " class from " credentials-plugin-2.6.2 " plugin.

          Jesse Glick added a comment -

          Best is a way to find a way to reproduce the error in a fresh installation—a minimal test case.

          If that does not seem feasible, but you can consistently reproduce the error in your installation—which sounds like it might be true based on

          Downgrade to version 2.6.1 restores operation.

          then, given the large number of changes here with no immediately apparent relationship to the stack trace, you will need to bisect. Having installed Git, Java 8, and Maven, something like

          git clone https://github.com/jenkinsci/credentials-plugin
cd credentials-plugin
git checkout credentials-2.6.2^
mvn -Pquick-build package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, verify that problem occurs
git bisect start
git bisect bad
git checkout fd56f466de08642f4b4db567668a6ab4c2f814dc # “prepare for next development iteration”
mvn -Pquick-build clean package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, verify that problem does not occur
git bisect good
# repeat from here:
mvn -Pquick-build clean package
cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi
# restart Jenkins, git bisect good/bad accordingly

          Jesse Glick added a comment - Best is a way to find a way to reproduce the error in a fresh installation—a minimal test case. If that does not seem feasible, but you can consistently reproduce the error in your installation—which sounds like it might be true based on Downgrade to version 2.6.1 restores operation. then, given the large number of changes here with no immediately apparent relationship to the stack trace, you will need to bisect. Having installed Git, Java 8, and Maven, something like git clone https://github.com/jenkinsci/credentials-plugin cd credentials-plugin git checkout credentials-2.6.2^ mvn -Pquick-build package cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi # restart Jenkins, verify that problem occurs git bisect start git bisect bad git checkout fd56f466de08642f4b4db567668a6ab4c2f814dc # “prepare for next development iteration” mvn -Pquick-build clean package cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi # restart Jenkins, verify that problem does not occur git bisect good # repeat from here: mvn -Pquick-build clean package cp target/credentials.hpi $JENKINS_HOME/plugins/credentials.jpi # restart Jenkins, git bisect good/bad accordingly

          Conrad T. Pino added a comment - - edited

          Since stack traces include no plugin class that should exclude plugin Java code bugs which leaves some other plugin file type processed by "jenkins.security.*" class tree where recursion is seen. How can we exploit this proposition to our advantage?

          A fresh installation into a different virtual machine with Ubuntu instead of Debian is possible; would that still help us?

          jglick thank you, "git bisect" example incredibly well targeted. How much time would you set aside to perform this task?

           

          Conrad T. Pino added a comment - - edited Since stack traces include no plugin class that should exclude plugin Java code bugs which leaves some other plugin file type processed by " jenkins.security.* " class tree where recursion is seen. How can we exploit this proposition to our advantage? A fresh installation into a different virtual machine with Ubuntu instead of Debian is possible; would that still help us? jglick thank you, "git bisect" example incredibly well targeted. How much time would you set aside to perform this task?  

          Jesse Glick added a comment -

          How much time would you set aside to perform this task?

          An hour perhaps?

          Jesse Glick added a comment - How much time would you set aside to perform this task? An hour perhaps?

            Unassigned Unassigned
            conrad_t_pino Conrad T. Pino
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: