Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47909

Last master version 2.73.3 won't start when usernames contain $

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      Docker: jenkins/jenkins:2.73.3-alpine
      Kubernetes: GKE v1.8.1-gke.0

      Upgrade from: jenkins/jenkins:2.73.2-alpine
    • Similar Issues:

      Description

      Hi,

      after upgrade our deployment in the last version 2.73.3, i had to rollback in 2.73.2.

      The master won't start:

       

      INFO: Loaded all jobs
      Nov 09, 2017 8:40:54 AM jenkins.util.groovy.GroovyHookScript execute
      INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Started Download metadata
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Finished Download metadata. 37 ms
      Nov 09, 2017 8:40:54 AM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed AllUsers.scanAll
      java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM hudson.util.BootFailure publish
      SEVERE: Failed to initialize Jenkins
      hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.WebAppMain$3.run(WebAppMain.java:252)
      Caused by: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
       at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
       at jenkins.model.Jenkins.executeReactor(Jenkins.java:1124)
       at jenkins.model.Jenkins.<init>(Jenkins.java:929)
       at hudson.model.Hudson.<init>(Hudson.java:86)
       at hudson.model.Hudson.<init>(Hudson.java:82)
       at hudson.WebAppMain$3.run(WebAppMain.java:235)
      Caused by: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Stopping Jenkins
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Started termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Completed termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager
      INFO: Stopping plugin manager
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpPersistQueue
      INFO: Persisting build queue
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects
      INFO: Waiting for node disconnection completion
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Jenkins stopped
      --> setting agent port for jnlp
      Nov 09, 2017 8:41:04 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
      SEVERE: A thread (Thread-3/50) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
      java.lang.NullPointerException: Cannot invoke method setSlaveAgentPort() on null object
       at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
       at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy:10)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
       at groovy.lang.Closure.call(Closure.java:414)
       at groovy.lang.Closure.call(Closure.java:408)
       at groovy.lang.Closure.run(Closure.java:495)
       at java.lang.Thread.run(Thread.java:745)
      

      The rollback in 2.73.2 has solved the problem.

       

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it

            Show
            oleg_nenashev Oleg Nenashev added a comment - I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it
            Hide
            danielbeck Daniel Beck added a comment -

            This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921.

            Vincent Robert Please provide a list of directory names inside JENKINS_HOME/users.

            Show
            danielbeck Daniel Beck added a comment - This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921 . Vincent Robert Please provide a list of directory names inside JENKINS_HOME/users .
            Hide
            danielbeck Daniel Beck added a comment -

            My guess would be there's a user whose name is $whco or that contains that string.

            Show
            danielbeck Daniel Beck added a comment - My guess would be there's a user whose name is $whco or that contains that string.
            Hide
            vrobert78 Vincent Robert added a comment -

            You are right, there is a user with a $ :

            drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35'

            Can I do a "rm -rf" of the folder ?

             

             

            Show
            vrobert78 Vincent Robert added a comment - You are right, there is a user with a $ : drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35' Can I do a "rm -rf" of the folder ?    
            Hide
            jglick Jesse Glick added a comment -

            Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.

            Show
            jglick Jesse Glick added a comment - Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.
            Hide
            jglick Jesse Glick added a comment -

            Vincent Robert as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.

            Show
            jglick Jesse Glick added a comment - Vincent Robert as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.
            Hide
            vrobert78 Vincent Robert added a comment -

            Jesse Glick , the hotfix is OK. I have deleted the folder, everything is fine by now !

             

            Show
            vrobert78 Vincent Robert added a comment - Jesse Glick , the hotfix is OK. I have deleted the folder, everything is fine by now !  
            Hide
            danielbeck Daniel Beck added a comment - - edited

            Any settings for that user (API tokens, etc.) will be lost in such a case.

            I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's c$whcomiti35 that would be c$0024whcomiti35.

            Show
            danielbeck Daniel Beck added a comment - - edited Any settings for that user (API tokens, etc.) will be lost in such a case. I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's  c$whcomiti35 that would be c$0024whcomiti35 .
            Hide
            martoeng Martin Walter added a comment -

            We had the same problem (JENKINS-47921). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?

            Show
            martoeng Martin Walter added a comment - We had the same problem ( JENKINS-47921 ). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?
            Hide
            vrobert78 Vincent Robert added a comment -

            Martin Walter It's the same for us, we do not use the internal authentication system, we use the github auth.

            In the users directory, many users are from externals repos.

             

            Show
            vrobert78 Vincent Robert added a comment - Martin Walter  It's the same for us, we do not use the internal authentication system, we use the github auth. In the users directory, many users are from externals repos.  
            Hide
            martoeng Martin Walter added a comment -

            So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

            Show
            martoeng Martin Walter added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?
            Hide
            danielbeck Daniel Beck added a comment -

            So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

            The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration.

            Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation?

            Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.

            Show
            danielbeck Daniel Beck added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again? The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration. Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b
            Log:
            JENKINS-47909 Handle false hex escapes.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b Log: JENKINS-47909 Handle false hex escapes.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d
            Log:
            JENKINS-47909 Migration test.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d Log: JENKINS-47909 Migration test.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/User.java
            http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad
            Log:
            JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/User.java http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad Log: JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831
            Log:
            JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831 Log: JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/model/User.java
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f
            Log:
            Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

            JENKINS-47909 Handle false hex escapes

            Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468
            Hide
            martoeng Martin Walter added a comment -

            They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.

            Show
            martoeng Martin Walter added a comment - They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            A hotfix has been merged towards 2.90.
            I will keep it open, because there may be other action items needed. Jesse Glick please close it if you feel it's fully done.

            Show
            oleg_nenashev Oleg Nenashev added a comment - A hotfix has been merged towards 2.90. I will keep it open, because there may be other action items needed. Jesse Glick please close it if you feel it's fully done.
            Hide
            burtsevyg Yuriy Burtsev added a comment -

            2.73.4 will be cool.

            Show
            burtsevyg Yuriy Burtsev added a comment - 2.73.4 will be cool.
            Hide
            jglick Jesse Glick added a comment -

            Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.

            Show
            jglick Jesse Glick added a comment - Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/model/User.java
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f
            Log:
            Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

            JENKINS-47909 Handle false hex escapes

            (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)
            Hide
            danielbeck Daniel Beck added a comment -
            Show
            danielbeck Daniel Beck added a comment - Late backport into 2.89.1 per https://groups.google.com/d/msg/jenkinsci-dev/VuRTcIqC-Zw/RauLrKI_BQAJ

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              vrobert78 Vincent Robert
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: