Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47909

Last master version 2.73.3 won't start when usernames contain $

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Docker: jenkins/jenkins:2.73.3-alpine
      Kubernetes: GKE v1.8.1-gke.0

      Upgrade from: jenkins/jenkins:2.73.2-alpine

      Hi,

      after upgrade our deployment in the last version 2.73.3, i had to rollback in 2.73.2.

      The master won't start:

       

      INFO: Loaded all jobs
      Nov 09, 2017 8:40:54 AM jenkins.util.groovy.GroovyHookScript execute
      INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Started Download metadata
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Finished Download metadata. 37 ms
      Nov 09, 2017 8:40:54 AM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed AllUsers.scanAll
      java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM hudson.util.BootFailure publish
      SEVERE: Failed to initialize Jenkins
      hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.WebAppMain$3.run(WebAppMain.java:252)
      Caused by: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
       at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
       at jenkins.model.Jenkins.executeReactor(Jenkins.java:1124)
       at jenkins.model.Jenkins.<init>(Jenkins.java:929)
       at hudson.model.Hudson.<init>(Hudson.java:86)
       at hudson.model.Hudson.<init>(Hudson.java:82)
       at hudson.WebAppMain$3.run(WebAppMain.java:235)
      Caused by: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Stopping Jenkins
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Started termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Completed termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager
      INFO: Stopping plugin manager
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpPersistQueue
      INFO: Persisting build queue
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects
      INFO: Waiting for node disconnection completion
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Jenkins stopped
      --> setting agent port for jnlp
      Nov 09, 2017 8:41:04 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
      SEVERE: A thread (Thread-3/50) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
      java.lang.NullPointerException: Cannot invoke method setSlaveAgentPort() on null object
       at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
       at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy:10)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
       at groovy.lang.Closure.call(Closure.java:414)
       at groovy.lang.Closure.call(Closure.java:408)
       at groovy.lang.Closure.run(Closure.java:495)
       at java.lang.Thread.run(Thread.java:745)
      

      The rollback in 2.73.2 has solved the problem.

       

          [JENKINS-47909] Last master version 2.73.3 won't start when usernames contain $

          Oleg Nenashev added a comment -

          I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it

          Oleg Nenashev added a comment - I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it

          Daniel Beck added a comment -

          This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921.

          vrobert78 Please provide a list of directory names inside JENKINS_HOME/users.

          Daniel Beck added a comment - This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921 . vrobert78 Please provide a list of directory names inside JENKINS_HOME/users .

          Daniel Beck added a comment -

          My guess would be there's a user whose name is $whco or that contains that string.

          Daniel Beck added a comment - My guess would be there's a user whose name is $whco or that contains that string.

          You are right, there is a user with a $ :

          drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35'

          Can I do a "rm -rf" of the folder ?

           

           

          Vincent Robert added a comment - You are right, there is a user with a $ : drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35' Can I do a "rm -rf" of the folder ?    

          Jesse Glick added a comment -

          Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.

          Jesse Glick added a comment - Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.

          Jesse Glick added a comment -

          vrobert78 as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.

          Jesse Glick added a comment - vrobert78 as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.

          jglick , the hotfix is OK. I have deleted the folder, everything is fine by now !

           

          Vincent Robert added a comment - jglick , the hotfix is OK. I have deleted the folder, everything is fine by now !  

          Daniel Beck added a comment - - edited

          Any settings for that user (API tokens, etc.) will be lost in such a case.

          I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's c$whcomiti35 that would be c$0024whcomiti35.

          Daniel Beck added a comment - - edited Any settings for that user (API tokens, etc.) will be lost in such a case. I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's  c$whcomiti35 that would be c$0024whcomiti35 .

          Martin Walter added a comment -

          We had the same problem (JENKINS-47921). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?

          Martin Walter added a comment - We had the same problem ( JENKINS-47921 ). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?

          martoeng It's the same for us, we do not use the internal authentication system, we use the github auth.

          In the users directory, many users are from externals repos.

           

          Vincent Robert added a comment - martoeng  It's the same for us, we do not use the internal authentication system, we use the github auth. In the users directory, many users are from externals repos.  

          Martin Walter added a comment -

          So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

          Martin Walter added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

          Daniel Beck added a comment -

          So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

          The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration.

          Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation?

          Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.

          Daniel Beck added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again? The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration. Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/jenkins/model/IdStrategy.java
          core/src/test/java/jenkins/model/IdStrategyTest.java
          http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b
          Log:
          JENKINS-47909 Handle false hex escapes.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b Log: JENKINS-47909 Handle false hex escapes.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          test/src/test/java/hudson/model/UserTest.java
          test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
          http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d
          Log:
          JENKINS-47909 Migration test.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d Log: JENKINS-47909 Migration test.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/model/User.java
          http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad
          Log:
          JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/User.java http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad Log: JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          test/src/test/java/hudson/model/UserTest.java
          test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
          http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831
          Log:
          JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831 Log: JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/User.java
          core/src/main/java/jenkins/model/IdStrategy.java
          core/src/test/java/jenkins/model/IdStrategyTest.java
          test/src/test/java/hudson/model/UserTest.java
          test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
          http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f
          Log:
          Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

          JENKINS-47909 Handle false hex escapes

          Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468

          Martin Walter added a comment -

          They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.

          Martin Walter added a comment - They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.

          Oleg Nenashev added a comment -

          A hotfix has been merged towards 2.90.
          I will keep it open, because there may be other action items needed. jglick please close it if you feel it's fully done.

          Oleg Nenashev added a comment - A hotfix has been merged towards 2.90. I will keep it open, because there may be other action items needed. jglick please close it if you feel it's fully done.

          Yuriy Burtsev added a comment -

          2.73.4 will be cool.

          Yuriy Burtsev added a comment - 2.73.4 will be cool.

          Jesse Glick added a comment -

          Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.

          Jesse Glick added a comment - Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/User.java
          core/src/main/java/jenkins/model/IdStrategy.java
          core/src/test/java/jenkins/model/IdStrategyTest.java
          test/src/test/java/hudson/model/UserTest.java
          test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
          http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f
          Log:
          Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

          JENKINS-47909 Handle false hex escapes

          (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)

          Daniel Beck added a comment -

          Daniel Beck added a comment - Late backport into 2.89.1 per https://groups.google.com/d/msg/jenkinsci-dev/VuRTcIqC-Zw/RauLrKI_BQAJ

            jglick Jesse Glick
            vrobert78 Vincent Robert
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: