Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47909

Last master version 2.73.3 won't start when usernames contain $

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Fixed
    • core
    • Docker: jenkins/jenkins:2.73.3-alpine
      Kubernetes: GKE v1.8.1-gke.0

      Upgrade from: jenkins/jenkins:2.73.2-alpine

    Description

      Hi,

      after upgrade our deployment in the last version 2.73.3, i had to rollback in 2.73.2.

      The master won't start:

       

      INFO: Loaded all jobs
      Nov 09, 2017 8:40:54 AM jenkins.util.groovy.GroovyHookScript execute
      INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Started Download metadata
      Nov 09, 2017 8:40:54 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Finished Download metadata. 37 ms
      Nov 09, 2017 8:40:54 AM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed AllUsers.scanAll
      java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM hudson.util.BootFailure publish
      SEVERE: Failed to initialize Jenkins
      hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.WebAppMain$3.run(WebAppMain.java:252)
      Caused by: org.jvnet.hudson.reactor.ReactorException: java.lang.Error: java.lang.reflect.InvocationTargetException
       at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
       at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
       at jenkins.model.Jenkins.executeReactor(Jenkins.java:1124)
       at jenkins.model.Jenkins.<init>(Jenkins.java:929)
       at hudson.model.Hudson.<init>(Hudson.java:86)
       at hudson.model.Hudson.<init>(Hudson.java:82)
       at hudson.WebAppMain$3.run(WebAppMain.java:235)
      Caused by: java.lang.Error: java.lang.reflect.InvocationTargetException
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
       at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
       ... 8 more
      Caused by: java.lang.NumberFormatException: For input string: "whco"
       at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
       at java.lang.Integer.parseInt(Integer.java:580)
       at jenkins.model.IdStrategy$CaseInsensitive.idFromFilename(IdStrategy.java:306)
       at hudson.model.User$AllUsers.scanAll(User.java:1056)
       ... 13 more
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Stopping Jenkins
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Started termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins$21 onAttained
      INFO: Completed termination
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager
      INFO: Stopping plugin manager
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpPersistQueue
      INFO: Persisting build queue
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects
      INFO: Waiting for node disconnection completion
      Nov 09, 2017 8:40:54 AM jenkins.model.Jenkins cleanUp
      INFO: Jenkins stopped
      --> setting agent port for jnlp
      Nov 09, 2017 8:41:04 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
      SEVERE: A thread (Thread-3/50) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
      java.lang.NullPointerException: Cannot invoke method setSlaveAgentPort() on null object
       at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
       at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy:10)
       at tcp-slave-agent-port$_run_closure1.doCall(tcp-slave-agent-port.groovy)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
       at groovy.lang.Closure.call(Closure.java:414)
       at groovy.lang.Closure.call(Closure.java:408)
       at groovy.lang.Closure.run(Closure.java:495)
       at java.lang.Thread.run(Thread.java:745)
      

      The rollback in 2.73.2 has solved the problem.

       

      Attachments

        Issue Links

          Activity

            vrobert78 Vincent Robert created issue -
            oleg_nenashev Oleg Nenashev added a comment -

            I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it

            oleg_nenashev Oleg Nenashev added a comment - I see no change which could cause such error in .3 directly, but I will raise the priority in order to investigate it
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Priority Minor [ 4 ] Critical [ 2 ]
            danielbeck Daniel Beck added a comment -

            This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921.

            vrobert78 Please provide a list of directory names inside JENKINS_HOME/users.

            danielbeck Daniel Beck added a comment - This is very likely caused by the user ID security fix (SECURITY-499). Same issue reported against 2.89 in JENKINS-47921 . vrobert78 Please provide a list of directory names inside JENKINS_HOME/users .
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-47921 [ JENKINS-47921 ]
            danielbeck Daniel Beck added a comment -

            My guess would be there's a user whose name is $whco or that contains that string.

            danielbeck Daniel Beck added a comment - My guess would be there's a user whose name is $whco or that contains that string.
            jglick Jesse Glick made changes -
            Link This issue blocks SECURITY-499 [ SECURITY-499 ]
            jglick Jesse Glick made changes -
            Labels lts-candidate regression
            danielbeck Daniel Beck made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            jglick Jesse Glick made changes -
            Assignee Rebecca Ysteboe [ rysteboe ]

            You are right, there is a user with a $ :

            drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35'

            Can I do a "rm -rf" of the folder ?

             

             

            vrobert78 Vincent Robert added a comment - You are right, there is a user with a $ : drwxr-xr-x 2 jenkins jenkins 4096 Oct 10 20:32 'c$whcomiti35' Can I do a "rm -rf" of the folder ?    
            jglick Jesse Glick added a comment -

            Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.

            jglick Jesse Glick added a comment - Filed a possible fix. Well, it would certainly fix the exception, I am just not sure the behavior thereafter will be right.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 3134 (Web Link)" [ 18011 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick added a comment -

            vrobert78 as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.

            jglick Jesse Glick added a comment - vrobert78 as a hotfix, yes you could do that. Any settings for that user (API tokens, etc.) will be lost in such a case.
            jglick Jesse Glick made changes -
            Assignee Rebecca Ysteboe [ rysteboe ] Jesse Glick [ jglick ]

            jglick , the hotfix is OK. I have deleted the folder, everything is fine by now !

             

            vrobert78 Vincent Robert added a comment - jglick , the hotfix is OK. I have deleted the folder, everything is fine by now !  
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            danielbeck Daniel Beck added a comment - - edited

            Any settings for that user (API tokens, etc.) will be lost in such a case.

            I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's c$whcomiti35 that would be c$0024whcomiti35.

            danielbeck Daniel Beck added a comment - - edited Any settings for that user (API tokens, etc.) will be lost in such a case. I tried locally, a safer workaround seems to be to add the string 0024 after every instance of $ in the user name. In the case of the reporter's  c$whcomiti35 that would be c$0024whcomiti35 .
            martoeng Martin Walter added a comment -

            We had the same problem (JENKINS-47921). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?

            martoeng Martin Walter added a comment - We had the same problem ( JENKINS-47921 ). The "bad" user was m...$reis. The fun fact here is that we authenticate against an Active Directory and we have no registered user names containing dollar signs. Perhaps It comes from a commit from an external repository and Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? If so, Jenkins would knock out himself like he did here?

            martoeng It's the same for us, we do not use the internal authentication system, we use the github auth.

            In the users directory, many users are from externals repos.

             

            vrobert78 Vincent Robert added a comment - martoeng  It's the same for us, we do not use the internal authentication system, we use the github auth. In the users directory, many users are from externals repos.  
            martoeng Martin Walter added a comment -

            So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

            martoeng Martin Walter added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?
            danielbeck Daniel Beck added a comment -

            So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again?

            The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration.

            Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation?

            Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.

            danielbeck Daniel Beck added a comment - So... may this happen any time in the future again when I update/restart Jenkins? Or are the usernames saved from now on in a way that this may not happen again? The latter, it's just that the migration didn't handle this particular case. My proposed workaround above is basically is a manual directory name migration. Jenkins recognised a user for a commit and saved his name to hard disk? Would that be an explanation? Yes, Jenkins uses the same storage and data structures for "users", both those able to log in, and those that are not. The latter are generally not saved to disk though.
            recampbell Ryan Campbell made changes -
            Summary Last master version 2.73.3 won't start after upgrade from 2.73.2 Last master version 2.73.3 won't start when usernames contain $

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b
            Log:
            JENKINS-47909 Handle false hex escapes.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java http://jenkins-ci.org/commit/jenkins/b6751222ba176b1aba396ea7dfffb977566b0d7b Log: JENKINS-47909 Handle false hex escapes.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d
            Log:
            JENKINS-47909 Migration test.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/e2d36820da22b734f1bce92545a950ab97d39f8d Log: JENKINS-47909 Migration test.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/User.java
            http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad
            Log:
            JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/User.java http://jenkins-ci.org/commit/jenkins/5d2b8248d361948de7a8ea9595521c9578f7fcad Log: JENKINS-47909 Prudent to add a log message when we are migrating a user record for any reason.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831
            Log:
            JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/48fcab9862e002546a2001ffa5a4584d93e5c831 Log: JENKINS-47909 @daniel-beck asks about usernames containing valid $ escapes. Behavior is partly broken.

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/model/User.java
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f
            Log:
            Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

            JENKINS-47909 Handle false hex escapes

            Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/7c06a9ba946800151c5d9fc38793ac4bbd3fea5f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes Compare: https://github.com/jenkinsci/jenkins/compare/4c11a0ffad4f...7c06a9ba9468
            martoeng Martin Walter added a comment -

            They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.

            martoeng Martin Walter added a comment - They shouldn't get persisted, yet they somehow do... although that was before we switched to AD authentication. Anyhow, thanks for the quick help. If futher information is required, don't hesitate and contact me.
            oleg_nenashev Oleg Nenashev added a comment -

            A hotfix has been merged towards 2.90.
            I will keep it open, because there may be other action items needed. jglick please close it if you feel it's fully done.

            oleg_nenashev Oleg Nenashev added a comment - A hotfix has been merged towards 2.90. I will keep it open, because there may be other action items needed. jglick please close it if you feel it's fully done.
            burtsevyg Yuriy Burtsev added a comment -

            2.73.4 will be cool.

            burtsevyg Yuriy Burtsev added a comment - 2.73.4 will be cool.
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-48131 [ JENKINS-48131 ]
            jglick Jesse Glick added a comment -

            Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.

            jglick Jesse Glick added a comment - Reported bug was fixed. There was discussion about far riskier changes to the entire storage model of users in Jenkins, but that is certainly out of scope here.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/model/User.java
            core/src/main/java/jenkins/model/IdStrategy.java
            core/src/test/java/jenkins/model/IdStrategyTest.java
            test/src/test/java/hudson/model/UserTest.java
            test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip
            http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f
            Log:
            Merge pull request #3134 from jglick/IdStrategy-NFE-JENKINS-47909

            JENKINS-47909 Handle false hex escapes

            (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/User.java core/src/main/java/jenkins/model/IdStrategy.java core/src/test/java/jenkins/model/IdStrategyTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/resources/hudson/model/UserTest/shellyUsernameMigrated.zip http://jenkins-ci.org/commit/jenkins/8058675317d84b51a73e53a9167c6d1660b3314f Log: Merge pull request #3134 from jglick/IdStrategy-NFE- JENKINS-47909 JENKINS-47909 Handle false hex escapes (cherry picked from commit 7c06a9ba946800151c5d9fc38793ac4bbd3fea5f)
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2565 (Web Link)" [ 18248 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate regression 2.89.1-fixed regression
            danielbeck Daniel Beck added a comment - Late backport into 2.89.1 per https://groups.google.com/d/msg/jenkinsci-dev/VuRTcIqC-Zw/RauLrKI_BQAJ

            People

              jglick Jesse Glick
              vrobert78 Vincent Robert
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: