Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5403

Perforce plugin hangs executor if username does not match email address

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • p4-plugin
    • None
    • Hudson 1.328
      Perforce plugin 1.0.16
      Extended Email (email-ext) plugin 2.2.1
      Perforce server 2007.2

      The new Perforce plugin can cause jobs to hang when trying to get the email address of a user whose Perforce username does not match the part of the email address before the @-sign. Given this user:

      $ p4 user -o bobby
      # A Perforce User Specification.
      #
      #  User:        The user's user name.
      #  Email:       The user's email address; for email review.
      #  Update:      The date this specification was last modified.
      #  Access:      The date this user was last active.  Read only.
      #  FullName:    The user's real name.
      #  JobView:     Selects jobs for inclusion during changelist creation.
      #  Password:    If set, user must have matching $P4PASSWD on client.
      #  Reviews:     Listing of depot files to be reviewed by user.
      
      User:   bobby
      
      Email:  bobsmith@example.com
      
      Update: 2009/10/27 13:23:04
      
      Access: 2010/01/27 09:36:35
      
      FullName:       Bobby Smith
      
      Password:       ******
      

      the following log entries and exception appear in the Tomcat logfiles:

      [install-server-main-L8-cluster-SPLUS-manager-2] $ p4 user -o bobby
      58973079 [Executor #1 for master : executing validate-server-main-L8-cluster-SPLUS #11] WARN perforce - Perforce process terminated suddenly
      58973080 [Executor #1 for master : executing validate-server-main-L8-cluster-SPLUS #11] WARN perforce - java.io.InterruptedIOException
              at java.io.PipedInputStream.read(Unknown Source)
              at java.io.PipedInputStream.read(Unknown Source)
              at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
              at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
              at sun.nio.cs.StreamDecoder.read(Unknown Source)
              at java.io.InputStreamReader.read(Unknown Source)
              at java.io.BufferedReader.fill(Unknown Source)
              at java.io.BufferedReader.readLine(Unknown Source)
              at java.io.BufferedReader.readLine(Unknown Source)
              at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:278)
              at com.tek42.perforce.parse.Users.getUser(Users.java:52)
              at hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:32)
              at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:86)
              at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:476)
              at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:303)
              at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:249)
              at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:241)
              at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:199)
              at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
              at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:501)
              at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:487)
              at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:475)
              at hudson.model.Build$RunnerImpl.cleanUp(Build.java:187)
              at hudson.model.Run.run(Run.java:1162)
              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
              at hudson.model.ResourceController.execute(ResourceController.java:88)
              at hudson.model.Executor.run(Executor.java:123)
      
      Could not get email address from Perforce: No output for: p4 user -o bobby
      

      Meanwhile, the console log is stuck at:

      Started by upstream project "install-server-main-L8-cluster-SPLUS-worker-1" build number 13
      Building on master
      Using master perforce client: validate-server
      [workspace] $ p4 workspace -o validate-server
      Changing P4 Client Root to: /opt/hudson/hudson/jobs/validate-server-main-L8-cluster-SPLUS/workspace/
      Saving modified client validate-server
      [workspace] $ p4 -s client -i
      Last sync'd change: 306721
      [workspace] $ p4 counter change
      [workspace] $ p4 -s changes //validate-server/...@306722,@306741
      Sync'ing workspace to changelist 306741.
      [workspace] $ p4 sync //validate-server/...@306741
      Sync complete, took 99 ms
      [deploy] $ /opt/hudson/hudson/tools/ant/bin/ant -file validate.xml -Dservice.url=http://searil6406v:8080/SplusServer test-spserver-available
      Buildfile: validate.xml
      
      test-spserver-available:
      Build timed out. Aborting
      Email was triggered for: Failure
      There are 1 triggered emails.
      Sending email for trigger: Failure
      [spinner]
      

          [JENKINS-5403] Perforce plugin hangs executor if username does not match email address

          Brian Bassett created issue -

          Brian Bassett added a comment -

          Forgot to set component information

          Brian Bassett added a comment - Forgot to set component information
          Brian Bassett made changes -
          Component/s New: perforce [ 15506 ]
          Affects Version/s New: current [ 10162 ]

          Rob Petti added a comment -

          Can you verify that it does NOT hang when the e-mail is set to bobby@example.com? The code does not extend any special treatment for email addresses; it's just another field in the perforce response.

          Also please ensure that all your perforce jobs are correctly configured, not just the one you are running.

          Rob Petti added a comment - Can you verify that it does NOT hang when the e-mail is set to bobby@example.com? The code does not extend any special treatment for email addresses; it's just another field in the perforce response. Also please ensure that all your perforce jobs are correctly configured, not just the one you are running.

          Brian Bassett added a comment -

          I can verify it does not hang with other users who have matching usernames and email addresses. eg,

          $ p4 user -o bbassett
          ...
          Email: bbassett@example.com
          ...

          Neither users have configuration records in HUDSON_HOME/users/${USERNAME}/config.xml.

          All Perforce jobs appear to be configured correctly. They sync correctly from the depot when triggered; if there is a different definition to "correctly configured", I'd appreciate knowing

          I forgot to mention this in the original report, but this configuration worked with version 1.0.15 of the Perforce plugin. Updating to 1.0.16 caused the issue to appear.

          Our Perforce server is not named "perforce"; it is instead named "seaperforce.example.com", since the company I work for was acquired and the new company already had a "perforce.example.com". This issue regularly trips up our users, and might be a possible cause here.

          Brian Bassett added a comment - I can verify it does not hang with other users who have matching usernames and email addresses. eg, $ p4 user -o bbassett ... Email: bbassett@example.com ... Neither users have configuration records in HUDSON_HOME/users/${USERNAME}/config.xml. All Perforce jobs appear to be configured correctly. They sync correctly from the depot when triggered; if there is a different definition to "correctly configured", I'd appreciate knowing I forgot to mention this in the original report, but this configuration worked with version 1.0.15 of the Perforce plugin. Updating to 1.0.16 caused the issue to appear. Our Perforce server is not named "perforce"; it is instead named "seaperforce.example.com", since the company I work for was acquired and the new company already had a "perforce.example.com". This issue regularly trips up our users, and might be a possible cause here.

          Rob Petti added a comment -

          As long as all the jobs can sync properly then that's fine. The MailAddressResolver classes cannot be tied to jobs, so it's forced to go through the perforce configurations one by one until it finds the user it's looking for (this includes disabled jobs as well, apparently). If a job isn't correctly configured, that could result in a hang (though really it should time out after a while).

          In any case, I can't reproduce this behavior at all using 1.0.16 on hudson 1.318, or 1.0.18 on hudson 1.339.

          It looks like the reason that it 'worked' in 1.0.15 is because the mail resolver was outright disabled. It looks like another user was running into the same issue, and just commented the entire thing out instead of trying to fix it.

          One thing that comes to mind is that while perforce commands are run on the slave during a build, mail resolution actually takes place on the master. If the perforce server is reachable from the slaves, but not the master, then that would produce exactly the behavior you are seeing.

          Rob Petti added a comment - As long as all the jobs can sync properly then that's fine. The MailAddressResolver classes cannot be tied to jobs, so it's forced to go through the perforce configurations one by one until it finds the user it's looking for (this includes disabled jobs as well, apparently). If a job isn't correctly configured, that could result in a hang (though really it should time out after a while). In any case, I can't reproduce this behavior at all using 1.0.16 on hudson 1.318, or 1.0.18 on hudson 1.339. It looks like the reason that it 'worked' in 1.0.15 is because the mail resolver was outright disabled. It looks like another user was running into the same issue, and just commented the entire thing out instead of trying to fix it. One thing that comes to mind is that while perforce commands are run on the slave during a build, mail resolution actually takes place on the master. If the perforce server is reachable from the slaves, but not the master, then that would produce exactly the behavior you are seeing.

          aflat added a comment -

          I'm having the exact same issue, it doesn't appear to timeout either, I let it go about 16 hours, and it was still hung. You also can't kill the job, I have to restart the Hudson master. A bit more info

          Hudson 1.345 and 1.343
          Perforce plugin 1.0.18
          Perforce 2009.1(the api did change with 2009.1 so this may be something to look at)

          It does work for users where the username and email match, but if any of the submitters don't match it hangs.

          aflat added a comment - I'm having the exact same issue, it doesn't appear to timeout either, I let it go about 16 hours, and it was still hung. You also can't kill the job, I have to restart the Hudson master. A bit more info Hudson 1.345 and 1.343 Perforce plugin 1.0.18 Perforce 2009.1(the api did change with 2009.1 so this may be something to look at) It does work for users where the username and email match, but if any of the submitters don't match it hangs.

          aflat added a comment - - edited

          A bit more info. I'm running the hudson master on a windows box, as a service, using the localsystem account. Even though I create a p4ticket file(as a daemon user), it looks like the perforce plugin is having a hard time accessing it. I was playing with the tek42 api's and I couldn't access my server using my daemon p4 account. Once I switched to my local account, then I could switch back to the daemon account, and everything was fine. So

          try {
                             Depot depot = new Depot();
                              depot.setUser("daemonuser");
                 
                              depot.setPassword("daemonuserspassword");
          
                              depot.setPort("perforce:1666");
                              System.out.println("de: "+ depot.info());
                              com.tek42.perforce.model.User pu =depot.getUsers().getUser("ARealUser");
          
                              if (pu.getEmail() != null && !pu.getEmail().equals(""))
                                  System.out.println("em" + pu.getEmail());
          
                          } catch (Exception e) {
                              System.out.println("Could not get email address from Perforce: "+e.getMessage());
                          }
          

          This failed, the p4 info worked, but I couldn't get the email. Once I changed the depot.setusername to ARealUser, then I could get the email. Changing the depot.setusername back to daemonuser worked after that. Logging into p4v doesn't seem to fix anything.

          So for valid testing it looks like you need to remove your ticket file.

          aflat added a comment - - edited A bit more info. I'm running the hudson master on a windows box, as a service, using the localsystem account. Even though I create a p4ticket file(as a daemon user), it looks like the perforce plugin is having a hard time accessing it. I was playing with the tek42 api's and I couldn't access my server using my daemon p4 account. Once I switched to my local account, then I could switch back to the daemon account, and everything was fine. So try { Depot depot = new Depot(); depot.setUser( "daemonuser" ); depot.setPassword( "daemonuserspassword" ); depot.setPort( "perforce:1666" ); System .out.println( "de: " + depot.info()); com.tek42.perforce.model.User pu =depot.getUsers().getUser( "ARealUser" ); if (pu.getEmail() != null && !pu.getEmail().equals("")) System .out.println( "em" + pu.getEmail()); } catch (Exception e) { System .out.println( "Could not get email address from Perforce: " +e.getMessage()); } This failed, the p4 info worked, but I couldn't get the email. Once I changed the depot.setusername to ARealUser, then I could get the email. Changing the depot.setusername back to daemonuser worked after that. Logging into p4v doesn't seem to fix anything. So for valid testing it looks like you need to remove your ticket file.

          Rob Petti added a comment -

          bbassett: are you also running hudson on a windows server? Is it running as a service?

          As a quick note, p4 info doesn't require a valid login (or even a username for that matter) in order to run, so it may not be the best test.

          Tested with:
          Hudson 1.345
          Perforce Plugin 1.0.18
          email-ext 2.5
          p4 client 2009.1
          p4d server 2009.2
          Perforce security level 3

          Removing the tickets file doesn't reproduce the problem on my end.

          I've made some changes to the perforce email resolver that will make it a bit more robust. They'll be in 1.0.19, so please test again using that release.

          Rob Petti added a comment - bbassett: are you also running hudson on a windows server? Is it running as a service? As a quick note, p4 info doesn't require a valid login (or even a username for that matter) in order to run, so it may not be the best test. Tested with: Hudson 1.345 Perforce Plugin 1.0.18 email-ext 2.5 p4 client 2009.1 p4d server 2009.2 Perforce security level 3 Removing the tickets file doesn't reproduce the problem on my end. I've made some changes to the perforce email resolver that will make it a bit more robust. They'll be in 1.0.19, so please test again using that release.

          Brian Bassett added a comment -

          No. We are running on a Linux master with both Linux and Windows slaves experiencing the issue.

          Brian Bassett added a comment - No. We are running on a Linux master with both Linux and Windows slaves experiencing the issue.

            rpetti Rob Petti
            bbassett Brian Bassett
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: