Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9675

Deleting a Job hangs and blocks all other Perforce activities

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • None
    • Master and Slaves on Linux (RedHat), observed behavior on both Jenkins 1.3999 and 1.411, Perforce Client is Rev. P4/LINUX26X86_64/2010.1/265509 (2010/09/23), Perforce server is P4D/LINUX26X86_64/2010.1/265509 (2010/09/23)

      I'm not sure with what version of the Perforce plugin this started, but currently, using 1.2.5, when we try to delete a job using the UI, it sometimes completes normally, printing the following to the log:

      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Workspace is being deleted; enabling one-time force sync.
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Using remote perforce client: hudson-cjo2011.03-initdb-76240347
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: [cjo2011.03-initdb] $ p4 workspace -o hudson-cjo2011.03-initdb-76240347
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Changing P4 Client Root to: /home/cruise/hudson/workspace/cjo2011.03-initdb
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Saving modified client hudson-cjo2011.03-initdb-76240347
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: [cjo2011.03-initdb] $ p4 -s client -i
      May 11, 2011 4:48:24 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: [cjo2011.03-initdb] $ p4 sync -k //hudson-cjo2011.03-initdb-76240347/...#0

      But more often it only gets this far:

      May 11, 2011 4:49:13 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Workspace is being deleted; enabling one-time force sync.
      May 11, 2011 4:49:13 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: Using remote perforce client: hudson-cjo2011.03-webservices-e2e-76240347
      May 11, 2011 4:49:13 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
      INFO: [cjo2011.03-webservices-e2e] $ p4 workspace -o hudson-cjo2011.03-webservices-e2e-76240347

      At this point the thread handling the request seems to be stuck at:

      Handling POST /view/bugfix/job/cjo2011.03-webservices-e2e/doDelete : RequestHandlerThread62
      java.lang.Object.wait(Native Method)
      hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:173)
      ...
      java.io.BufferedReader.readLine(BufferedReader.java:362)
      com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:329)
      com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:291)
      com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
      hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1183)
      hudson.plugins.perforce.PerforceSCM.processWorkspaceBeforeDeletion(PerforceSCM.java:2202)

      I've attached the full stack trace from the 'monitoring' page (jenkinsDeleteThread_stackTrace.txt). When this happens, all perforce polling stops. If I kill this thread using the monitor page, perforce activities start up again, but the job that was to be deleted is still present.

      We've also tried deleting a job with the CLI client, and this also hangs, but if we kill the client, the job does get deleted. I've also attached the log output from when this hung client is killed (jenkinsKilledCLIClient_logOutput.txt).

      Let me know if there is any other information I can provide to help debug this.

          [JENKINS-9675] Deleting a Job hangs and blocks all other Perforce activities

          Rob Petti added a comment -

          Are the slaves that the job ran on still up and running when you delete the job?

          Basically, when you request a job deletion (regardless of whether you use the CLI or the web GUI), the perforce plugin will sync the workspace to revision #0 (see ENKINS-8118). If for some reason the slave is unstable or in the process of shutting down, it will hang.

          Rob Petti added a comment - Are the slaves that the job ran on still up and running when you delete the job? Basically, when you request a job deletion (regardless of whether you use the CLI or the web GUI), the perforce plugin will sync the workspace to revision #0 (see ENKINS-8118). If for some reason the slave is unstable or in the process of shutting down, it will hang.

          emmulator added a comment -

          We have definitely seen this behavior when all the slaves were up and responding normally.

          emmulator added a comment - We have definitely seen this behavior when all the slaves were up and responding normally.

          Rob Petti added a comment -

          And it only ever hangs when deleting a job?

          I'm trying to get a handle on why this perforce action would be different from any other that takes place in the plugin...

          Rob Petti added a comment - And it only ever hangs when deleting a job? I'm trying to get a handle on why this perforce action would be different from any other that takes place in the plugin...

          emmulator added a comment -

          Yes, we've only ever seen it hang when deleting a Job.

          emmulator added a comment - Yes, we've only ever seen it hang when deleting a Job.

          emmulator added a comment -

          I've received some notifications recently for fixes on <https://issues.jenkins-ci.org/browse/JENKINS-9673> and <https://issues.jenkins-ci.org/browse/JENKINS-9674>, which are duplicates of this issue. All three claim to have been created by me, but I don't recall creating the issue multiple times or having any trouble when I created it. However, if the issue has been fixed, this one should be updated too. I can't actually see what changes were made for the other issues, could someone please take a look at those and update this issue appropriately? Also, it's not clear from those other issues whether the fix affected the core or just the Perforce plugin?

          emmulator added a comment - I've received some notifications recently for fixes on < https://issues.jenkins-ci.org/browse/JENKINS-9673 > and < https://issues.jenkins-ci.org/browse/JENKINS-9674 >, which are duplicates of this issue. All three claim to have been created by me, but I don't recall creating the issue multiple times or having any trouble when I created it. However, if the issue has been fixed, this one should be updated too. I can't actually see what changes were made for the other issues, could someone please take a look at those and update this issue appropriately? Also, it's not clear from those other issues whether the fix affected the core or just the Perforce plugin?

          I think that JIRA has been creating multiple copies of some issues (including this one). The notifications that you received were where those "additional" copies were being closed down as duplicates of this one.

          I am not aware of a fix being made for your issue yet.

          Richard Mortimer added a comment - I think that JIRA has been creating multiple copies of some issues (including this one). The notifications that you received were where those "additional" copies were being closed down as duplicates of this one. I am not aware of a fix being made for your issue yet.

          Rob Petti added a comment -

          Yeah, you can see that the issues were resolved with the "Duplicate" resolution, rather than "Fixed". This issue is still not fixed.

          Rob Petti added a comment - Yeah, you can see that the issues were resolved with the "Duplicate" resolution, rather than "Fixed". This issue is still not fixed.

          emmulator added a comment -

          Aha – you're right. I guess I got confused because they were actually each closed as duplicates of each other, and neither of them mentioned this ticket, so I thought something had been done there but not here. Now I understand, thanks.

          emmulator added a comment - Aha – you're right. I guess I got confused because they were actually each closed as duplicates of each other, and neither of them mentioned this ticket, so I thought something had been done there but not here. Now I understand, thanks.

          Ben Sluis added a comment -

          I also see this issue on the latest version of the plugin (1.3.5) with Jenkins 1.437 on Windows OS for master and slaves.
          It occurs every time I try to delete a job that executed on slave machines.

          Log:
          Dec 12, 2011 12:56:23 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
          INFO: [build_debug] $ "C:\Program Files\Perforce\p4.exe" workspace o jenkins_build_debug-1971592563

          Dec 12, 2011 12:56:22 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
          INFO: Using remote perforce client: jenkins_build_debug--1971592563

          Dec 12, 2011 12:56:20 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
          INFO: Workspace is being deleted; enabling one-time force sync.

          How do I view the stack trace?

          Ben Sluis added a comment - I also see this issue on the latest version of the plugin (1.3.5) with Jenkins 1.437 on Windows OS for master and slaves. It occurs every time I try to delete a job that executed on slave machines. Log: Dec 12, 2011 12:56:23 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion INFO: [build_debug] $ "C:\Program Files\Perforce\p4.exe" workspace o jenkins_build_debug -1971592563 Dec 12, 2011 12:56:22 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion INFO: Using remote perforce client: jenkins_build_debug--1971592563 Dec 12, 2011 12:56:20 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion INFO: Workspace is being deleted; enabling one-time force sync. How do I view the stack trace?

          Rob Petti added a comment -

          http://jenkinsurl/threadDump will show you the stacks for all threads.

          I fixed a potential hang in remote execution calls that might resolve this issue, at least partially. You can find the snapshot here: http://files.robpetti.com/perforce-plugin/target/perforce.hpi or just wait until the next release (which should be within a week).

          Rob Petti added a comment - http://jenkinsurl/threadDump will show you the stacks for all threads. I fixed a potential hang in remote execution calls that might resolve this issue, at least partially. You can find the snapshot here: http://files.robpetti.com/perforce-plugin/target/perforce.hpi or just wait until the next release (which should be within a week).

            Unassigned Unassigned
            emmulator emmulator
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: