Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12176

Unable to delete a job that has a fstrigger

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • fstrigger-plugin
    • None

      I have a fstrigger that polls every minute, and most of the time when I try to delete the job I get an error:
      java.io.IOException: Unable to delete /some/path/.nfs00000000018e26ca00236d2f

      so I look at that file, and it contains the fstrigger polling log.

      If I disable the fstrigger, then I am able to delete the job.

          [JENKINS-12176] Unable to delete a job that has a fstrigger

          Gregory Boissinot added a comment -

          I'm afraid I can't reproduce.
          I did in my environment lsof | grep 'trigger-polling-folder.log' and nothing appends.

          Maybe it occurs for special use cases when there are problems.
          Do you have other input in order I try to reproduce it?

          Gregory Boissinot added a comment - I'm afraid I can't reproduce. I did in my environment lsof | grep 'trigger-polling-folder.log' and nothing appends. Maybe it occurs for special use cases when there are problems. Do you have other input in order I try to reproduce it?

          Jim Searle added a comment -

          What about:
          lsof | grep 'trigger-polling-files.log'
          But that doesn't show anything for me either...

          But this currently shows 603 lines for me:
          ls -la /proc/5498/fd | grep trigger-polling-files.log
          where 5498 is the jenkins process id, and I don't have 603 jobs using fstrigger. I see a duplicates for some jobs.

          Actually, I was looking at the duplicates, I didn't look at every one, but those that I did look at, the job is disabled!
          The trigger-polling-files.log says: "The job is not buildable. Activate it to poll again."
          Hopefully that will help you debug?

          Sorry my responses are slow, but for some reason jira is not notifying me of comments, so I have to keep checking.

          Jim Searle added a comment - What about: lsof | grep 'trigger-polling-files.log' But that doesn't show anything for me either... But this currently shows 603 lines for me: ls -la /proc/5498/fd | grep trigger-polling-files.log where 5498 is the jenkins process id, and I don't have 603 jobs using fstrigger. I see a duplicates for some jobs. Actually, I was looking at the duplicates, I didn't look at every one, but those that I did look at, the job is disabled! The trigger-polling-files.log says: "The job is not buildable. Activate it to poll again." Hopefully that will help you debug? Sorry my responses are slow, but for some reason jira is not notifying me of comments, so I have to keep checking.

          Gregory Boissinot added a comment -

          I am on a MacOSX system.
          What is /proc?

          Gregory Boissinot added a comment - I am on a MacOSX system. What is /proc?

          Jim Searle added a comment -

          It's a UNIX thing, there is a directory in /proc for each process id, and /proc/<pid>/fd contains a list of all files opened by that <pid>. Sorry, I don't know OSX at all...

          I was following the instructions here:
          https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error

          Can you see anything in the code where if a job is disabled it might not close the file?

          Jim Searle added a comment - It's a UNIX thing, there is a directory in /proc for each process id, and /proc/<pid>/fd contains a list of all files opened by that <pid>. Sorry, I don't know OSX at all... I was following the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error Can you see anything in the code where if a job is disabled it might not close the file?

          Jim Searle added a comment -

          Any clue what is going on here?

          Jim Searle added a comment - Any clue what is going on here?

          Gregory Boissinot added a comment -

          I tested with the File Leak Detector Plugin and nothing appends in my environment.
          I've got not way to reproduce and anyway to fix it.
          Feel free to look at the source code if you have any Java skills.

          Gregory Boissinot added a comment - I tested with the File Leak Detector Plugin and nothing appends in my environment. I've got not way to reproduce and anyway to fix it. Feel free to look at the source code if you have any Java skills.

          Jim Searle added a comment -

          Thanks for continuing to look into this. It seems that Jenkins is running the trigger multiple times, we have a lot of jobs using the file-trigger plugin, with a polling time of every minute... I guess we need to increase this polling time to avoid this?

          Poll @ 10:22:28
          524 descriptors are open

          #5 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:18:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          #98 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:20:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          #280 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:21:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          #370 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:22:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          #404 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:19:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          Poll @ 10:26

          #2 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:24:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          #127 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:25:28 PDT 2012
          at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
          at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
          at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
          at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          Jim Searle added a comment - Thanks for continuing to look into this. It seems that Jenkins is running the trigger multiple times, we have a lot of jobs using the file-trigger plugin, with a polling time of every minute... I guess we need to increase this polling time to avoid this? Poll @ 10:22:28 524 descriptors are open #5 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:18:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #98 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:20:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #280 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:21:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #370 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:22:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #404 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:19:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Poll @ 10:26 #2 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:24:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #127 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:25:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462)

          Gregory Boissinot added a comment -

          Could you test with fstrigger 0.31?
          It should fix the issue.
          Thanks

          Gregory Boissinot added a comment - Could you test with fstrigger 0.31? It should fix the issue. Thanks

          Jim Searle added a comment -

          Thanks! I've installed 0.31 and will let you know.

          Jim Searle added a comment - Thanks! I've installed 0.31 and will let you know.

          Jim Searle added a comment -

          Looks like this is fixed now. Thanks for the support!

          Jim Searle added a comment - Looks like this is fixed now. Thanks for the support!

            gbois Gregory Boissinot
            jimsearle Jim Searle
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: