[JENKINS-12176] Unable to delete a job that has a fstrigger

Gregory Boissinot added a comment - 2011-12-21 13:21

Thanks for raising this issue.
Could you give your Jenkins core version and the fstrigger Jenkins plugin version you are using.

Additionally, is '/some/path' a directory polled by the FSTrigger plugin?

Gregory Boissinot added a comment - 2011-12-21 13:21 Thanks for raising this issue. Could you give your Jenkins core version and the fstrigger Jenkins plugin version you are using. Additionally, is '/some/path' a directory polled by the FSTrigger plugin?

Jim Searle added a comment - 2011-12-22 00:24 - edited

Jenkins version 1.436
fstrigger version 1.18
The /some/path is /JENKINS_HOME/JOB_NAME directory.

thanks for looking into it.

Jim Searle added a comment - 2011-12-22 00:24 - edited Jenkins version 1.436 fstrigger version 1.18 The /some/path is /JENKINS_HOME/JOB_NAME directory. thanks for looking into it.

Gregory Boissinot added a comment - 2011-12-27 20:51

Your monitored folder is the job folder itself. Therefore when you delete the job, it must not have left active processes on this folder.
I think it should be fixed with FSTrigger 0.21. Before this version, there were left processes because they were not properly closed.
Could you upgrade the plugin and test it?

Gregory Boissinot added a comment - 2011-12-27 20:51 Your monitored folder is the job folder itself. Therefore when you delete the job, it must not have left active processes on this folder. I think it should be fixed with FSTrigger 0.21. Before this version, there were left processes because they were not properly closed. Could you upgrade the plugin and test it?

Jim Searle added a comment - 2011-12-27 21:02

The monitored folder is not the job folder, it is a different path. The file that is locked is the trigger-polling-files.log file in the job workspace.

But, I will try the 0.21 to see if that solves the problem.

Jim Searle added a comment - 2011-12-27 21:02 The monitored folder is not the job folder, it is a different path. The file that is locked is the trigger-polling-files.log file in the job workspace. But, I will try the 0.21 to see if that solves the problem.

Gregory Boissinot added a comment - 2012-01-02 20:51

Should be fixed.
The issue was certainly of the same reasons of ~~JENKINS-11907~~ for an another XTrigger plugin.

Gregory Boissinot added a comment - 2012-01-02 20:51 Should be fixed. The issue was certainly of the same reasons of JENKINS-11907 for an another XTrigger plugin.

Jim Searle added a comment - 2012-06-04 18:14

We are debugging a "too many open files" issue, and following the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error

I do a: ls -la /proc/2141/fd
on our jenkins machine, and it looks like every single job that uses the fstrigger keeps it's log file open all the time:
/<jenkins_home>/jobs/<job_name>/trigger-polling-files.log

Jim Searle added a comment - 2012-06-04 18:14 We are debugging a "too many open files" issue, and following the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error I do a: ls -la /proc/2141/fd on our jenkins machine, and it looks like every single job that uses the fstrigger keeps it's log file open all the time: /<jenkins_home>/jobs/<job_name>/trigger-polling-files.log

Gregory Boissinot added a comment - 2012-06-04 19:32

I tested. I cannot reproduce.
Could you check with a clean environment and the last version.
Thanks

Gregory Boissinot added a comment - 2012-06-04 19:32 I tested. I cannot reproduce. Could you check with a clean environment and the last version. Thanks

Jim Searle added a comment - 2012-06-05 18:50 - edited

The file handles plugin reports that it is coming from xtrigger:

#4 $JENKINS_HOM/jobs/$JOB_NAME/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 04 17:25:00 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Jim Searle added a comment - 2012-06-05 18:50 - edited The file handles plugin reports that it is coming from xtrigger: #4 $JENKINS_HOM/jobs/$JOB_NAME/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 04 17:25:00 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462)

Gregory Boissinot added a comment - 2012-06-05 20:16

I'm afraid I can't reproduce.
I did in my environment lsof | grep 'trigger-polling-folder.log' and nothing appends.

Maybe it occurs for special use cases when there are problems.
Do you have other input in order I try to reproduce it?

Gregory Boissinot added a comment - 2012-06-05 20:16 I'm afraid I can't reproduce. I did in my environment lsof | grep 'trigger-polling-folder.log' and nothing appends. Maybe it occurs for special use cases when there are problems. Do you have other input in order I try to reproduce it?

Jim Searle added a comment - 2012-06-05 22:01

What about:
lsof | grep 'trigger-polling-files.log'
But that doesn't show anything for me either...

But this currently shows 603 lines for me:
ls -la /proc/5498/fd | grep trigger-polling-files.log
where 5498 is the jenkins process id, and I don't have 603 jobs using fstrigger. I see a duplicates for some jobs.

Actually, I was looking at the duplicates, I didn't look at every one, but those that I did look at, the job is disabled!
The trigger-polling-files.log says: "The job is not buildable. Activate it to poll again."
Hopefully that will help you debug?

Sorry my responses are slow, but for some reason jira is not notifying me of comments, so I have to keep checking.

Jim Searle added a comment - 2012-06-05 22:01 What about: lsof | grep 'trigger-polling-files.log' But that doesn't show anything for me either... But this currently shows 603 lines for me: ls -la /proc/5498/fd | grep trigger-polling-files.log where 5498 is the jenkins process id, and I don't have 603 jobs using fstrigger. I see a duplicates for some jobs. Actually, I was looking at the duplicates, I didn't look at every one, but those that I did look at, the job is disabled! The trigger-polling-files.log says: "The job is not buildable. Activate it to poll again." Hopefully that will help you debug? Sorry my responses are slow, but for some reason jira is not notifying me of comments, so I have to keep checking.

Gregory Boissinot added a comment - 2012-06-05 22:28

I am on a MacOSX system.
What is /proc?

Gregory Boissinot added a comment - 2012-06-05 22:28 I am on a MacOSX system. What is /proc?

Jim Searle added a comment - 2012-06-05 22:41

It's a UNIX thing, there is a directory in /proc for each process id, and /proc/<pid>/fd contains a list of all files opened by that <pid>. Sorry, I don't know OSX at all...

I was following the instructions here:
https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error

Can you see anything in the code where if a job is disabled it might not close the file?

Jim Searle added a comment - 2012-06-05 22:41 It's a UNIX thing, there is a directory in /proc for each process id, and /proc/<pid>/fd contains a list of all files opened by that <pid>. Sorry, I don't know OSX at all... I was following the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error Can you see anything in the code where if a job is disabled it might not close the file?

Jim Searle added a comment - 2012-06-08 19:49

Any clue what is going on here?

Jim Searle added a comment - 2012-06-08 19:49 Any clue what is going on here?

Gregory Boissinot added a comment - 2012-06-09 20:41

I tested with the File Leak Detector Plugin and nothing appends in my environment.
I've got not way to reproduce and anyway to fix it.
Feel free to look at the source code if you have any Java skills.

Gregory Boissinot added a comment - 2012-06-09 20:41 I tested with the File Leak Detector Plugin and nothing appends in my environment. I've got not way to reproduce and anyway to fix it. Feel free to look at the source code if you have any Java skills.

Jim Searle added a comment - 2012-06-11 17:32

Thanks for continuing to look into this. It seems that Jenkins is running the trigger multiple times, we have a lot of jobs using the file-trigger plugin, with a polling time of every minute... I guess we need to increase this polling time to avoid this?

Poll @ 10:22:28
524 descriptors are open

#5 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:18:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

#98 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:20:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

#280 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:21:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

#370 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:22:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

#404 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:19:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Poll @ 10:26

#2 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:24:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

#127 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:25:28 PDT 2012
at java.io.FileOutputStream.<init>(FileOutputStream.java:181)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)
at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)
at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Jim Searle added a comment - 2012-06-11 17:32 Thanks for continuing to look into this. It seems that Jenkins is running the trigger multiple times, we have a lot of jobs using the file-trigger plugin, with a polling time of every minute... I guess we need to increase this polling time to avoid this? Poll @ 10:22:28 524 descriptors are open #5 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:18:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #98 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:20:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #280 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:21:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #370 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:22:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #404 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:19:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Poll @ 10:26 #2 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:24:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) #127 $JENKINS_HOME/jobs/<JOB_NAME>/trigger-polling-files.log by thread:Jenkins cron thread on Mon Jun 11 10:25:28 PDT 2012 at java.io.FileOutputStream.<init>(FileOutputStream.java:181) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97) at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90) at org.jenkinsci.lib.xtrigger.AbstractTrigger.run(AbstractTrigger.java:130) at hudson.triggers.Trigger.checkTriggers(Trigger.java:259) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:207) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462)

Gregory Boissinot added a comment - 2012-06-11 21:59

Could you test with fstrigger 0.31?
It should fix the issue.
Thanks

Gregory Boissinot added a comment - 2012-06-11 21:59 Could you test with fstrigger 0.31? It should fix the issue. Thanks

Jim Searle added a comment - 2012-06-12 22:25

Thanks! I've installed 0.31 and will let you know.

Jim Searle added a comment - 2012-06-12 22:25 Thanks! I've installed 0.31 and will let you know.

Jim Searle added a comment - 2012-06-14 21:58

Looks like this is fixed now. Thanks for the support!

Jim Searle added a comment - 2012-06-14 21:58 Looks like this is fixed now. Thanks for the support!

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Gregory Boissinot added a comment - 2011-12-21 13:21

Expand comment: Gregory Boissinot added a comment - 2011-12-21 13:21

Collapse comment: Jim Searle added a comment - 2011-12-22 00:24, Edited by Jim Searle - 2011-12-22 00:26

Expand comment: Jim Searle added a comment - 2011-12-22 00:24, Edited by Jim Searle - 2011-12-22 00:26

Collapse comment: Gregory Boissinot added a comment - 2011-12-27 20:51

Expand comment: Gregory Boissinot added a comment - 2011-12-27 20:51

Collapse comment: Jim Searle added a comment - 2011-12-27 21:02

Expand comment: Jim Searle added a comment - 2011-12-27 21:02

Collapse comment: Gregory Boissinot added a comment - 2012-01-02 20:51

Expand comment: Gregory Boissinot added a comment - 2012-01-02 20:51

Collapse comment: Jim Searle added a comment - 2012-06-04 18:14

Expand comment: Jim Searle added a comment - 2012-06-04 18:14

Collapse comment: Gregory Boissinot added a comment - 2012-06-04 19:32

Expand comment: Gregory Boissinot added a comment - 2012-06-04 19:32

Collapse comment: Jim Searle added a comment - 2012-06-05 18:50, Edited by Jim Searle - 2012-06-05 19:57

Expand comment: Jim Searle added a comment - 2012-06-05 18:50, Edited by Jim Searle - 2012-06-05 19:57

Collapse comment: Gregory Boissinot added a comment - 2012-06-05 20:16

Expand comment: Gregory Boissinot added a comment - 2012-06-05 20:16

Collapse comment: Jim Searle added a comment - 2012-06-05 22:01

Expand comment: Jim Searle added a comment - 2012-06-05 22:01

Collapse comment: Gregory Boissinot added a comment - 2012-06-05 22:28

Expand comment: Gregory Boissinot added a comment - 2012-06-05 22:28

Collapse comment: Jim Searle added a comment - 2012-06-05 22:41

Expand comment: Jim Searle added a comment - 2012-06-05 22:41

Collapse comment: Jim Searle added a comment - 2012-06-08 19:49

Expand comment: Jim Searle added a comment - 2012-06-08 19:49

Collapse comment: Gregory Boissinot added a comment - 2012-06-09 20:41

Expand comment: Gregory Boissinot added a comment - 2012-06-09 20:41

Collapse comment: Jim Searle added a comment - 2012-06-11 17:32

Expand comment: Jim Searle added a comment - 2012-06-11 17:32

Collapse comment: Gregory Boissinot added a comment - 2012-06-11 21:59

Expand comment: Gregory Boissinot added a comment - 2012-06-11 21:59

Collapse comment: Jim Searle added a comment - 2012-06-12 22:25

Expand comment: Jim Searle added a comment - 2012-06-12 22:25

Collapse comment: Jim Searle added a comment - 2012-06-14 21:58

Expand comment: Jim Searle added a comment - 2012-06-14 21:58

People

Dates