[JENKINS-29078] hudson.triggers.Trigger$Cron doRun throwing a NullPointerException

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: core
Labels:
None
Environment:
1.617
RHEL

Similar Issues:
Powered by SuggestiMate

Show

Hi, it seems that Jenkins is throwing a NullPointerException on the below in the background logs but i can't seem to tell why.

Jun 25, 2015 5:38:00 PM hudson.triggers.Trigger$Cron doRun
WARNING: Cron thread throw an exception
java.lang.NullPointerException
        at hudson.model.Descriptor.toMap(Descriptor.java:880)
        at hudson.util.DescribableList.toMap(DescribableList.java:151)
        at hudson.model.AbstractProject.getTriggers(AbstractProject.java:1615)
        at hudson.triggers.Trigger.checkTriggers(Trigger.java:259)
        at hudson.triggers.Trigger$Cron.doRun(Trigger.java:215)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Untitled spreadsheet - Sheet1 (1).pdf
8 kB
2016-06-25 00:15
Untitled spreadsheet - Sheet2.pdf
3 kB
2016-06-25 00:16

is duplicated by

JENKINS-31624 hudson.triggers.Trigger$Cron doRun Cron thread throw an exception

Resolved

is related to

JENKINS-36232 NullPointerException during SCM polling

Resolved

Chin Boon Oh created issue - 2015-06-25 09:41

Phil Dickson added a comment - 2015-07-02 12:05

None of our cron scheduled tasks are running!
This seems pretty serious.

Phil Dickson added a comment - 2015-07-02 12:05 None of our cron scheduled tasks are running! This seems pretty serious.

Phil Dickson added a comment - 2015-07-02 12:37

Downgrading to 1.615 fixes things for us.

Phil Dickson added a comment - 2015-07-02 12:37 Downgrading to 1.615 fixes things for us.

Phil Dickson added a comment - 2015-07-02 12:44

Looks like ~~JENKINS-27549~~ is the probable cause

Phil Dickson added a comment - 2015-07-02 12:44 Looks like JENKINS-27549 is the probable cause

Phil Dickson made changes - 2015-07-03 14:10

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Chin Boon Oh added a comment - 2015-07-20 03:01

Hi danielbeck, i tried the command but it is not displaying anything.

Chin Boon Oh added a comment - 2015-07-20 03:01 Hi danielbeck , i tried the command but it is not displaying anything.

Daniel Beck added a comment - 2015-07-20 11:11

Sorry about that. Please try:

Jenkins.instance.getAllItems(AbstractProject).each { p ->
  try {
    p.getTriggers()
  } catch (NullPointerException npe) {
    println p.fullName
  }
}
return

Daniel Beck added a comment - 2015-07-20 11:11 Sorry about that. Please try: Jenkins.instance.getAllItems(AbstractProject).each { p -> try { p.getTriggers() } catch (NullPointerException npe) { println p.fullName } } return

gene lzr added a comment - 2015-07-23 01:38 - edited

Actually, downgrading to 1.615 didn't help. I still observe the problem. Ran the script and it printed a name of a Jenkins job. I tried saving it today and got an error page (with evil Jenkins).

What do I do now?

config.xml:

<?xml version='1.0' encoding='UTF-8'?>
<project>
    <actions/>
    <description></description>
    <keepDependencies>false</keepDependencies>
    <properties/>
    <scm class="hudson.scm.NullSCM"/>
    <canRoam>true</canRoam>
    <disabled>false</disabled>
    <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
    <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
    <triggers>
        <null/>
    </triggers>
    <concurrentBuild>false</concurrentBuild>
    <builders>
        <hudson.tasks.Shell>
            <command>sudo -u nobody script.sh path
            </command>
        </hudson.tasks.Shell>
    </builders>
    <publishers>
        <hudson.tasks.Mailer plugin="mailer@1.15">
            <recipients>ops-alert@example.com</recipients>
            <dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>
            <sendToIndividuals>false</sendToIndividuals>
        </hudson.tasks.Mailer>
    </publishers>
    <buildWrappers/>
</project>

When I tried to re-save that project I got this exception:

javax.servlet.ServletException: java.lang.NullPointerException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
	at org.eclipse.jetty.ajp.Ajp13Connection.access$2900(Ajp13Connection.java:45)
	at org.eclipse.jetty.ajp.Ajp13Connection$RequestHandler.headerComplete(Ajp13Connection.java:237)
	at org.eclipse.jetty.ajp.Ajp13Parser.parseNext(Ajp13Parser.java:505)
	at org.eclipse.jetty.ajp.Ajp13Parser.parseAvailable(Ajp13Parser.java:158)
	at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at hudson.model.AbstractProject.submit(AbstractProject.java:1873)
	at hudson.model.Project.submit(Project.java:230)
	at hudson.model.Job.doConfigSubmit(Job.java:1206)
	at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:788)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	... 67 more

gene lzr added a comment - 2015-07-23 01:38 - edited Actually, downgrading to 1.615 didn't help. I still observe the problem. Ran the script and it printed a name of a Jenkins job. I tried saving it today and got an error page (with evil Jenkins). What do I do now? config.xml: <?xml version= '1.0' encoding= 'UTF-8' ?> <project> <actions/> <description> </description> <keepDependencies> false </keepDependencies> <properties/> <scm class= "hudson.scm.NullSCM" /> <canRoam> true </canRoam> <disabled> false </disabled> <blockBuildWhenDownstreamBuilding> false </blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding> false </blockBuildWhenUpstreamBuilding> <triggers> <null/> </triggers> <concurrentBuild> false </concurrentBuild> <builders> <hudson.tasks.Shell> <command> sudo -u nobody script.sh path </command> </hudson.tasks.Shell> </builders> <publishers> <hudson.tasks.Mailer plugin= "mailer@1.15" > <recipients> ops-alert@example.com </recipients> <dontNotifyEveryUnstableBuild> false </dontNotifyEveryUnstableBuild> <sendToIndividuals> false </sendToIndividuals> </hudson.tasks.Mailer> </publishers> <buildWrappers/> </project> When I tried to re-save that project I got this exception: javax.servlet.ServletException: java.lang.NullPointerException at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.ajp.Ajp13Connection.access$2900(Ajp13Connection.java:45) at org.eclipse.jetty.ajp.Ajp13Connection$RequestHandler.headerComplete(Ajp13Connection.java:237) at org.eclipse.jetty.ajp.Ajp13Parser.parseNext(Ajp13Parser.java:505) at org.eclipse.jetty.ajp.Ajp13Parser.parseAvailable(Ajp13Parser.java:158) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.NullPointerException at hudson.model.AbstractProject.submit(AbstractProject.java:1873) at hudson.model.Project.submit(Project.java:230) at hudson.model.Job.doConfigSubmit(Job.java:1206) at hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) ... 67 more

Daniel Beck added a comment - 2015-07-23 14:04

Hm. Weird. Difficult to say how a null trigger got in there. Do you have the Job Config History Plugin that could show you information on how the job config was changed to add that <null/> element?

You should be able to fix this by changing the XML to <triggers/> instead of what that element is currently while Jenkins is stopped.

Daniel Beck added a comment - 2015-07-23 14:04 Hm. Weird. Difficult to say how a null trigger got in there. Do you have the Job Config History Plugin that could show you information on how the job config was changed to add that <null/> element? You should be able to fix this by changing the XML to <triggers/> instead of what that element is currently while Jenkins is stopped.

gene lzr added a comment - 2015-07-23 21:21

I do have Job Config History installed but I already deleted the project I'm not sure if config history is kept after deletion.

Let me ask this though. Why does a single project with incorrect config.xml prevent other projects from being scheduled. This seems very wrong to me.

gene lzr added a comment - 2015-07-23 21:21 I do have Job Config History installed but I already deleted the project I'm not sure if config history is kept after deletion. Let me ask this though. Why does a single project with incorrect config.xml prevent other projects from being scheduled. This seems very wrong to me.

Assignee:: Unassigned

Reporter:: Chin Boon Oh

Votes:: 2 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2015-06-25 09:41

Updated:: 2016-06-29 16:42

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Phil Dickson added a comment - 2015-07-02 12:05

Expand comment: Phil Dickson added a comment - 2015-07-02 12:05

Collapse comment: Phil Dickson added a comment - 2015-07-02 12:37

Expand comment: Phil Dickson added a comment - 2015-07-02 12:37

Collapse comment: Phil Dickson added a comment - 2015-07-02 12:44

Expand comment: Phil Dickson added a comment - 2015-07-02 12:44

Collapse comment: Chin Boon Oh added a comment - 2015-07-20 03:01

Expand comment: Chin Boon Oh added a comment - 2015-07-20 03:01

Collapse comment: Daniel Beck added a comment - 2015-07-20 11:11

Expand comment: Daniel Beck added a comment - 2015-07-20 11:11

Collapse comment: gene lzr added a comment - 2015-07-23 01:38, Edited by gene lzr - 2015-07-23 02:20

Expand comment: gene lzr added a comment - 2015-07-23 01:38, Edited by gene lzr - 2015-07-23 02:20

Collapse comment: Daniel Beck added a comment - 2015-07-23 14:04

Expand comment: Daniel Beck added a comment - 2015-07-23 14:04

Collapse comment: gene lzr added a comment - 2015-07-23 21:21

Expand comment: gene lzr added a comment - 2015-07-23 21:21

People

Dates