[JENKINS-57154] Regression in github-oauth-plugin 0.32 breaks /configureSecurity page

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: github-oauth-plugin
Labels:
None
Environment:
OS: Ubuntu 18.04.2 - 64 bit
Java: openjdk version "1.8.0_191"
github-oauth-plugin: 0.32
Jenkins: 2.164.2

Similar Issues:
Powered by SuggestiMate

Show
Released As:
github-oauth-0.33

After upgrading to github-oauth-plugin 0.32 I started to see this error in /configureSecurity when it tries to retrieve the name of a github user:

HTTP ERROR 403
Problem accessing /descriptorByName/hudson.security.ProjectMatrixAuthorizationStrategy/checkName.
Reason:    Forbidden

The first user has its name retrieved successfully but all others have the error mentioned above.

See the attachment users.png.

The workaround for now is revert to 0.31.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

users.png
29 kB
2019-04-23 12:38

is duplicated by

JENKINS-57595 with GitHub OAuth plugin 0.31->0.32 Matrix-based security errors

Closed

JENKINS-56997 this.me NullPointerException in GithubAuthenticationToken.java oauth

Closed

Christian Höltje added a comment - 2019-07-18 15:51

Doing some googling...

The has_recent_activity=1 cookie seems to be coming from GitHub. I suspect that the cookie parser being used by Jenkins is broken.

Christian Höltje added a comment - 2019-07-18 15:51 Doing some googling... The has_recent_activity=1 cookie seems to be coming from GitHub. I suspect that the cookie parser being used by Jenkins is broken.

Sam Gleske added a comment - 2019-08-04 00:38

I've tried a couple of ways to reproduce this locally and I'm not able to reproduce it locally. I configured plugin 0.31 and upgraded to 0.32 with no problems. I'll try another fresh install and use 0.29 since I see others reporting they're upgrading from that version.

Sam Gleske added a comment - 2019-08-04 00:38 I've tried a couple of ways to reproduce this locally and I'm not able to reproduce it locally. I configured plugin 0.31 and upgraded to 0.32 with no problems. I'll try another fresh install and use 0.29 since I see others reporting they're upgrading from that version.

Sam Gleske added a comment - 2019-08-04 00:58 - edited

Okay I was able to replicate the issue. Replication steps:

Have two GitHub users. githubadmin and githubuser for example where githubadmin is a Jenkins admin and github user is a non-admin user in Jenkins.
Have both users log in and authorize with GitHub OAuth.
Configure project-based matrix authorization and add Overall:Read to githubuser and Overall:Administer to githubadmin.
IMPORTANT: On githubuser log into GitHub settings and de-authorize the OAuth app. This means Jenkins will have a token for the user but it won't be valid because the user de-authorized the app.
Using githubadmin I visited the configureSecurity page in Jenkins and got the following stack trace.

githubuser (name changed intentionally to be generic)

java.lang.NullPointerException
	at org.jenkinsci.plugins.GithubAuthenticationToken.<init>(GithubAuthenticationToken.java:205)
	at org.jenkinsci.plugins.GithubSecurityRealm.loadUserByUsername(GithubSecurityRealm.java:700)
	at org.jenkinsci.plugins.matrixauth.AuthorizationContainerDescriptor.doCheckName_(AuthorizationContainerDescriptor.java:140)
	at hudson.security.GlobalMatrixAuthorizationStrategy$DescriptorImpl.doCheckName(GlobalMatrixAuthorizationStrategy.java:222)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:505)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
	at java.lang.Thread.run(Thread.java:748)

The root cause lies within impersonate. When users are validated it doesn't use the admin token. Instead, it attempts to use the token for each individual user in the project-based matrix authorization form.

I'll need to investigate the fix but have identified the root cause.

Sam Gleske added a comment - 2019-08-04 00:58 - edited Okay I was able to replicate the issue. Replication steps: Have two GitHub users. githubadmin and githubuser for example where githubadmin is a Jenkins admin and github user is a non-admin user in Jenkins. Have both users log in and authorize with GitHub OAuth. Configure project-based matrix authorization and add Overall:Read to githubuser and Overall:Administer to githubadmin. IMPORTANT: On githubuser log into GitHub settings and de-authorize the OAuth app. This means Jenkins will have a token for the user but it won't be valid because the user de-authorized the app. Using githubadmin I visited the configureSecurity page in Jenkins and got the following stack trace. githubuser (name changed intentionally to be generic) java.lang.NullPointerException at org.jenkinsci.plugins.GithubAuthenticationToken.<init>(GithubAuthenticationToken.java:205) at org.jenkinsci.plugins.GithubSecurityRealm.loadUserByUsername(GithubSecurityRealm.java:700) at org.jenkinsci.plugins.matrixauth.AuthorizationContainerDescriptor.doCheckName_(AuthorizationContainerDescriptor.java:140) at hudson.security.GlobalMatrixAuthorizationStrategy$DescriptorImpl.doCheckName(GlobalMatrixAuthorizationStrategy.java:222) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748) The root cause lies within impersonate. When users are validated it doesn't use the admin token. Instead, it attempts to use the token for each individual user in the project-based matrix authorization form. I'll need to investigate the fix but have identified the root cause.

Sam Gleske added a comment - 2019-08-04 01:08

https://github.com/jenkinsci/github-oauth-plugin/blob/github-oauth-0.32/src/main/java/org/jenkinsci/plugins/GithubSecurityRealm.java#L694-L700

is the problematic section of code

Sam Gleske added a comment - 2019-08-04 01:08 https://github.com/jenkinsci/github-oauth-plugin/blob/github-oauth-0.32/src/main/java/org/jenkinsci/plugins/GithubSecurityRealm.java#L694-L700 is the problematic section of code

Sam Gleske added a comment - 2019-08-04 01:16

This seems to have been caused by https://github.com/jenkinsci/github-oauth-plugin/pull/109

However, PR 109 is pretty important for how impersonation works. Need to figure out a happy medium.

Sam Gleske added a comment - 2019-08-04 01:16 This seems to have been caused by https://github.com/jenkinsci/github-oauth-plugin/pull/109 However, PR 109 is pretty important for how impersonation works. Need to figure out a happy medium.

Sam Gleske added a comment - 2019-08-04 02:23

Here's the fix https://github.com/jenkinsci/github-oauth-plugin/pull/115

Sam Gleske added a comment - 2019-08-04 02:23 Here's the fix https://github.com/jenkinsci/github-oauth-plugin/pull/115

Sam Gleske added a comment - 2019-08-06 04:03

https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/github-oauth/0.33/github-oauth-0.33.hpi has been release and I verified the fix by upgrading locally to the new version. It should be available in the update center in roughly 8 hours or so.

Sam Gleske added a comment - 2019-08-06 04:03 https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/github-oauth/0.33/github-oauth-0.33.hpi has been release and I verified the fix by upgrading locally to the new version. It should be available in the update center in roughly 8 hours or so.

Jon Cormier added a comment - 2019-08-06 17:28

I installed 0.33 and the problem no longer appears for me. Thanks sag47

Jon Cormier added a comment - 2019-08-06 17:28 I installed 0.33 and the problem no longer appears for me. Thanks sag47

Steve Ims added a comment - 2019-08-06 21:58

0.33 working for me too. Thanks sag47 !

Steve Ims added a comment - 2019-08-06 21:58 0.33 working for me too. Thanks sag47 !

Sam Gleske added a comment - 2019-08-10 14:19

joncormier steveims no problem; thanks for reporting back your own testing results since it helps me validate the solution was a fix.

Sam Gleske added a comment - 2019-08-10 14:19 joncormier steveims no problem; thanks for reporting back your own testing results since it helps me validate the solution was a fix.

Assignee:: Sam Gleske

Reporter:: Francisco Guimaraes

Votes:: 27 Vote for this issue

Watchers:: 36 Start watching this issue

Created:: 2019-04-23 12:36

Updated:: 2019-08-10 14:19

Resolved:: 2019-08-06 04:03

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Christian Höltje added a comment - 2019-07-18 15:51

Expand comment: Christian Höltje added a comment - 2019-07-18 15:51

Collapse comment: Sam Gleske added a comment - 2019-08-04 00:38

Expand comment: Sam Gleske added a comment - 2019-08-04 00:38

Collapse comment: Sam Gleske added a comment - 2019-08-04 00:58, Edited by Sam Gleske - 2019-08-04 01:00

Expand comment: Sam Gleske added a comment - 2019-08-04 00:58, Edited by Sam Gleske - 2019-08-04 01:00

Collapse comment: Sam Gleske added a comment - 2019-08-04 01:08

Expand comment: Sam Gleske added a comment - 2019-08-04 01:08

Collapse comment: Sam Gleske added a comment - 2019-08-04 01:16

Expand comment: Sam Gleske added a comment - 2019-08-04 01:16

Collapse comment: Sam Gleske added a comment - 2019-08-04 02:23

Expand comment: Sam Gleske added a comment - 2019-08-04 02:23

Collapse comment: Sam Gleske added a comment - 2019-08-06 04:03

Expand comment: Sam Gleske added a comment - 2019-08-06 04:03

Collapse comment: Jon Cormier added a comment - 2019-08-06 17:28

Expand comment: Jon Cormier added a comment - 2019-08-06 17:28

Collapse comment: Steve Ims added a comment - 2019-08-06 21:58

Expand comment: Steve Ims added a comment - 2019-08-06 21:58

Collapse comment: Sam Gleske added a comment - 2019-08-10 14:19

Expand comment: Sam Gleske added a comment - 2019-08-10 14:19

People

Dates