[JENKINS-32275] Flexible publish plugin doesn't always instantiate correct child plugin

Type: Bug
Resolution: Not A Defect
Priority: Major
Component/s: flexible-publish-plugin
Labels:
None
Environment:
Windows, Jenkins LTS 1.596.3. Flexible Publish 0.15.2

Similar Issues:
Powered by SuggestiMate

Show

I have discovered that some plugins provide inconsistent behavior with the flexible publish plugin and have isolated a specific use case that may be helpful for debugging and discussion.

(Keep in mind that I have very little knowledge of the internal workings of Jenkins, so take some of my technical assumptions below within that context)

Problem
Using the flexible publish plugin to conditionally set the "Description" of a build using the Description Setter plugin seems to work correctly so long as you provide a single conditional operation to the flexible publish plugin, but when more conditional actions are added subsequent instances of the description setter plugin do not expose all of their available options on the Jenkins UI. In this example case, the first instance exposes an "Advanced..." button allowing the customization of extra parameters not presented in the default list of options, where as subsequent instances do not provide such a button. See the attached sample.png for a visual.

Closer examination of the XML generated by the Jenkins UI reveals that the Flexible Publish plugin is making reference to two different Jenkins plugins / classes for these divergent instances. In this example, the first instance of the Description Setter action is using a "DescriptionSetterPublisher" object, where as all subsequent instances are using a "DescriptionSetterBuilder" object, which may help shed some light on the problem. Below is a snippet from the config.xml for the job configuration presented in the attached screenshot:

<org.jenkins__ci.plugins.flexible__publish.FlexiblePublisher plugin="flexible-publish@0.15.2">
    <publishers>
        <org.jenkins__ci.plugins.flexible__publish.ConditionalPublisher>
            <condition class="org.jenkins_ci.plugins.run_condition.logic.Not" plugin="run-condition@1.0">
                <condition class="org.jenkins_ci.plugins.run_condition.core.BooleanCondition">
                    <token>${IS_CLEAN_BUILD}</token>
                </condition>
            </condition>
            <publisherList>
                <hudson.plugins.descriptionsetter.DescriptionSetterPublisher plugin="description-setter@1.10">
                    <regexp/>
                    <regexpForFailed/>
                    <description>OracleVersion: ${OracleVersion}</description>
                    <descriptionForFailed>OracleVersion: ${OracleVersion} </descriptionForFailed>
                    <setForMatrix>false</setForMatrix>
                </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
            </publisherList>
            <runner class="org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail" plugin="run-condition@1.0"/>
            <executionStrategy class="org.jenkins_ci.plugins.flexible_publish.strategy.FailAtEndExecutionStrategy"/>
        </org.jenkins__ci.plugins.flexible__publish.ConditionalPublisher>
        
        <org.jenkins__ci.plugins.flexible__publish.ConditionalPublisher>
            <condition class="org.jenkins_ci.plugins.run_condition.core.BooleanCondition" plugin="run-condition@1.0">
                <token>${IS_CLEAN_BUILD}</token>
            </condition>
            <publisherList>
                <hudson.plugins.descriptionsetter.DescriptionSetterBuilder plugin="description-setter@1.10">
                    <regexp/>
                    <description>OracleVersion: both</description>
                </hudson.plugins.descriptionsetter.DescriptionSetterBuilder>
            </publisherList>
            <runner class="org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail" plugin="run-condition@1.0"/>
            <executionStrategy class="org.jenkins_ci.plugins.flexible_publish.strategy.FailAtEndExecutionStrategy"/>
        </org.jenkins__ci.plugins.flexible__publish.ConditionalPublisher>
    </publishers>
</org.jenkins__ci.plugins.flexible__publish.FlexiblePublisher>

Based on this information, it appears as though somehow the Flexible Publisher correctly instantiates the first child object - since the options that are presented on the UI do appear to be complete and correct in this case - but subsequent instances are not correctly instantiated for some reason.

I am assuming here that the underlying problem is some inherent implementation problem in the Flexible Publish plugin, however I suppose it could be a problem with the Description Setter plugin as well. I probably should mention that I have found similar behavior with other Jenkins plugins when used under the Flexible Publisher which does seem to reinforce my assumption. Also, given the fact the first instance of a child plugin under Flexible Publish does get instantiated correctly it suggests that the interaction between the two does work correctly in some cases which again seems to reinforce my assumption since if the link between the publisher and the child plugins were broken I would expect them to never work which is clearly not the case.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

build_vs_postbuild_options.png
26 kB
2016-01-20 18:58
duplicate_actions_example.png
37 kB
2016-01-20 18:51
post_build_actions.png
39 kB
2016-01-20 19:02
sample.png
39 kB
2016-01-04 15:43
Screen Shot 2016-01-21 at 07.49.00.png
310 kB
2016-01-20 22:49

is related to

JENKINS-32550 Ambiguous actions in list of available actions

Open

Kevin Phillips added a comment - 2016-01-19 12:25

Yes, we do indeed have the any-build-step plugin installed so it is totally possible that is the culprit here.

As for a screen shot of the set build description plugin, I assume you are looking for an example of what the set build description plugin looks like when not used in conjunction with the conditional publisher. I'll attach screen shots of that here shortly.

As for testing the behavior in a clean Jenkins install, that would require slightly more effort but I'll see what I can do.

Kevin Phillips added a comment - 2016-01-19 12:25 Yes, we do indeed have the any-build-step plugin installed so it is totally possible that is the culprit here. As for a screen shot of the set build description plugin, I assume you are looking for an example of what the set build description plugin looks like when not used in conjunction with the conditional publisher. I'll attach screen shots of that here shortly. As for testing the behavior in a clean Jenkins install, that would require slightly more effort but I'll see what I can do.

ikedam added a comment - 2016-01-19 23:41

> I assume you are looking for an example of what the set build description plugin looks like when not used in conjunction with the conditional publisher.

No.
As you use any-build-step, I think your problem is that "Set build description" of build steps is used even though you select "Set build description" of post build steps.
I want to see

How you are sure you select "Set build description" of post build steps, not "Set build description" of build steps.
Whether actions are listed differently for the first condition and second condition.

I want screenshots for that purpose.

ikedam added a comment - 2016-01-19 23:41 > I assume you are looking for an example of what the set build description plugin looks like when not used in conjunction with the conditional publisher. No. As you use any-build-step, I think your problem is that "Set build description" of build steps is used even though you select "Set build description" of post build steps. I want to see How you are sure you select "Set build description" of post build steps, not "Set build description" of build steps. Whether actions are listed differently for the first condition and second condition. I want screenshots for that purpose.

Kevin Phillips added a comment - 2016-01-20 18:36

Sorry for the wait. I've managed to setup a clean test environment with a clean instance of Jenkins 1.596.3 to try and reproduce this problem. It would appear that the behavior I'm seeing on our production server is slightly different than the behavior that I'm now seeing in this clean test environment, however the underlying problem is still reproducible. In fact, the behavior in the clean environment is inconsistent as well (to be explained later).

Here is what I believe to be a minimum set of steps that are needed to reliably reproduce the problem:

launch a clean environment for Jenkins 1.596.3
Install the description setter plugin (v1.10)
Install the Flexible Publish plugin (v0.15.2)
Install the Any Build Step plugin (v0.1)
Under the Manage Jenkins -> Configure System panel locate the newly added "Flexible Publish" section
Select the "Any build step" from the dropdown list next to the "Allowed build steps" option
Save your changes
Create a new job / project of type "Freestyle project"
Under the Post-build actions, add a new "Set build description action"
Next add a new Flexible Publish action also under the post-build section
Add a conditional action of type "Set build description" to the Flexible Publish
Add a second conditional action of type "Set build description" in the same area as the previous one
Add an additional conditional action block to the Flexible Publish action by clicking the "Add conditional action" button beneath the first section
Add one more conditional action of type "Set build description" to this newly added conditional section

Now, if you review the configuration options presented with each of the "Set build description" instances you will see that some of them will have the "Advanced" options while others will not.

As for the inconsistent behavior I mentioned earlier, I found that sometimes when I created new jobs in my clean environment that just creating a single post-build action of type Flexible Publish with a single sub-action of type "set build description" was sufficient to reproduce the problem, but then repeating the exact same steps to create a new job in the same way would not reproduce the problem. I can't explain why sometimes a single instance of either of these actions was sufficient to reproduce the problem while other times multiple instances were required. If I can provide any more detailed information to you to help isolate that bit just let me know.

NOTE: Screen shots relating to my findings will be forthcoming.

Kevin Phillips added a comment - 2016-01-20 18:36 Sorry for the wait. I've managed to setup a clean test environment with a clean instance of Jenkins 1.596.3 to try and reproduce this problem. It would appear that the behavior I'm seeing on our production server is slightly different than the behavior that I'm now seeing in this clean test environment, however the underlying problem is still reproducible. In fact, the behavior in the clean environment is inconsistent as well (to be explained later). Here is what I believe to be a minimum set of steps that are needed to reliably reproduce the problem: launch a clean environment for Jenkins 1.596.3 Install the description setter plugin (v1.10) Install the Flexible Publish plugin (v0.15.2) Install the Any Build Step plugin (v0.1) Under the Manage Jenkins -> Configure System panel locate the newly added "Flexible Publish" section Select the "Any build step" from the dropdown list next to the "Allowed build steps" option Save your changes Create a new job / project of type "Freestyle project" Under the Post-build actions, add a new "Set build description action" Next add a new Flexible Publish action also under the post-build section Add a conditional action of type "Set build description" to the Flexible Publish Add a second conditional action of type "Set build description" in the same area as the previous one Add an additional conditional action block to the Flexible Publish action by clicking the "Add conditional action" button beneath the first section Add one more conditional action of type "Set build description" to this newly added conditional section Now, if you review the configuration options presented with each of the "Set build description" instances you will see that some of them will have the "Advanced" options while others will not. As for the inconsistent behavior I mentioned earlier, I found that sometimes when I created new jobs in my clean environment that just creating a single post-build action of type Flexible Publish with a single sub-action of type "set build description" was sufficient to reproduce the problem, but then repeating the exact same steps to create a new job in the same way would not reproduce the problem. I can't explain why sometimes a single instance of either of these actions was sufficient to reproduce the problem while other times multiple instances were required. If I can provide any more detailed information to you to help isolate that bit just let me know. NOTE: Screen shots relating to my findings will be forthcoming.

Kevin Phillips added a comment - 2016-01-20 18:50

And now the plot thickens. As another test case I decided to customize the "Conditional buildstep" options which are installed as part of the build any plugin and discovered something interesting - there are now two instances of the "set build description" action in the drop down list for the flexible publisher. Here are the exact steps to reproduce:

launch clean instance of Jenkins with the 3 plugins mentioned in my previous comment
Go to the Manage Jenkins -> Configure System panel and find the "Conditional buildstep" section
Under the "Allowed build steps" option select "Any build step" from the drop down menu
Also under the "Configure System" panel locate the "Flexible publish" section
Under the "Allowed build steps" option select "Any build step" from the drop down menu here as well
Click save
Create a new job / project of type "Freestyle project"
Under Post-build actions, add a new "Flexible publish" action
Click the "Add" button under the Flexible Publish action to expose the list of supported sub-actions. If you review the list carefully you'll see there are two entries for "set build description" (see attached screenshot)

Based on my ad-hoc tests, one of these entries inserts a "build" action while the other inserts a "post-build" action, the latter of which exposes "Advanced" options for customizing the behavior based on the status of the build operation. Unfortunately there is no way to distinguish which is which based on the name alone.

I have confirmed that we do have similar behavior on our production servers as well, which may partially explain away the behavior we are seeing. However I believe the underlying problem gets exacerbated when you only have one of the two "Allowed build steps" set to "Any build step". In those cases it appears as though there is only one "set build description" element presented in the list of actions, and sometimes it instantiates a "build" action while other times it instantiates a "post-build" action. Again, the behavior there seems somewhat random based on my preliminary ad-hoc tests.

Kevin Phillips added a comment - 2016-01-20 18:50 And now the plot thickens. As another test case I decided to customize the "Conditional buildstep" options which are installed as part of the build any plugin and discovered something interesting - there are now two instances of the "set build description" action in the drop down list for the flexible publisher. Here are the exact steps to reproduce: launch clean instance of Jenkins with the 3 plugins mentioned in my previous comment Go to the Manage Jenkins -> Configure System panel and find the "Conditional buildstep" section Under the "Allowed build steps" option select "Any build step" from the drop down menu Also under the "Configure System" panel locate the "Flexible publish" section Under the "Allowed build steps" option select "Any build step" from the drop down menu here as well Click save Create a new job / project of type "Freestyle project" Under Post-build actions, add a new "Flexible publish" action Click the "Add" button under the Flexible Publish action to expose the list of supported sub-actions. If you review the list carefully you'll see there are two entries for "set build description" (see attached screenshot) Based on my ad-hoc tests, one of these entries inserts a "build" action while the other inserts a "post-build" action, the latter of which exposes "Advanced" options for customizing the behavior based on the status of the build operation. Unfortunately there is no way to distinguish which is which based on the name alone. I have confirmed that we do have similar behavior on our production servers as well, which may partially explain away the behavior we are seeing. However I believe the underlying problem gets exacerbated when you only have one of the two "Allowed build steps" set to "Any build step". In those cases it appears as though there is only one "set build description" element presented in the list of actions, and sometimes it instantiates a "build" action while other times it instantiates a "post-build" action. Again, the behavior there seems somewhat random based on my preliminary ad-hoc tests.

Kevin Phillips added a comment - 2016-01-20 18:57

I thought I may as well add some more info based on my latest ad-hoc tests that may help further isolate this problem:

the bug doesn't seem to be reproducible when only using the Flexible Publish plugin alone, so it seems likely the Any Build Step plugin may be to blame
it would appear that when adding a build step of type "set build description" there are no "Advanced" options for customizing the behavior of the action based on the build status (which makes sense to me)
it would appear that when adding a post-build step of type "set build description" there is an "Advanced" options area to customize the behavior based on the status of the build operation. This fact, combined with the previous one, should help distinguish which format of the plugin is being used at any given time.
it would appear that the "build" action of type "set build description" allows one to instantiate multiple copies of itself as independent build operations, however "post-build" actions of type "set build description" limit themselves to just a single instance per project.
- There is one exception to this rule: when adding a conditional build step under a Flexible Publish action of type "set build description", you are able to add multiple instances of the action as desired. Perhaps this slight change in behavior may explain why sometimes we get one type of action over another (ie: after the first post-build action gets instantiated maybe subsequent ones are forced to be build actions instead ... although this behavior was inconsistent at best in my trials)

Kevin Phillips added a comment - 2016-01-20 18:57 I thought I may as well add some more info based on my latest ad-hoc tests that may help further isolate this problem: the bug doesn't seem to be reproducible when only using the Flexible Publish plugin alone, so it seems likely the Any Build Step plugin may be to blame it would appear that when adding a build step of type "set build description" there are no "Advanced" options for customizing the behavior of the action based on the build status (which makes sense to me) it would appear that when adding a post-build step of type "set build description" there is an "Advanced" options area to customize the behavior based on the status of the build operation. This fact, combined with the previous one, should help distinguish which format of the plugin is being used at any given time. it would appear that the "build" action of type "set build description" allows one to instantiate multiple copies of itself as independent build operations, however "post-build" actions of type "set build description" limit themselves to just a single instance per project. There is one exception to this rule: when adding a conditional build step under a Flexible Publish action of type "set build description", you are able to add multiple instances of the action as desired. Perhaps this slight change in behavior may explain why sometimes we get one type of action over another (ie: after the first post-build action gets instantiated maybe subsequent ones are forced to be build actions instead ... although this behavior was inconsistent at best in my trials)

Kevin Phillips added a comment - 2016-01-20 19:04

FYI: For kicks I took a quick look through our production system to see if there were duplication references to the Artifact Deployer actions as well, and there are! This would seem to suggest that at least part of the problem we have been experiencing is a result of duplicate entries with the exact same names showing up in the list of available actions, but only under certain specific conditions / circumstances.

Kevin Phillips added a comment - 2016-01-20 19:04 FYI: For kicks I took a quick look through our production system to see if there were duplication references to the Artifact Deployer actions as well, and there are! This would seem to suggest that at least part of the problem we have been experiencing is a result of duplicate entries with the exact same names showing up in the list of available actions, but only under certain specific conditions / circumstances.

ikedam added a comment - 2016-01-20 22:51

There are two "set build description".
Different configuration sections for those two "set build description". (of build step and of post build step)

Those above behaviors are expected and consistent ones.

The problem you point is

You can't add more than two "set build description" of post build step.

Right?

I can't reproduce the problem in my environment.
I attached a screen shot showing that I could add more than two "set build description" of post build step.

I can't get what you want to explain with post_build_actions.png.
Any-build-step only affects the behavior of flexible-publish (and conditional-buildstep) and never affects "Add post-build action" of projects.
Attach a screenshot of the list of "Add" of flexible-publish actions for the second "set build description".

ikedam added a comment - 2016-01-20 22:51 There are two "set build description". Different configuration sections for those two "set build description". (of build step and of post build step) Those above behaviors are expected and consistent ones. The problem you point is You can't add more than two "set build description" of post build step. Right? I can't reproduce the problem in my environment. I attached a screen shot showing that I could add more than two "set build description" of post build step. I can't get what you want to explain with post_build_actions.png. Any-build-step only affects the behavior of flexible-publish (and conditional-buildstep) and never affects "Add post-build action" of projects. Attach a screenshot of the list of "Add" of flexible-publish actions for the second "set build description".

Kevin Phillips added a comment - 2016-01-21 12:17 - edited

The problem you point is

You can't add more than two "set build description" of post build step.

Right?

Actually no. The original problem was that sometimes adding one of these actions as a post-build step resulted in the "Advanced" button being enabled while other times it was not.

I can't get what you want to explain with post_build_actions.png.
Any-build-step only affects the behavior of flexible-publish (and conditional-buildstep) and never affects "Add post-build action" of projects.

I think therein lies some of my confusion. I thought the any-build-step plugin worked on arbitrary build / post-build steps. I hadn't realized that it only applies to flexible-publish and conditional-buildstep. I will limit subsequent testing to just those to areas.

Also, given this fact I think the second point for confusion that I hadn't realized until yesterday was that in this case plugins such as set-build-description show up twice in the list of available actions - one representing a build step and one representing a post-build step - each of which present different options in the UI. I suspect that part of my problem in reporting this bug is that during testing I would occasionally click the former while other times I'd click the latter without realizing it.

At least one improvement that could be made to the UI to avoid confusion would be to append / prepend some kind of indicator to distinguish duplicate action names like this. Maybe the any-build-step plugin could be enhanced such that if it detects two actions with the same name it could put a "(build)" or "(post-build)" label in front of the action name to avoid the ambiguity.

That being said, when I was ad-hoc testing yesterday in my clean Jenkins environment I could have sworn that I managed to reproduce the behavior of the actions being created as build and post-build versions at random. I will do a bit more ad-hoc testing this morning to try and find a reproducible use case. If I can not find one I think it would be safe to say that this defect was actually attributed to human error caused by ambiguous duplication of action names, in which case enhancing the any-build-step plugin like what I suggested above could be a reasonable solution to this problem.

Kevin Phillips added a comment - 2016-01-21 12:17 - edited The problem you point is You can't add more than two "set build description" of post build step. Right? Actually no. The original problem was that sometimes adding one of these actions as a post-build step resulted in the "Advanced" button being enabled while other times it was not. I can't get what you want to explain with post_build_actions.png. Any-build-step only affects the behavior of flexible-publish (and conditional-buildstep) and never affects "Add post-build action" of projects. I think therein lies some of my confusion. I thought the any-build-step plugin worked on arbitrary build / post-build steps. I hadn't realized that it only applies to flexible-publish and conditional-buildstep. I will limit subsequent testing to just those to areas. Also, given this fact I think the second point for confusion that I hadn't realized until yesterday was that in this case plugins such as set-build-description show up twice in the list of available actions - one representing a build step and one representing a post-build step - each of which present different options in the UI. I suspect that part of my problem in reporting this bug is that during testing I would occasionally click the former while other times I'd click the latter without realizing it. At least one improvement that could be made to the UI to avoid confusion would be to append / prepend some kind of indicator to distinguish duplicate action names like this. Maybe the any-build-step plugin could be enhanced such that if it detects two actions with the same name it could put a "(build)" or "(post-build)" label in front of the action name to avoid the ambiguity. That being said, when I was ad-hoc testing yesterday in my clean Jenkins environment I could have sworn that I managed to reproduce the behavior of the actions being created as build and post-build versions at random. I will do a bit more ad-hoc testing this morning to try and find a reproducible use case. If I can not find one I think it would be safe to say that this defect was actually attributed to human error caused by ambiguous duplication of action names, in which case enhancing the any-build-step plugin like what I suggested above could be a reasonable solution to this problem.

Kevin Phillips added a comment - 2016-01-21 13:37

All attempts to reproduce the "random" behavior I had originally experienced when reporting this defect have failed. I suspect the problem the entire time is that sometimes our users would be selecting the first action name from the list of actions while other times they were selecting the second instance. Since the action names are identical there was no easy / obvious way to tell them apart. In our case we have a large number of plugins installed so the duplicate action names were separated from one another by dozens of other actions so the duplication was not easily noticed making the behavior "seem" random.

I would be content having this defect resolved and I have created a separate improvement request to find some way to disambiguate duplicate entries in the actions list when using the any-buildstep-plugin. See JENKINS-32550 for details.

Thanks ikedam for your prompt replies and assistance with debugging this problem with me. It is greatly appreciated.

Kevin Phillips added a comment - 2016-01-21 13:37 All attempts to reproduce the "random" behavior I had originally experienced when reporting this defect have failed. I suspect the problem the entire time is that sometimes our users would be selecting the first action name from the list of actions while other times they were selecting the second instance. Since the action names are identical there was no easy / obvious way to tell them apart. In our case we have a large number of plugins installed so the duplicate action names were separated from one another by dozens of other actions so the duplication was not easily noticed making the behavior "seem" random. I would be content having this defect resolved and I have created a separate improvement request to find some way to disambiguate duplicate entries in the actions list when using the any-buildstep-plugin. See JENKINS-32550 for details. Thanks ikedam for your prompt replies and assistance with debugging this problem with me. It is greatly appreciated.

ikedam added a comment - 2016-01-21 13:38

I found that the issue was caused for you are confused with the same label "Set build description for a build step and a post build step.

I close this issue with "Not a defect".

You can create a new ticket for any-build-step plugin and request an improvement.
This ticket is filled with confused descriptions and I don't think it's good idea to reuse this ticket.

ikedam added a comment - 2016-01-21 13:38 I found that the issue was caused for you are confused with the same label "Set build description for a build step and a post build step. I close this issue with "Not a defect". You can create a new ticket for any-build-step plugin and request an improvement. This ticket is filled with confused descriptions and I don't think it's good idea to reuse this ticket.

Assignee:: ikedam

Reporter:: Kevin Phillips

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2016-01-04 15:53

Updated:: 2016-01-21 13:38

Resolved:: 2016-01-21 13:38

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Kevin Phillips added a comment - 2016-01-19 12:25

Expand comment: Kevin Phillips added a comment - 2016-01-19 12:25

Collapse comment: ikedam added a comment - 2016-01-19 23:41

Expand comment: ikedam added a comment - 2016-01-19 23:41

Collapse comment: Kevin Phillips added a comment - 2016-01-20 18:36

Expand comment: Kevin Phillips added a comment - 2016-01-20 18:36

Collapse comment: Kevin Phillips added a comment - 2016-01-20 18:50

Expand comment: Kevin Phillips added a comment - 2016-01-20 18:50

Collapse comment: Kevin Phillips added a comment - 2016-01-20 18:57

Expand comment: Kevin Phillips added a comment - 2016-01-20 18:57

Collapse comment: Kevin Phillips added a comment - 2016-01-20 19:04

Expand comment: Kevin Phillips added a comment - 2016-01-20 19:04

Collapse comment: ikedam added a comment - 2016-01-20 22:51

Expand comment: ikedam added a comment - 2016-01-20 22:51

Collapse comment: Kevin Phillips added a comment - 2016-01-21 12:17, Edited by Kevin Phillips - 2016-01-21 12:18

Expand comment: Kevin Phillips added a comment - 2016-01-21 12:17, Edited by Kevin Phillips - 2016-01-21 12:18

Collapse comment: Kevin Phillips added a comment - 2016-01-21 13:37

Expand comment: Kevin Phillips added a comment - 2016-01-21 13:37

Collapse comment: ikedam added a comment - 2016-01-21 13:38

Expand comment: ikedam added a comment - 2016-01-21 13:38

People

Dates