[JENKINS-38592] pipelines data for json is getting a bit too crazy (for dashboard)

Type: Bug
Resolution: Duplicate
Priority: Major
Component/s: blueocean-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show
Epic Link:
Performance
Sprint:
pacific, atlantic

In many cases the piplines json for the dashboard (search results) is blowing out to 14kb for 26 "rows" of data. This isn't a huge json transfer, but when you look closer, the impact of plugins adding to actions appears to be huge, and may be the cause of some problems.

As part of looking at a support ticket tscherler uncovered that for 26 pipelines, there is really 20 lines of attributes that are used by the dashboard == 520 lines (roughly).
However there is 10k lines in the response - this is a bit out of control.

It seems that every plugin installed adds actions for every little thing, which the dashboard doesn't need.

For example

        "_class": "io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl",
        "_links": {
          "self": {
            "_class": "io.jenkins.blueocean.rest.hal.Link",
            "href": "/blue/rest/organizations/jenkins/... (hidden)"
          }
        },
        "_class": "com.cloudbees.plugins.credentials    .ViewCredentialsAction",
        "stores": {},
        "urlName": "credentials"
      }
    ],

is added to every pipeline (along with 100's of others). It really depends on the plugins.

Also

         {
            "_class": "hudson.model.TextParameterDefinition",
            "defaultParameterValue": {
              "_class": "hudson.model.StringParameterValue",
              "name": "jiraComment",
              "value": ""
            },
            "description": "Additional info for theissue's comment",
            "name": "jiraComment",
            "type": "TextParameterDefinition"
          },

will never be of use to the dashboard (well probably).

Some questions:

Is there a better query to use to make the response smaller/more compact? (even just eyeballing the reponse could be helped by not returning the metadata optionally). I expect the calculation of this is quite expensive.

Could this be a source of the slowdown when people have enough/certain plugins installed?

duplicates

JENKINS-38979 Investigate tree api for plugins + dashboard

Open

Michael Neale created issue - 2016-09-29 01:36

Michael Neale added a comment - 2016-09-29 01:38 - edited

FYI here it is for 3 pipelines and a minimal install: https://gist.github.com/michaelneale/9127c8e14d16baf05910807bd8ac8cc9

This isn't as problematic as other "real world" examples. It is still 90% noise for the pipeline screen, but the action list seems fairly finite. The problem is with every single plugin adding to it, when it is returned in the listing. It also doesn't seem bugger (no wayward whitespace in _class or nulls).

(putting in current sprint as there is work on to look at search response times)

Michael Neale added a comment - 2016-09-29 01:38 - edited FYI here it is for 3 pipelines and a minimal install: https://gist.github.com/michaelneale/9127c8e14d16baf05910807bd8ac8cc9 This isn't as problematic as other "real world" examples. It is still 90% noise for the pipeline screen, but the action list seems fairly finite. The problem is with every single plugin adding to it, when it is returned in the listing. It also doesn't seem bugger (no wayward whitespace in _class or nulls). (putting in current sprint as there is work on to look at search response times)

Michael Neale made changes - 2016-09-29 01:38

Epic Link

New: JENKINS-35759 [ 171771 ]

Michael Neale made changes - 2016-09-29 01:39

Epic Link

Original: JENKINS-35759 [ 171771 ]

New: JENKINS-37957 [ 174099 ]

Michael Neale made changes - 2016-09-29 01:39

Sprint

New: pacific [ 101 ]

Michael Neale made changes - 2016-09-29 01:39

Description

Original: In many cases the /piplines json is blowing out to 14kb for 26 "rows" of data. This isn't a huge json transfer, but when you look closer, the impact of plugins adding to actions appears to be huge, and may be the cause of some problems.

As part of looking at a support ticket [~tscherler] uncovered that for 26 pipelines, there is really 20 lines of attributes that are used by the dashboard == 520 lines (roughly).
However there is 10k lines in the response - this is a bit out of control.

It seems that every plugin installed adds actions for every little thing, which the dashboard doesn't need.

For example

{noformat}
        "_class": "io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl",
        "_links": {
          "self": {
            "_class": "io.jenkins.blueocean.rest.hal.Link",
            "href": "/blue/rest/organizations/jenkins/... (hidden)"
          }
        },
        "_class": "com.cloudbees.plugins.credentials .ViewCredentialsAction",
        "stores": {},
        "urlName": "credentials"
      }
    ],
{noformat}

is added to every pipeline (along with 100's of others). Note also the whitespace (may be another problem). It really depends on the plugins.

Also

{noformat}
         {
            "_class": "hudson.model.TextParameterDefinition",
            "defaultParameterValue": {
              "_class": "hudson.model.StringParameterValue",
              "name": "jiraComment",
              "value": ""
            },
            "description": "Additional info for theissue's comment",
            "name": "jiraComment",
            "type": "TextParameterDefinition"
          },
{noformat}

will never be of use to the dashboard (well probably).

Some questions:

Is there a better query to use to make the response smaller/more compact? (even just eyeballing the reponse could be helped by not returning the metadata optionally). I expect the calculation of this is quite expensive.

Could this be a source of the slowdown when people have enough/certain plugins installed?

New: In many cases the piplines json for the dashboard (search results) is blowing out to 14kb for 26 "rows" of data. This isn't a huge json transfer, but when you look closer, the impact of plugins adding to actions appears to be huge, and may be the cause of some problems.

As part of looking at a support ticket [~tscherler] uncovered that for 26 pipelines, there is really 20 lines of attributes that are used by the dashboard == 520 lines (roughly).
However there is 10k lines in the response - this is a bit out of control.

It seems that every plugin installed adds actions for every little thing, which the dashboard doesn't need.

For example

{noformat}
        "_class": "io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl",
        "_links": {
          "self": {
            "_class": "io.jenkins.blueocean.rest.hal.Link",
            "href": "/blue/rest/organizations/jenkins/... (hidden)"
          }
        },
        "_class": "com.cloudbees.plugins.credentials .ViewCredentialsAction",
        "stores": {},
        "urlName": "credentials"
      }
    ],
{noformat}

is added to every pipeline (along with 100's of others). Note also the whitespace (may be another problem). It really depends on the plugins.

Also

{noformat}
         {
            "_class": "hudson.model.TextParameterDefinition",
            "defaultParameterValue": {
              "_class": "hudson.model.StringParameterValue",
              "name": "jiraComment",
              "value": ""
            },
            "description": "Additional info for theissue's comment",
            "name": "jiraComment",
            "type": "TextParameterDefinition"
          },
{noformat}

will never be of use to the dashboard (well probably).

Some questions:

Is there a better query to use to make the response smaller/more compact? (even just eyeballing the reponse could be helped by not returning the metadata optionally). I expect the calculation of this is quite expensive.

Could this be a source of the slowdown when people have enough/certain plugins installed?

Michael Neale added a comment - 2016-09-29 01:41

tscherler can you confirm if the metadata is used at all on the dashboard now? (if you know) - perhaps the solution is to not have it for the listing (but I don't know if some is used). Whilst future plugins may use it - it is less likely on dashboard listing.

Michael Neale added a comment - 2016-09-29 01:41 tscherler can you confirm if the metadata is used at all on the dashboard now? (if you know) - perhaps the solution is to not have it for the listing (but I don't know if some is used). Whilst future plugins may use it - it is less likely on dashboard listing.

Thorsten Scherler added a comment - 2016-09-29 08:46

Quick comment on the whitespaces in the attributes, I did the formating with regexp so that may be extra whitespaces that are not shown in real world response.

AFAIK we use bits of the metadata. There is some code using the links and some use capabilities but that we do strictly on the root class/response object. The one exception is logs, where we have to scan the actions of a step to determines, whether we support logs.

Thorsten Scherler added a comment - 2016-09-29 08:46 Quick comment on the whitespaces in the attributes, I did the formating with regexp so that may be extra whitespaces that are not shown in real world response. AFAIK we use bits of the metadata. There is some code using the links and some use capabilities but that we do strictly on the root class/response object. The one exception is logs, where we have to scan the actions of a step to determines, whether we support logs.

Michael Neale made changes - 2016-10-04 05:12

Rank

New: Ranked higher

Michael Neale made changes - 2016-10-04 05:12

Description

Original: In many cases the piplines json for the dashboard (search results) is blowing out to 14kb for 26 "rows" of data. This isn't a huge json transfer, but when you look closer, the impact of plugins adding to actions appears to be huge, and may be the cause of some problems.

As part of looking at a support ticket [~tscherler] uncovered that for 26 pipelines, there is really 20 lines of attributes that are used by the dashboard == 520 lines (roughly).
However there is 10k lines in the response - this is a bit out of control.

It seems that every plugin installed adds actions for every little thing, which the dashboard doesn't need.

For example

{noformat}
        "_class": "io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl",
        "_links": {
          "self": {
            "_class": "io.jenkins.blueocean.rest.hal.Link",
            "href": "/blue/rest/organizations/jenkins/... (hidden)"
          }
        },
        "_class": "com.cloudbees.plugins.credentials .ViewCredentialsAction",
        "stores": {},
        "urlName": "credentials"
      }
    ],
{noformat}

is added to every pipeline (along with 100's of others). Note also the whitespace (may be another problem). It really depends on the plugins.

Also

{noformat}
         {
            "_class": "hudson.model.TextParameterDefinition",
            "defaultParameterValue": {
              "_class": "hudson.model.StringParameterValue",
              "name": "jiraComment",
              "value": ""
            },
            "description": "Additional info for theissue's comment",
            "name": "jiraComment",
            "type": "TextParameterDefinition"
          },
{noformat}

will never be of use to the dashboard (well probably).

Some questions:

Is there a better query to use to make the response smaller/more compact? (even just eyeballing the reponse could be helped by not returning the metadata optionally). I expect the calculation of this is quite expensive.

Could this be a source of the slowdown when people have enough/certain plugins installed?

New: In many cases the piplines json for the dashboard (search results) is blowing out to 14kb for 26 "rows" of data. This isn't a huge json transfer, but when you look closer, the impact of plugins adding to actions appears to be huge, and may be the cause of some problems.

As part of looking at a support ticket [~tscherler] uncovered that for 26 pipelines, there is really 20 lines of attributes that are used by the dashboard == 520 lines (roughly).
However there is 10k lines in the response - this is a bit out of control.

It seems that every plugin installed adds actions for every little thing, which the dashboard doesn't need.

For example

{noformat}
        "_class": "io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl",
        "_links": {
          "self": {
            "_class": "io.jenkins.blueocean.rest.hal.Link",
            "href": "/blue/rest/organizations/jenkins/... (hidden)"
          }
        },
        "_class": "com.cloudbees.plugins.credentials .ViewCredentialsAction",
        "stores": {},
        "urlName": "credentials"
      }
    ],
{noformat}

is added to every pipeline (along with 100's of others). It really depends on the plugins.

Also

{noformat}
         {
            "_class": "hudson.model.TextParameterDefinition",
            "defaultParameterValue": {
              "_class": "hudson.model.StringParameterValue",
              "name": "jiraComment",
              "value": ""
            },
            "description": "Additional info for theissue's comment",
            "name": "jiraComment",
            "type": "TextParameterDefinition"
          },
{noformat}

will never be of use to the dashboard (well probably).

Some questions:

Is there a better query to use to make the response smaller/more compact? (even just eyeballing the reponse could be helped by not returning the metadata optionally). I expect the calculation of this is quite expensive.

Could this be a source of the slowdown when people have enough/certain plugins installed?

Assignee:: Vivek Pandey

Reporter:: Michael Neale

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2016-09-29 01:36

Updated:: 2016-10-24 02:37

Resolved:: 2016-10-24 02:37

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Michael Neale added a comment - 2016-09-29 01:38, Edited by Michael Neale - 2016-09-29 01:39

Expand comment: Michael Neale added a comment - 2016-09-29 01:38, Edited by Michael Neale - 2016-09-29 01:39

Collapse comment: Michael Neale added a comment - 2016-09-29 01:41

Expand comment: Michael Neale added a comment - 2016-09-29 01:41

Collapse comment: Thorsten Scherler added a comment - 2016-09-29 08:46

Expand comment: Thorsten Scherler added a comment - 2016-09-29 08:46

People

Dates