Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39794

API to check that MBP indexing has completed

    XMLWordPrintable

Details

    • tasman, frank, tethys, christmas, pannonian

    Attachments

      Issue Links

        Activity

          jamesdumay James Dumay created issue -
          jamesdumay James Dumay made changes -
          Field Original Value New Value
          Epic Link JENKINS-36588 [ 172614 ]
          jamesdumay James Dumay made changes -
          Rank Ranked higher
          jamesdumay James Dumay added a comment -

          Need to add SSE events for start and stop of indexing

          jamesdumay James Dumay added a comment - Need to add SSE events for start and stop of indexing
          jamesdumay James Dumay made changes -
          Sprint tasman [ 136 ]
          jamesdumay James Dumay made changes -
          Rank Ranked lower
          jamesdumay James Dumay made changes -
          Assignee Vivek Pandey [ vivek ]
          jamesdumay James Dumay made changes -
          Sprint tasman [ 136 ] tasman, frank [ 136, 156 ]
          jamesdumay James Dumay made changes -
          Sprint tasman, frank [ 136, 156 ] tasman, tethys [ 136, 161 ]
          jamesdumay James Dumay made changes -
          Rank Ranked higher
          jamesdumay James Dumay made changes -
          Sprint tasman, tethys [ 136, 161 ] tasman, frank [ 136, 156 ]
          jamesdumay James Dumay made changes -
          Sprint tasman, frank [ 136, 156 ] tasman, frank, christmas [ 136, 156, 176 ]
          tfennelly Tom FENNELLY added a comment -

          Need to add SSE events for start and stop of indexing

          An SSE event or a REST API? They seem like 2 different things in how they are described. Maybe the description is just wrong.

          Also, maybe a bit more detail on the problem that we're trying to solve. AFAIK, MBP indexing results in SSE events when new pipeline jobs are created etc. Is there a bug with that (not created or not consumed), or is something else needed?

          tfennelly Tom FENNELLY added a comment - Need to add SSE events for start and stop of indexing An SSE event or a REST API? They seem like 2 different things in how they are described. Maybe the description is just wrong. Also, maybe a bit more detail on the problem that we're trying to solve. AFAIK, MBP indexing results in SSE events when new pipeline jobs are created etc. Is there a bug with that (not created or not consumed), or is something else needed?
          vivek Vivek Pandey made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          vivek Vivek Pandey added a comment -

          tfennelly jamesdumay can provide more details but from what I understand basically we need SSE events for start and stop of indexing so that UI can show indexing status.

          So I tried it on BO master branch and triggering indexing of MBP is not resulting in to an SSE events, that is by looking in to chrome networking req/resp. I suspect BranchIndexing.run() basically FolderComputation.run() is not firing off RunListener.fireStarted() or other events. On blueocean side of things look ok, SSEService.js is subscribing to job channel and accepting all events.

          vivek Vivek Pandey added a comment - tfennelly jamesdumay can provide more details but from what I understand basically we need SSE events for start and stop of indexing so that UI can show indexing status. So I tried it on BO master branch and triggering indexing of MBP is not resulting in to an SSE events, that is by looking in to chrome networking req/resp. I suspect BranchIndexing.run() basically FolderComputation.run() is not firing off RunListener.fireStarted() or other events. On blueocean side of things look ok, SSEService.js is subscribing to job channel and accepting all events.
          tfennelly Tom FENNELLY added a comment -

          vivek Hey Vivek ... the network tab is not going to tell you anything useful here. You need to turn on the SSE event logging in order to see the events coming through to the client. What I would expect is that you'd see job_crud_created events being logged to the console (among other things).

          tfennelly Tom FENNELLY added a comment - vivek Hey Vivek ... the network tab is not going to tell you anything useful here. You need to turn on the SSE event logging in order to see the events coming through to the client. What I would expect is that you'd see job_crud_created events being logged to the console (among other things).
          tfennelly Tom FENNELLY added a comment -

          But first ... lets please describe what we are trying to do from a higher level. Then we can figure out what we need or if there's a bug.

          tfennelly Tom FENNELLY added a comment - But first ... lets please describe what we are trying to do from a higher level. Then we can figure out what we need or if there's a bug.
          vivek Vivek Pandey added a comment -

          jamesdumay See above comments, please add more details to it so that its clear where/what needs to be done. For one I think intent is to send events for org scan and not MBP indexing or maybe both.

          vivek Vivek Pandey added a comment - jamesdumay See above comments, please add more details to it so that its clear where/what needs to be done. For one I think intent is to send events for org scan and not MBP indexing or maybe both.
          jamesdumay James Dumay added a comment -

          vivek best catch up with cliffmeyers and see what he prefers to determine MBP/Org folder indexing status.

          jamesdumay James Dumay added a comment - vivek best catch up with cliffmeyers and see what he prefers to determine MBP/Org folder indexing status.
          vivek Vivek Pandey added a comment -

          jamesdumay Other day cliffmeyers mentioned, he cares about org scan events, begin and end. Cliff, is that it? tfennelly wants some description of it, like high level use case. cliffmeyers can you sync up your morning with Tom?

          vivek Vivek Pandey added a comment - jamesdumay Other day cliffmeyers mentioned, he cares about org scan events, begin and end. Cliff, is that it? tfennelly wants some description of it, like high level use case. cliffmeyers can you sync up your morning with Tom?
          jamesdumay James Dumay made changes -
          Sprint tasman, frank, christmas [ 136, 156, 176 ] tasman, frank, tethys, christmas [ 136, 156, 161, 176 ]
          jamesdumay James Dumay made changes -
          Priority Minor [ 4 ] Blocker [ 1 ]
          jamesdumay James Dumay made changes -
          Priority Blocker [ 1 ] Critical [ 2 ]
          jamesdumay James Dumay added a comment -

          Blocking Cliffs Creation epic

          jamesdumay James Dumay added a comment - Blocking Cliffs Creation epic
          cliffmeyers Cliff Meyers added a comment -

          jamesdumay brody would help to have some input from product and UX here. After the user creates a Github org folder, what kind of feedback are we going to show at the bottom to illustrate the progress? Are we going to wait until indexing of the org folder is complete to navigate the user back to dashboard? If so we probably just need to know that the org folder's indexing started and stopped. If we are going to show something more detailed to the user, then we would need more granular events.

          cliffmeyers Cliff Meyers added a comment - jamesdumay brody would help to have some input from product and UX here. After the user creates a Github org folder, what kind of feedback are we going to show at the bottom to illustrate the progress? Are we going to wait until indexing of the org folder is complete to navigate the user back to dashboard? If so we probably just need to know that the org folder's indexing started and stopped. If we are going to show something more detailed to the user, then we would need more granular events.
          jamesdumay James Dumay added a comment -

          That's right Cliff. I think it would be good to know if the indexing stopped with an error or success too.

          jamesdumay James Dumay added a comment - That's right Cliff. I think it would be good to know if the indexing stopped with an error or success too.
          michaelneale Michael Neale made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          michaelneale Michael Neale added a comment - - edited

          cliffmeyers yes, we want to show it is busy, until it isn't. Once it has finished, then the final state can be displayed unless there was an error (vivek says that the api can tell you if it is finished with an error or not). If there was an error, we should show it (I doubt it will be a very nice error, but we will see. With github 90% of the time is permission so we should capture that before setting it up. The reminaining 9.99% is usually API rate limiting or network, and that should be helped by the update to the SCM api and plugins next week.

          An SSE event will make this work nicely in theory...

          tfennelly you able to help vivek and cliff here? we might need some more hooks to get MBP and perhaps github org folder events published to the SSE bus, as there doesn't appear to be anything there (may not even be listeners) just yet. Cliff needs to know that a given pipeline has finished indexing at least (and in the case of github org folder scanning for repos, that it has finished).

          michaelneale Michael Neale added a comment - - edited cliffmeyers yes, we want to show it is busy, until it isn't. Once it has finished, then the final state can be displayed unless there was an error (vivek says that the api can tell you if it is finished with an error or not). If there was an error, we should show it (I doubt it will be a very nice error, but we will see. With github 90% of the time is permission so we should capture that before setting it up. The reminaining 9.99% is usually API rate limiting or network, and that should be helped by the update to the SCM api and plugins next week. An SSE event will make this work nicely in theory... tfennelly you able to help vivek and cliff here? we might need some more hooks to get MBP and perhaps github org folder events published to the SSE bus, as there doesn't appear to be anything there (may not even be listeners) just yet. Cliff needs to know that a given pipeline has finished indexing at least (and in the case of github org folder scanning for repos, that it has finished).
          michaelneale Michael Neale made changes -
          Assignee Vivek Pandey [ vivek ] Tom FENNELLY [ tfennelly ]
          jamesdumay James Dumay made changes -
          Rank Ranked higher
          tfennelly Tom FENNELLY added a comment -

          michaelneale cliffmeyers I'll lend a hand as best I can as soon as someone adds some high level details on what this is about, as requested a few times above.

          E.g.

          • What's the higher level use case ("We want the user to be able to see ........ when ....... etc etc") ? + maybe a screenshot mockup if we have any.
          • What specific events do we think we need fired on the bus on the backend?
          tfennelly Tom FENNELLY added a comment - michaelneale cliffmeyers I'll lend a hand as best I can as soon as someone adds some high level details on what this is about, as requested a few times above. E.g. What's the higher level use case ("We want the user to be able to see ........ when ....... etc etc") ? + maybe a screenshot mockup if we have any. What specific events do we think we need fired on the bus on the backend?
          cliffmeyers Cliff Meyers added a comment -

          tfennelly for context I would check out these two links:

          Github Creation Flow, especially steps 7-9: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-ConnectingtoGithub.com
          Git Creation flow, steps 2.1 and 2.2: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-Git

          So once the user presses the create button, the UI will call the backend API to begin creation, but that really only begins the process. Jenkins has saved and created the folder or pipeline, but it hasn't been indexed yet, so the data is just an empty shell, and navigating to any page in the UI at this moment isn't going to show anything useful. The UI will stay in the creation UI for a bit while "the system is working" and then perform some navigation to an appropriate page at the right time:

          1. When creating a Github org folder a.k.a. "auto discover": the UI should stay in the creation flow until all of the pipelines for that org have been created. This is the concept we've been discussing of getting events for when an org folder's indexing starts and completes. I suspect the progress indicator at the bottom would move through these states:
            1. (making API call to begin creation) - "Saving Changes..."
            2. (after API calls completes) - "Waiting for Indexing..."
            3. (after receiving "indexing started" event) - "Creating Pipelines"
            4. (after receiving "indexing complete" event) - "Pipelines Created!" - navigate back to Dashboard, show the pipelines that were created. If the UI could make a REST API call to fetch additional data about the Github Org folder (for example, the names of the pipelines in it, similar to a MBP's "branchNames" property) then we could show some cool info like "22 Pipelines Created!"
          2. When creating a single repo (either from Github or Git flow), it's not an org folder indexing that's relevant, but the indexing of a specific multibranch pipeline. The progress indicator would have these states...
            1. (making API call to begin creation) - "Saving Changes..."
            2. (after API calls completes) - "Waiting for Indexing..."
            3. (after receiving "indexing started" event) - "Creating Pipeline"
            4. (after receiving "indexing complete" event) - "Pipeline Created!" - navigate to /activity tab for that pipeline

          For 2.4, we might eventually smarten it up a bit to navigate either to editor or to in-progress run if the repo had a Jenkinsfile but I don't think that really relates to any of the eventing we're talking about here, just an FYI.

          So to summarize, perhaps this would serve as a suitable ticket description:
          Add the following SSE events:

          • Github org folder indexing started - include a HAL href to the org folder itself
          • Github org folder indexing completed - also with HAL href
          • Multibranch pipeline indexing started - with HAL href to MBP
          • Multibranch pipeline indexing completed - with HAL href
          cliffmeyers Cliff Meyers added a comment - tfennelly for context I would check out these two links: Github Creation Flow, especially steps 7-9: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-ConnectingtoGithub.com Git Creation flow, steps 2.1 and 2.2: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-Git So once the user presses the create button, the UI will call the backend API to begin creation, but that really only begins the process. Jenkins has saved and created the folder or pipeline, but it hasn't been indexed yet, so the data is just an empty shell, and navigating to any page in the UI at this moment isn't going to show anything useful. The UI will stay in the creation UI for a bit while "the system is working" and then perform some navigation to an appropriate page at the right time: When creating a Github org folder a.k.a. "auto discover": the UI should stay in the creation flow until all of the pipelines for that org have been created. This is the concept we've been discussing of getting events for when an org folder's indexing starts and completes. I suspect the progress indicator at the bottom would move through these states: (making API call to begin creation) - "Saving Changes..." (after API calls completes) - "Waiting for Indexing..." (after receiving "indexing started" event) - "Creating Pipelines" (after receiving "indexing complete" event) - "Pipelines Created!" - navigate back to Dashboard, show the pipelines that were created. If the UI could make a REST API call to fetch additional data about the Github Org folder (for example, the names of the pipelines in it, similar to a MBP's "branchNames" property) then we could show some cool info like "22 Pipelines Created!" When creating a single repo (either from Github or Git flow), it's not an org folder indexing that's relevant, but the indexing of a specific multibranch pipeline. The progress indicator would have these states... (making API call to begin creation) - "Saving Changes..." (after API calls completes) - "Waiting for Indexing..." (after receiving "indexing started" event) - "Creating Pipeline" (after receiving "indexing complete" event) - "Pipeline Created!" - navigate to /activity tab for that pipeline For 2.4, we might eventually smarten it up a bit to navigate either to editor or to in-progress run if the repo had a Jenkinsfile but I don't think that really relates to any of the eventing we're talking about here, just an FYI. So to summarize, perhaps this would serve as a suitable ticket description: Add the following SSE events: Github org folder indexing started - include a HAL href to the org folder itself Github org folder indexing completed - also with HAL href Multibranch pipeline indexing started - with HAL href to MBP Multibranch pipeline indexing completed - with HAL href
          cliffmeyers Cliff Meyers added a comment -

          Also potentially related to this, cc michaelneale vivek

          I noticed that if I create a new Git project based off a bogus URL (e.g. http://www.foo) that creation immediately returns with success. I understand the API constraints and why it needs to create it immediately, but it feels like we need to let Jenkins to do a little work first and give the user some kind of indication that something went wrong?

          If that's the case, then maybe we need some other kind of event that indicates an indexing failure, or that indexing didn't find anything useful?

          cliffmeyers Cliff Meyers added a comment - Also potentially related to this, cc michaelneale vivek I noticed that if I create a new Git project based off a bogus URL (e.g. http://www.foo ) that creation immediately returns with success. I understand the API constraints and why it needs to create it immediately, but it feels like we need to let Jenkins to do a little work first and give the user some kind of indication that something went wrong? If that's the case, then maybe we need some other kind of event that indicates an indexing failure, or that indexing didn't find anything useful?
          vivek Vivek Pandey added a comment -

          cliffmeyers This kind of validation can be done, like given git url is valid git or not. Please note, even if its valid URL, indexing might succeed with no branch as there may not be any Jenkinsfile. You are right an event to indicate any valid pipeline branch is found or no pipeline found might help.

          vivek Vivek Pandey added a comment - cliffmeyers This kind of validation can be done, like given git url is valid git or not. Please note, even if its valid URL, indexing might succeed with no branch as there may not be any Jenkinsfile. You are right an event to indicate any valid pipeline branch is found or no pipeline found might help.
          tfennelly Tom FENNELLY added a comment -

          cliffmeyers Would it be possible to have more generic events for "Github org folder started" etc? Otherwise I'd like us to drill into why it needs to be explicit to Github so I can get a better idea.

          As for errors etc ... can that kind of thing not be a status/result on the end event ?

          tfennelly Tom FENNELLY added a comment - cliffmeyers Would it be possible to have more generic events for "Github org folder started" etc? Otherwise I'd like us to drill into why it needs to be explicit to Github so I can get a better idea. As for errors etc ... can that kind of thing not be a status/result on the end event ?
          cliffmeyers Cliff Meyers added a comment -

          tfennelly you're right, it doesn't (and probably shouldn't) be specified to Github. It could probably be basic events about "indexing started" and "indexing completed", perhaps with some additional metadata that could express the type of the thing that was indexing? If we have HAL href's the UI might very well be ignore the types anyways and just compare based off href, but other consumers might need some kind of type info.

          For errors, that would be fine. If I recall, a job "completes" but has a state of failed, success ,etc? If so, probably good to follow that Jenkins convention.

          cliffmeyers Cliff Meyers added a comment - tfennelly you're right, it doesn't (and probably shouldn't) be specified to Github. It could probably be basic events about "indexing started" and "indexing completed", perhaps with some additional metadata that could express the type of the thing that was indexing? If we have HAL href's the UI might very well be ignore the types anyways and just compare based off href, but other consumers might need some kind of type info. For errors, that would be fine. If I recall, a job "completes" but has a state of failed, success ,etc? If so, probably good to follow that Jenkins convention.
          tfennelly Tom FENNELLY made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          tfennelly Tom FENNELLY added a comment -

          cliffmeyers So I guess we need to find out what's available in pipeline (or one of its sub plugins) that we can use to trigger the publishing of these events on the pubsub bus.

          tfennelly Tom FENNELLY added a comment - cliffmeyers So I guess we need to find out what's available in pipeline (or one of its sub plugins) that we can use to trigger the publishing of these events on the pubsub bus.
          tfennelly Tom FENNELLY added a comment -

          Actually looks easy enough ... requires an addition to the pubsub module.

          tfennelly Tom FENNELLY added a comment - Actually looks easy enough ... requires an addition to the pubsub module.
          tfennelly Tom FENNELLY added a comment -

          Experiments under way with Cliff. Waiting for feedback on https://github.com/jenkinsci/blueocean-plugin/pull/700

          tfennelly Tom FENNELLY added a comment - Experiments under way with Cliff. Waiting for feedback on https://github.com/jenkinsci/blueocean-plugin/pull/700
          cliffmeyers Cliff Meyers made changes -
          Link This issue blocks JENKINS-41141 [ JENKINS-41141 ]
          tfennelly Tom FENNELLY added a comment -

          cliffmeyers and I spent some time yesterday investigating the pipeline API looking for a way to get what he needs and we think we found something that will cover at least the normal MBP indexing requirements i.e. sending an "indexing complete with success/fail status" even at the right time. I should have a prototype that he can test later.

          tfennelly Tom FENNELLY added a comment - cliffmeyers and I spent some time yesterday investigating the pipeline API looking for a way to get what he needs and we think we found something that will cover at least the normal MBP indexing requirements i.e. sending an "indexing complete with success/fail status" even at the right time . I should have a prototype that he can test later.
          jamesdumay James Dumay made changes -
          Sprint tasman, frank, tethys, christmas [ 136, 156, 161, 176 ] tasman, frank, tethys, christmas, pannonian [ 136, 156, 161, 176, 211 ]
          tfennelly Tom FENNELLY made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          tfennelly Tom FENNELLY made changes -
          Resolution Fixed [ 1 ]
          Status In Review [ 10005 ] Resolved [ 5 ]
          tfennelly Tom FENNELLY added a comment -

          Changes for the new event data are now pushed upstream.

          tfennelly Tom FENNELLY added a comment - Changes for the new event data are now pushed upstream.

          People

            tfennelly Tom FENNELLY
            jamesdumay James Dumay
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: