• Icon: Task Task
    • Resolution: Fixed
    • Icon: Critical Critical
    • blueocean-plugin
    • None
    • tasman, frank, tethys, christmas, pannonian

        [JENKINS-39794] API to check that MBP indexing has completed

        Cliff Meyers added a comment -

        tfennelly for context I would check out these two links:

        Github Creation Flow, especially steps 7-9: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-ConnectingtoGithub.com
        Git Creation flow, steps 2.1 and 2.2: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-Git

        So once the user presses the create button, the UI will call the backend API to begin creation, but that really only begins the process. Jenkins has saved and created the folder or pipeline, but it hasn't been indexed yet, so the data is just an empty shell, and navigating to any page in the UI at this moment isn't going to show anything useful. The UI will stay in the creation UI for a bit while "the system is working" and then perform some navigation to an appropriate page at the right time:

        1. When creating a Github org folder a.k.a. "auto discover": the UI should stay in the creation flow until all of the pipelines for that org have been created. This is the concept we've been discussing of getting events for when an org folder's indexing starts and completes. I suspect the progress indicator at the bottom would move through these states:
          1. (making API call to begin creation) - "Saving Changes..."
          2. (after API calls completes) - "Waiting for Indexing..."
          3. (after receiving "indexing started" event) - "Creating Pipelines"
          4. (after receiving "indexing complete" event) - "Pipelines Created!" - navigate back to Dashboard, show the pipelines that were created. If the UI could make a REST API call to fetch additional data about the Github Org folder (for example, the names of the pipelines in it, similar to a MBP's "branchNames" property) then we could show some cool info like "22 Pipelines Created!"
        2. When creating a single repo (either from Github or Git flow), it's not an org folder indexing that's relevant, but the indexing of a specific multibranch pipeline. The progress indicator would have these states...
          1. (making API call to begin creation) - "Saving Changes..."
          2. (after API calls completes) - "Waiting for Indexing..."
          3. (after receiving "indexing started" event) - "Creating Pipeline"
          4. (after receiving "indexing complete" event) - "Pipeline Created!" - navigate to /activity tab for that pipeline

        For 2.4, we might eventually smarten it up a bit to navigate either to editor or to in-progress run if the repo had a Jenkinsfile but I don't think that really relates to any of the eventing we're talking about here, just an FYI.

        So to summarize, perhaps this would serve as a suitable ticket description:
        Add the following SSE events:

        • Github org folder indexing started - include a HAL href to the org folder itself
        • Github org folder indexing completed - also with HAL href
        • Multibranch pipeline indexing started - with HAL href to MBP
        • Multibranch pipeline indexing completed - with HAL href

        Cliff Meyers added a comment - tfennelly for context I would check out these two links: Github Creation Flow, especially steps 7-9: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-ConnectingtoGithub.com Git Creation flow, steps 2.1 and 2.2: https://cloudbees.atlassian.net/wiki/display/UX/Pipeline+Creation#PipelineCreation-Git So once the user presses the create button, the UI will call the backend API to begin creation, but that really only begins the process. Jenkins has saved and created the folder or pipeline, but it hasn't been indexed yet, so the data is just an empty shell, and navigating to any page in the UI at this moment isn't going to show anything useful. The UI will stay in the creation UI for a bit while "the system is working" and then perform some navigation to an appropriate page at the right time: When creating a Github org folder a.k.a. "auto discover": the UI should stay in the creation flow until all of the pipelines for that org have been created. This is the concept we've been discussing of getting events for when an org folder's indexing starts and completes. I suspect the progress indicator at the bottom would move through these states: (making API call to begin creation) - "Saving Changes..." (after API calls completes) - "Waiting for Indexing..." (after receiving "indexing started" event) - "Creating Pipelines" (after receiving "indexing complete" event) - "Pipelines Created!" - navigate back to Dashboard, show the pipelines that were created. If the UI could make a REST API call to fetch additional data about the Github Org folder (for example, the names of the pipelines in it, similar to a MBP's "branchNames" property) then we could show some cool info like "22 Pipelines Created!" When creating a single repo (either from Github or Git flow), it's not an org folder indexing that's relevant, but the indexing of a specific multibranch pipeline. The progress indicator would have these states... (making API call to begin creation) - "Saving Changes..." (after API calls completes) - "Waiting for Indexing..." (after receiving "indexing started" event) - "Creating Pipeline" (after receiving "indexing complete" event) - "Pipeline Created!" - navigate to /activity tab for that pipeline For 2.4, we might eventually smarten it up a bit to navigate either to editor or to in-progress run if the repo had a Jenkinsfile but I don't think that really relates to any of the eventing we're talking about here, just an FYI. So to summarize, perhaps this would serve as a suitable ticket description: Add the following SSE events: Github org folder indexing started - include a HAL href to the org folder itself Github org folder indexing completed - also with HAL href Multibranch pipeline indexing started - with HAL href to MBP Multibranch pipeline indexing completed - with HAL href

        Cliff Meyers added a comment -

        Also potentially related to this, cc michaelneale vivek

        I noticed that if I create a new Git project based off a bogus URL (e.g. http://www.foo) that creation immediately returns with success. I understand the API constraints and why it needs to create it immediately, but it feels like we need to let Jenkins to do a little work first and give the user some kind of indication that something went wrong?

        If that's the case, then maybe we need some other kind of event that indicates an indexing failure, or that indexing didn't find anything useful?

        Cliff Meyers added a comment - Also potentially related to this, cc michaelneale vivek I noticed that if I create a new Git project based off a bogus URL (e.g. http://www.foo ) that creation immediately returns with success. I understand the API constraints and why it needs to create it immediately, but it feels like we need to let Jenkins to do a little work first and give the user some kind of indication that something went wrong? If that's the case, then maybe we need some other kind of event that indicates an indexing failure, or that indexing didn't find anything useful?

        Vivek Pandey added a comment -

        cliffmeyers This kind of validation can be done, like given git url is valid git or not. Please note, even if its valid URL, indexing might succeed with no branch as there may not be any Jenkinsfile. You are right an event to indicate any valid pipeline branch is found or no pipeline found might help.

        Vivek Pandey added a comment - cliffmeyers This kind of validation can be done, like given git url is valid git or not. Please note, even if its valid URL, indexing might succeed with no branch as there may not be any Jenkinsfile. You are right an event to indicate any valid pipeline branch is found or no pipeline found might help.

        Tom FENNELLY added a comment -

        cliffmeyers Would it be possible to have more generic events for "Github org folder started" etc? Otherwise I'd like us to drill into why it needs to be explicit to Github so I can get a better idea.

        As for errors etc ... can that kind of thing not be a status/result on the end event ?

        Tom FENNELLY added a comment - cliffmeyers Would it be possible to have more generic events for "Github org folder started" etc? Otherwise I'd like us to drill into why it needs to be explicit to Github so I can get a better idea. As for errors etc ... can that kind of thing not be a status/result on the end event ?

        Cliff Meyers added a comment -

        tfennelly you're right, it doesn't (and probably shouldn't) be specified to Github. It could probably be basic events about "indexing started" and "indexing completed", perhaps with some additional metadata that could express the type of the thing that was indexing? If we have HAL href's the UI might very well be ignore the types anyways and just compare based off href, but other consumers might need some kind of type info.

        For errors, that would be fine. If I recall, a job "completes" but has a state of failed, success ,etc? If so, probably good to follow that Jenkins convention.

        Cliff Meyers added a comment - tfennelly you're right, it doesn't (and probably shouldn't) be specified to Github. It could probably be basic events about "indexing started" and "indexing completed", perhaps with some additional metadata that could express the type of the thing that was indexing? If we have HAL href's the UI might very well be ignore the types anyways and just compare based off href, but other consumers might need some kind of type info. For errors, that would be fine. If I recall, a job "completes" but has a state of failed, success ,etc? If so, probably good to follow that Jenkins convention.

        Tom FENNELLY added a comment -

        cliffmeyers So I guess we need to find out what's available in pipeline (or one of its sub plugins) that we can use to trigger the publishing of these events on the pubsub bus.

        Tom FENNELLY added a comment - cliffmeyers So I guess we need to find out what's available in pipeline (or one of its sub plugins) that we can use to trigger the publishing of these events on the pubsub bus.

        Tom FENNELLY added a comment -

        Actually looks easy enough ... requires an addition to the pubsub module.

        Tom FENNELLY added a comment - Actually looks easy enough ... requires an addition to the pubsub module.

        Tom FENNELLY added a comment -

        Experiments under way with Cliff. Waiting for feedback on https://github.com/jenkinsci/blueocean-plugin/pull/700

        Tom FENNELLY added a comment - Experiments under way with Cliff. Waiting for feedback on https://github.com/jenkinsci/blueocean-plugin/pull/700

        Tom FENNELLY added a comment -

        cliffmeyers and I spent some time yesterday investigating the pipeline API looking for a way to get what he needs and we think we found something that will cover at least the normal MBP indexing requirements i.e. sending an "indexing complete with success/fail status" even at the right time. I should have a prototype that he can test later.

        Tom FENNELLY added a comment - cliffmeyers and I spent some time yesterday investigating the pipeline API looking for a way to get what he needs and we think we found something that will cover at least the normal MBP indexing requirements i.e. sending an "indexing complete with success/fail status" even at the right time . I should have a prototype that he can test later.

        Tom FENNELLY added a comment -

        Changes for the new event data are now pushed upstream.

        Tom FENNELLY added a comment - Changes for the new event data are now pushed upstream.

          tfennelly Tom FENNELLY
          jamesdumay James Dumay
          Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

            Created:
            Updated:
            Resolved: