Need to add SSE events for start and stop of indexing

Need to add SSE events for start and stop of indexing

An SSE event or a REST API? They seem like 2 different things in how they are described. Maybe the description is just wrong.

Also, maybe a bit more detail on the problem that we're trying to solve. AFAIK, MBP indexing results in SSE events when new pipeline jobs are created etc. Is there a bug with that (not created or not consumed), or is something else needed?

tfennelly jamesdumay can provide more details but from what I understand basically we need SSE events for start and stop of indexing so that UI can show indexing status.

So I tried it on BO master branch and triggering indexing of MBP is not resulting in to an SSE events, that is by looking in to chrome networking req/resp. I suspect BranchIndexing.run() basically FolderComputation.run() is not firing off RunListener.fireStarted() or other events. On blueocean side of things look ok, SSEService.js is subscribing to job channel and accepting all events.

vivek Hey Vivek ... the network tab is not going to tell you anything useful here. You need to turn on the SSE event logging in order to see the events coming through to the client. What I would expect is that you'd see job_crud_created events being logged to the console (among other things).

But first ... lets please describe what we are trying to do from a higher level. Then we can figure out what we need or if there's a bug.

jamesdumay See above comments, please add more details to it so that its clear where/what needs to be done. For one I think intent is to send events for org scan and not MBP indexing or maybe both.

vivek best catch up with cliffmeyers and see what he prefers to determine MBP/Org folder indexing status.

jamesdumay Other day cliffmeyers mentioned, he cares about org scan events, begin and end. Cliff, is that it? tfennelly wants some description of it, like high level use case. cliffmeyers can you sync up your morning with Tom?

Blocking Cliffs Creation epic

jamesdumay brody would help to have some input from product and UX here. After the user creates a Github org folder, what kind of feedback are we going to show at the bottom to illustrate the progress? Are we going to wait until indexing of the org folder is complete to navigate the user back to dashboard? If so we probably just need to know that the org folder's indexing started and stopped. If we are going to show something more detailed to the user, then we would need more granular events.

That's right Cliff. I think it would be good to know if the indexing stopped with an error or success too.

cliffmeyers yes, we want to show it is busy, until it isn't. Once it has finished, then the final state can be displayed unless there was an error (vivek says that the api can tell you if it is finished with an error or not). If there was an error, we should show it (I doubt it will be a very nice error, but we will see. With github 90% of the time is permission so we should capture that before setting it up. The reminaining 9.99% is usually API rate limiting or network, and that should be helped by the update to the SCM api and plugins next week.

An SSE event will make this work nicely in theory...

tfennelly you able to help vivek and cliff here? we might need some more hooks to get MBP and perhaps github org folder events published to the SSE bus, as there doesn't appear to be anything there (may not even be listeners) just yet. Cliff needs to know that a given pipeline has finished indexing at least (and in the case of github org folder scanning for repos, that it has finished).

michaelneale cliffmeyers I'll lend a hand as best I can as soon as someone adds some high level details on what this is about, as requested a few times above.

E.g.

• What's the higher level use case ("We want the user to be able to see ........ when ....... etc etc") ? + maybe a screenshot mockup if we have any.
• What specific events do we think we need fired on the bus on the backend?

Also potentially related to this, cc michaelneale vivek

I noticed that if I create a new Git project based off a bogus URL (e.g. http://www.foo) that creation immediately returns with success. I understand the API constraints and why it needs to create it immediately, but it feels like we need to let Jenkins to do a little work first and give the user some kind of indication that something went wrong?

If that's the case, then maybe we need some other kind of event that indicates an indexing failure, or that indexing didn't find anything useful?

cliffmeyers This kind of validation can be done, like given git url is valid git or not. Please note, even if its valid URL, indexing might succeed with no branch as there may not be any Jenkinsfile. You are right an event to indicate any valid pipeline branch is found or no pipeline found might help.

cliffmeyers Would it be possible to have more generic events for "Github org folder started" etc? Otherwise I'd like us to drill into why it needs to be explicit to Github so I can get a better idea.

As for errors etc ... can that kind of thing not be a status/result on the end event ?

tfennelly you're right, it doesn't (and probably shouldn't) be specified to Github. It could probably be basic events about "indexing started" and "indexing completed", perhaps with some additional metadata that could express the type of the thing that was indexing? If we have HAL href's the UI might very well be ignore the types anyways and just compare based off href, but other consumers might need some kind of type info.

For errors, that would be fine. If I recall, a job "completes" but has a state of failed, success ,etc? If so, probably good to follow that Jenkins convention.

cliffmeyers So I guess we need to find out what's available in pipeline (or one of its sub plugins) that we can use to trigger the publishing of these events on the pubsub bus.

Actually looks easy enough ... requires an addition to the pubsub module.

Experiments under way with Cliff. Waiting for feedback on https://github.com/jenkinsci/blueocean-plugin/pull/700

cliffmeyers and I spent some time yesterday investigating the pipeline API looking for a way to get what he needs and we think we found something that will cover at least the normal MBP indexing requirements i.e. sending an "indexing complete with success/fail status" even at the right time. I should have a prototype that he can test later.

Changes for the new event data are now pushed upstream.