Workaround: use jackson2-api plugin version 2.10.3 for now

Workaround: use jackson2-api plugin version 2.10.3 for now

Thanks timja, just to confirm the workaround works. Downgraded "Jackson 2 API" to 2.10.3 and we are getting again all those lovely indications.

Ciao

timja - It looks like version `3.0.3` and `4.0.1` of `mongojack` addresses the GitHub issue that you raised.

What else needs to be done to the Build Failure Analyzer plugin to get this working again?

updating the plugin to use the new sdk it currently uses 2.x

Hi ,

Our Jenkins version is 2.222.3 and Our Build failure Analyzer plugin version is 1.24.2.

We too facing same issue and had to downgrade Jackson 2 API" to 2.10.2

But whenever Jenkins master restarts automatically Jackson 2 API get upgrade to its latest version 2.11.0.

Please any one can tell me which are all plugin has dependency on Jackson 2 API

Any idea when this will be fixed? Downgrading Jackson API is not an option here since that would require too many other plugins to be downgraded too.

Hi,
It would be helpful if we get any idea or hints to fix this without downgrading Jackson 2 API plugin

This is blocking my company from upgrading to LTS 2.235.

This is blocking us from updating the Kubernetes plugin (dependabot has updated jackson to 2.11) https://plugins.jenkins.io/kubernetes/#releases

Since we also use Kubernetes plugin in conjunction with BFA.

Looking for an update on this. We have no working BFA and can't downgrade the Jackson 2 API plugin.

I would also like to have a fix for this as we use other tools to triage issues based on the BFA data and we cannot drop that support.

I had a quick look at the BFA code and it is not a trivial change. The mongojack migration documentation is incomplete, and without much background into the code I am not sure how to proceed. I've managed to get the pom upgraded and the DB connection fixed, but a lot of the queries are built with BasicDBObjects and I don't think that translates directly? Maybe someone more familiar can comment. https://github.com/jenkinsci/build-failure-analyzer-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/MongoDBKnowledgeBase.java#L544-L561

I have started a migration, but yeah, it's not trivial, guess that's what you get from waiting to upgrade from 2.x to 4.x

A lot of the MongoDBKnowledgeBase needs to be rewritten.

Right now it's on the backburner but I hope to return to it soon.

Is there anything that we can do to help? BFA is an essential part of our infra as we use it to detect build that have failed for infra reason, and it then automatically rebuilds just this part of the build. Without it infra failures propagate to the whole pipeline.

Is there anything that we can do to help t_westling ?

Running into the same issue. BFA is really useful and having it back would be a huge help.

Would it theoretically be possible to use older version of BFA/mongoJack, and use direct jackson dependency inside the pom.file,
so mongoJack would not hit the global newer jackson class from jackso2-api plugin?

Quick update for anyone interested:

I have been able to upgrade to mongojack 4.0.1 with mongodb-driver-sync 4.1.0.

The most basic use cases work now, with the changes in the APIs from 2.x to 4.x.

What is left is to try out the statistics part as well as fix tests.

Sounds promising. Thanks a lot!

Thanks twestling for the progress on these changes.

We are also using BFA across multiple Jenkins masters and this feature is really providing added value !

Would pinning the jackson library inside of the BFA jar to the old version work as was suggested above? That would be a quick temporary fix while we wait for the new release.

On a separate topic, t_westling we also pull the BFA mongo DB statistics data in a separate app to pull in dashboards and other statistics, and it is sensitive to changes in the DB schemas/tables. Hopefully there won't be any changes there?

Hi t_westling, is there anything that we can do to help?

I upgraded Jenkins this morning, faced this issue (again - I'm not a smart person), and now I can't seem to be able to downgrade again.

The reason for the lack of a solution/update is that I haven't been able to carve out the time at work to focus on this.

I am where I was in the beginning of October, all of the "normal" use cases work, but I'm having difficulties translating the mongo operations for statistics to the new apis( not that I think it is impossible, I am just not very proficient in the api).

I hope that I will return to this soon. Not really sure how you could help, I mostly need to get the operations for getting and putting statistics correct.

Is there any chance you could provide an incomplete version of the patch in a temporary branch on https://github.com/jenkinsci/build-failure-analyzer-plugin ?
I understand that the work is incomplete but this is pretty blocking on my side so I can dedicate some time for it to test the patch and eventually provide some missing pieces.

Sorry to be insistent but is there any update?

This is such a time saving plugin for identifying failure causes automatically, however this issue is currently blocking our upgrade. Thanks for your work on this issue t_westling! Any updates (including that you don't have time for this for a few months), would be appreciated for the purposes of planning upgrade cycles.

I prepared migration script for mongodb knowledge base to build-failure-analyzer.xml, maybe someone else might find it useful 

https://github.com/cvakiitho/bfa-migrate-from-mongo 

cvakiitho, thanks a lot for the script, but I don't quite understand how to use it. The README says "dump knowledgebase from mongodb to failure_cause.json", but mongodump dumps the DB to a directory containing BSON files. Also, what to do with result.xml?

Hi,

We are also seeing the same issue with Jenkins master version 2.249.2. Can you let us know if  there is any update ?

Cannot Downgrade the jackson-api plugin to 2.10.3 as this is causing dependency errors with kubernetes plugin

Kubernetes plugin (1.27.3)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.2 or higher

Kubernetes Client API Plugin (4.11.1)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.2 or higher

Jenkins Jira plugin (3.1.1)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.0 or higher

Kubernetes Credentials Plugin (0.7.0)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.0 or higher

Code Coverage API Plugin (1.2.0)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.1 or higher

DataTables.net API Plugin (1.10.21-2)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.0 or higher

ECharts API Plugin (4.9.0-2)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.1 or higher

Bitbucket Branch Source Plugin (2.9.4)

Update required: Jackson 2 API Plugin (2.10.3) to be updated to 2.11.0 or higher

Thanks cvakiitho for providing https://github.com/cvakiitho/bfa-migrate-from-mongo , that allowed us to start using the plugin again even if we are not using mongodb.

For others that would like to do that as well:

• I cloned https://github.com/cvakiitho/bfa-migrate-from-mongo
• I used mongoexport --db jenkins --collection failureCauses to get a snapshot of the rules stored in mongodb
• I updated bfa-migrate-from-mongo/failure_cause.js with the result of mongoexport

• cd bfa-migrate-from-mongo
  npm install
  node ./index.js
  

  After that the result.xml should be updated with the rules that you had in mongodb

• The result.xml should be copied in place of build-failure-analyzer.xml in your jenkins installation folder.
  I did that and tried to reload the jenkins configuration but that wouldn't work.
  So I stopped Jenkins, copied the files and started it again. And rules did load properly.
  There might be a more optimized version of that but in the meantime, I'm glad that this works

FYI this is how we use the BFA plugin: https://gist.github.com/CoRfr/b0c064c578e19133ed180209167f3ba3
We have rules that have a 'retrigger' category, that is used by a modified build() pipeline function (see gist above).
If a pipeline triggers a build, and that sub-build fails because of a known issue that has the 'retrigger' category, the sub-build is automatically retriggered without the pipeline having to handle the issue. This allows us to tolerate things like node failures without it having too much impact on our pipelines.

Sorry for the radio silence. I now have a work in progress commit on my fork.
From my tests, everything mongo-wise works with Jenkins 2.222.4.
Tests are failing and some general cleanup is probably needed, but for anyone interested, you should be able to try it out. This has been tested on a clean setup without any old database, and come to think of it, I probably need to change some stuff in regards to how the statistics are saved.

I would wait with trying it out on your main database, consider it a pre-alpha version:

https://github.com/TWestling/build-failure-analyzer-plugin

After test fixing a proper pull request will appear.

Hi,

Has anyone tried the fork yet? I tried it and I didn't succeed, I got an error when building it but it is the first time I try to build a Jenkins plugin, so I might be missing something. Actually, I didn't manage to make it work in our Jenkins instance and eventually I am building it locally in a docker container. The error I find is that some objects are not found in org.mongojack (e.g.: DBCursor). The version installed of mongojack is the right one 4.0.1. But when I check the mongojack code for 4.0.1. I don't see any DBCursor defined.

It would be great to know if someone managed to make it work and even greater, some instructions or any post/pointer. 

Thanks!

There shouldn't be any refertences to dbcursor in the code anymore, since that is one difference between the 2.x and the 4.x versions of mongojack.

I have gotten all the tests green now, but I still have some issues with the graph support, specifically one graph.

I will upload a new commit soon with the changes for the tests, maybe you can give that a go once it exists.

I have now created a pull request with working tests:

https://github.com/jenkinsci/build-failure-analyzer-plugin/pull/136

I would be thankful if anyone could try this out and get back with feedback, specifically
someone that uses the built-in graph support, since we don't at my company, so my tests aren't
real-world ones.

From what I can see, everything is working.

I tried, but I don't even see the "Failure Cause Management" link anymore to configure the BFA plugin or failure causes. I also don't see any permission entries (Matrix-based security). This is on Jenkins 2.263.1, which was released last week.

Update: With 1.27.1 installed the link shows up, it disappears again after installing your test version.

Update 2: The global configuration block also disappears after installing your version.

I don't have the issues Dirk mentions;

• I still see the failure cause management link, no discernible UI differences from before
• I still see the global configuration block

This is with Jenkins 2.263.1

I did have to build with 'mvn install -DskipTests=true' - 71 tests seem to all be failing on my system (attached logsout.txt)

I don't use the graphs but I do have them enabled; I've always had a problem that some of the graphs don't load, but at a glance the ones that do look similar to those before.

EDIT: The MAX graphs with this plugin are identical to the graphs I have for the previous version. (4 graphs load and show identical data, 3 graphs don't load)

OK, maybe I did it wrong. I just cloned (your fork of) the plugin repository and built it using "mvn hpi:hpi", then uploaded the HPI file to a test Jenkins controller and restarted Jenkins.

OK, after running the same build command as krtrego, it's also working for me now.

While working on upgrading to the new mongojack api, I had the most trouble with getting graphs to work.
This led me believe that we really shouldn't have the graph support inside the plugin. It bloats the code base
with what feels like very little bang for the buck. We don't use it at my company, instead we send failure cause information
via the MQNotifier plugin to Elastic + Kibana.

I haven't actually heard of anyone using the graph support in a very long time. The implementation was a good idea at the time,
but 7 years later there are better ways of visualizing this data.

After this pull request has been merged, I plan on releasing a 2.0-beta and let the potential users of the graph support adapt for a while.

If anyone is strongly against the removal, please let me know, it would at the very least be interesting to see if anyone is using it.

Oh, yes, we're using it. But we also have an Elkstack server running, so we might be able to migrate to your solution...

t_westling thanks a lot for you work on this. I will also try it in our staging environment. Can you confirm there are no changes to the database schema? We use another dashboard tool to query the BFA mongo DB directly and graph transient issues. Even a small change like a new field or empty field removed can require a change to the aggregation requests.

edit: re your original question, graphs are off here.

sbeaulie: I checked what is saved in the database and did actually notice a change in the statistics part:

"failureCauses" : [
    {
        "failureCause" : DBRef("failureCauses", ObjectId("5fbf63x93c3d411b9310f5f3"), "<DATABASENAME>"),
        "indications" : [
             --              

-

-

        ]
    }
]

The DATABASENAME that is part of the DBRef is added in my change in the current version it looks like: 

DBRef("failureCauses", ObjectId("5fd71c71b1xb6e00075fc56a")).

Update 2020-12-14: The failureCauses collection is unchanged.

Update:2020-12-15: After testing the new version on the same jenkins master + the same database, I see no difference in the json structure in neither collection. Please try it out in your staging environment.

Thank you t_westling

I updated our server and forced a few BFA failures, and it seems to be working as expected + our tool is OK too. I have yet to look at the schema, but it is reassuring.

One concern, and I'm posting a screenshot here, is the amount of plugins it updated, which is way bigger than I expected. I did get failures too for two of them, but the server restarted successfully. Any concern from your perspective on the updated plugins? I looked at the POM and assume most of them are transient dependencies?

Are all of those from just upgrading to the new BFA version?

I tried upgrading first from Jenkins 2.222.1 to 2263.1, then upgrading BFA from 1.27.0 to the PR version, and got very few unexpected updates.

Yes that's from applying the hpi built with mvn install from your PR. Mind you this server had old Matrix/Slack plugin versions but more interesting are the new libs installed (the ones in green)

On my staging instance I found the plugins sbeaulie mentions:
-Plugin Utilities API
-Bootstrap 4 API
--Font Awesome API
--Popper.js API
--JQuery3 API

But they aren't on my production instance. Looking into it, they appear to be from the latest version of the JUnit plugin which has a dependency on Bootstrap 4 and Plugin Utilities API. The Bootstrap 4 plugin then has dependencies on the rest.

I had upgraded all the plugins (including JUnit) before installing the beta BFA plugin which is when I assume they got installed; after upgrading all plugins I didn't see any new dependencies when I installed BFA. BFA does have a dependency on JUnit though, so I assume that's where the new plugins came from for sbeaulie.

Thanks krtrego for the dependency tree.

That makes me wonder if we need to depend on Junit at all?

Since we have functionality that scans failed test cases, we reference JUnit classes in the code, so yeah, I think we do.

I thought it was coming from core, since it's in hudson.tasks.test.* but I confirm it's the Junit plugin https://github.com/jenkinsci/junit-plugin/tree/master/src/main/java/hudson/tasks/test

Hi,

for the records, we have just upgraded to Jenkins 2.263.1, using BFA 2.0.0-beta-1, and everything seems to be working ok here:

• Indications are being detected ok.
• Information is being sent to mongodb ok.
• The failures management UI seems to be working ok too.

Those are the features we are using and became broken. Glad to have them back to work, yay!

Ciao

tried out 2.0.0-beta-1 with jenkins core  2.249.3 works well for us 

t_westling Thank you so much for your work on this.

There is one problem with this being in beta: It isn't available like the other plugins, you have to sideload it. This makes our support department reluctant to adopt the upgrade to the plugin for this very reason, even if it provides the functionality we want: regex matching for failure causes. We don't care about the graphing.

Since there is no plans to support the graphing functionality, can you please consider releasing it officially? Just disclose it in the release notes: "Graphing functionality removed."

I'm not convinced waiting to release it accomplishes your goal: "This is also the reason that this is released as a beta, to give the users of the graph support some time to adapt." They are still going to have to take the time to adapt regardless of the beta BFA because nobody (graph users or not) can upgrade jenkins with the existing released version of BFA.

It seems to me there is only two choices right now if you do depend on the graphing functionality.

1. Don't upgrade jenkins at all.
2. Take over plugin maintenance and do it yourself

Meanwhile it forces those who don't care about the graphing functionality to also wait while this adapting time takes place (maybe because you use another graphing tool), which after reading the comments here, more users seem fall on this side of things.

1. Don't upgrade jenkins at all because your support people feel that the beta is unsafe or a hassle
2. Upgrade jenkins and side load the beta

If you release this officially then the graph users remain in the same boat (don't upgrade jenkins or do-it-yourself), but life gets easier for those of us who don't care about the graphing functionality since we can simply update it the same way as the rest of the plugins.

Best regards,

Lisa

ps. For other commenters here, it would be interesting to know what people do for the graphing functionality these days either specifically related to failure causes, or for other jenkins job metrics (build duration, and so on.)

pps. Meanwhile I'll continue my lobbying of my support department to install this anyhow

Hi there,

I agree with Lisa's comment, we are in the same situation in my company: we are not relying on the graph feature, and because of the beta version of the plugin, our platform team is reluctant to install it.

Thanks again Tomas for you effort, looking forward the new official release

We, OTOH, do use the graphs.