-
Bug
-
Resolution: Fixed
-
Critical
-
- Jenkins v2.401.3 (LTS), Running as a docker container on an ephemeral linux node
- Sonar plugin v2.16.0 & also v2.16.1
- Builds running on remote jenkins agents, using docker image `jenkins/inbound-agent:3148.v532a_7e715ee3-5-jdk17`
- Reproducible on our test jenkins instance (on another but equivalent node)- Jenkins v2.401.3 (LTS), Running as a docker container on an ephemeral linux node - Sonar plugin v2.16.0 & also v2.16.1 - Builds running on remote jenkins agents, using docker image `jenkins/inbound-agent:3148.v532a_7e715ee3-5-jdk17` - Reproducible on our test jenkins instance (on another but equivalent node)
We have maven builds using the Sonar plugin to wrap the build randomly failing AFTER the sonar analysis has successfully been performed. This means it's not the maven build running the sonar analysis, but something in the sonar plugin that is run AFTER the wrapped build step, which crashes. This happens non-deterministically on just some of the builds for any given build job, but some builds seem to be affected more often than others.
This looks like a JEP-200 issue, but it has only started occurring recently on our main jenkins instance that hasn't been changed/updated lately.
Main jenkins log entry when problem occurs:
Oct 18, 2023 1:37:49 PM WARNING jenkins.security.ClassFilterImpl notifyRejected okhttp3.OkHttpClient in file:/var/jenkins_home/plugins/okhttp-api/WEB-INF/lib/okhttp-4.11.0.jar might be dangerous, so rejecting; see https://www.jenkins.io/redirect/class-filter/
Stack-Trace in the log of affected builds:
[INFO] ANALYSIS SUCCESSFUL, you can find the results at: http://sonar.xxxxx [INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report [INFO] More about the report processing at http://sonar.xxxxx [INFO] Analysis total time: 25.409 s [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for xxxxx 0.0.0-SNAPSHOT: [INFO] [INFO] xxxxx ............................................. SUCCESS [ 32.031 s] [INFO] xxxxx-api ......................................... SKIPPED [INFO] xxxxx-core ........................................ SKIPPED [INFO] xxxxx-remote ...................................... SKIPPED [INFO] xxxxx-archunit .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 34.752 s [INFO] Finished at: 2023-10-18T13:58:57+02:00 [INFO] ------------------------------------------------------------------------ ERROR: Failed to parse POMs java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.maven.MavenModuleSet#buildWrappers for class hudson.maven.MavenModuleSet at hudson.XmlFile.write(XmlFile.java:220) at hudson.model.AbstractItem.save(AbstractItem.java:619) at hudson.model.Job.save(Job.java:194) at hudson.model.AbstractProject.save(AbstractProject.java:288) at hudson.maven.MavenModuleSet.reconfigure(MavenModuleSet.java:1199) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:1019) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:689) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526) at hudson.model.Run.execute(Run.java:1900) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) Caused by: java.lang.RuntimeException: Failed to serialize hudson.maven.MavenModuleSet#buildWrappers for class hudson.maven.MavenModuleSet at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1303) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1292) at com.thoughtworks.xstream.XStream.toXML(XStream.java:1265) at hudson.XmlFile.write(XmlFile.java:213) ... 11 more Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.sonar.SonarBuildWrapper#client for class hudson.plugins.sonar.SonarBuildWrapper at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66) at hudson.util.DescribableList$ConverterImpl.marshal(DescribableList.java:274) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270) ... 24 more Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.sonar.client.HttpClient#okHttpClient for class hudson.plugins.sonar.client.HttpClient at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270) ... 40 more Caused by: java.lang.UnsupportedOperationException: Refusing to marshal okhttp3.OkHttpClient for security reasons; see https://www.jenkins.io/redirect/class-filter/ at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:622) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270) ... 49 more
The root cause is the JEP-200 style failure to serialize the okhttp3.OkHttpClient:
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal okhttp3.OkHttpClient for security reasons; see https://www.jenkins.io/redirect/class-filter/
But I think the actual cause is rather the sonar plugin even trying to serialize (marshal) an HTTP client internal/private field in the first place, so one of those locations:
Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.sonar.SonarBuildWrapper#client for class hudson.plugins.sonar.SonarBuildWrapper
Caused by: java.lang.RuntimeException: Failed to serialize hudson.plugins.sonar.client.HttpClient#okHttpClient for class hudson.plugins.sonar.client.HttpClient
Let me know if I can provide anything else here.
[JENKINS-72201] Sonar plugin fails wrapped build trying to marshal okhttp3.OkHttpClient
Eric Giffon
added a comment - Can you try with the version 2.15 of the plugin? I suspect the issue was introduced with 2.16, although I'm not sure why this serialization happens.
Eric Giffon
added a comment - Can you try with the version 2.15 of the plugin? I suspect the issue was introduced with 2.16, although I'm not sure why this serialization happens. 
Sam Bernet
added a comment - I will try this. Unfortunately I already upgraded to 2.16.1 as part of the investigation, so I can only downgrade to 2.16 through the Jenkins UI (and this is our production instance, so I'll have to do the fiddling on the test instance first, and do stuff outside main business hours).
I will report back here.
Sam Bernet
added a comment - I will try this. Unfortunately I already upgraded to 2.16.1 as part of the investigation, so I can only downgrade to 2.16 through the Jenkins UI (and this is our production instance, so I'll have to do the fiddling on the test instance first, and do stuff outside main business hours).
I will report back here. Sam Bernet
added a comment - I have downgraded our test instance to 2.15 sonar plugin version, and indeed it works with that one. I have now run the previously failing job successfully 4 times in a row.
To double-check, I upgraded the plugin to 2.16.1 again on that very same instance, and run the same job again - but now of course it doesn't fail anymore 
But running a few OTHER builds, I had one failing again with the stack trace noted here.
Because builds don't consistently fail, I can't tell 100% 2.15 does not have the issue, but my results at least suggest that (no failures with 2.15 so far).
So I agree this can probably be pinpointed to have been introduced in 2.16 (and is also present in 2.16.1).
Going forward, would you suggest I downgrade the plugin on our production instance as well ericg , or is this being looked into actively? Is there something else I can do / provide / test to bring this forward?
Sam Bernet
added a comment - I have downgraded our test instance to 2.15 sonar plugin version, and indeed it works with that one. I have now run the previously failing job successfully 4 times in a row.
To double-check, I upgraded the plugin to 2.16.1 again on that very same instance, and run the same job again - but now of course it doesn't fail anymore But running a few OTHER builds, I had one failing again with the stack trace noted here.
Because builds don't consistently fail, I can't tell 100% 2.15 does not have the issue, but my results at least suggest that (no failures with 2.15 so far).
So I agree this can probably be pinpointed to have been introduced in 2.16 (and is also present in 2.16.1).
Going forward, would you suggest I downgrade the plugin on our production instance as well ericg , or is this being looked into actively? Is there something else I can do / provide / test to bring this forward? Sam Bernet
added a comment - We now reverted prod to use 2.15 to remove the pain from our devs. This is looking fine so far.
We still have the test instance available to reproduce / test / fix stuff.
Sam Bernet
added a comment - We now reverted prod to use 2.15 to remove the pain from our devs. This is looking fine so far.
We still have the test instance available to reproduce / test / fix stuff. Eric Giffon
added a comment - Thank you for the confirmation algoripper
I didn't manage to reproduce the issue, but I created a ticket on Sonar issue tracking to prevent it: SONARJNKNS-369
It should be fixed for the next release. It isn't planned yet, so I would suggest staying with 2.15 for now.
Eric Giffon
added a comment - Thank you for the confirmation algoripper
I didn't manage to reproduce the issue, but I created a ticket on Sonar issue tracking to prevent it: SONARJNKNS-369
It should be fixed for the next release. It isn't planned yet, so I would suggest staying with 2.15 for now. Sam Bernet
added a comment - Thanks for the heads-up and for promoting this to a SONAR issue ericg 
. We will stay with 2.15 on PROD, and use our staging to test new releases when they get available.
Sam Bernet
added a comment - Thanks for the heads-up and for promoting this to a SONAR issue ericg . We will stay with 2.15 on PROD, and use our staging to test new releases when they get available. Eric Giffon
added a comment - Version 2.17 was released, it contains a fix for this issue. 
Feel free to adapt priority as needed. I have set to critical (for the context of this plugin) as this randomly fails builds, even though the actual sonar analysis has run just fine.
Possible workaround on a more broad level include skipping sonar analysis, removing sonar plugin, or just re-running the job "until it works" (as it's an intermittent failure). In our case with large sets of downstream modules that are triggered by maven snapshot builds, this means whole build chains are stopped inadvertently.
For the context of this plugin, I think it's still critical, though, as I'm not aware of a workaround that doesn't involve dropping sonar from the build.