-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
SonarQube Scanner for Jenkins Version 2.14
Jenkins 2.348
Summary
Sonarqube is unstable, and many invalid requests were sent to sonar server, making it overwhelmed.
Symptom
We see in a period of time, some users encountered the below errors from jenkins output:
Output from jenkins jobs:
10:17:59 ERROR: hudson.AbortException: org.sonarqube.ws.client.HttpException: Error 500 on https://mia-sonarqube.citrite.net/api/qualitygates/project_status?analysisId=AYDxR88IXL7txuHUc1Gy : <html>
10:17:59 <head><title>500 Internal Server Error</title></head>
10:17:59 <body>
10:17:59 <center><h1>500 Internal Server Error</h1></center>
10:17:59 <hr><center>nginx</center>
10:17:59 </body>
10:17:59 </html>
10:17:59
We see lots errors in nginx logs proxy.error.log, indicating the nginx is struggling:
/var/log/nginx/proxy.error.log on Sonarqube Server 2022/05/20 10:02:34 [alert] 2206486#2206486: *5776138 socket() failed (24: Too many open files) while connecting to upstream, client: 10.73.56.17, server: sonarqube.citrite.net, request: "GET /api/ce/task?id=AXfMn5u9YOE9PTUmvnJ5 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/ce/task?id=AXfMn5u9YOE9PTUmvnJ5", host: "mia-sonarqube.citrite.net" 2022/05/20 10:02:34 [alert] 2206486#2206486: *5776143 socket() failed (24: Too many open files) while connecting to upstream, client: 10.73.56.17, server: sonarqube.citrite.net, request: "GET /api/ce/task?id=AXgBYIQ8azMRLrQ9va1h HTTP/1.1", upstream: "http://127.0.0.1:9000/api/ce/task?id=AXgBYIQ8azMRLrQ9va1h", host: "mia-sonarqube.citrite.net" 2022/05/20 10:02:34 [alert] 2206486#2206486: *5776149 socket() failed (24: Too many open files) while connecting to upstream, client: 10.73.56.17, server: sonarqube.citrite.net, request: "GET /api/ce/task?id=AYADebOOIC-rwevTC-JS HTTP/1.1", upstream: "http://127.0.0.1:9000/api/ce/task?id=AYADebOOIC-rwevTC-JS", host: "mia-sonarqube.citrite.net"
We see lots invalid requests showing in /var/log/nginx/proxy.access.log, they are from jenkins servers, probably initiated by jenkins plugin related to sonar
proxy.access.log on Sonarqube Server 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AX-R2RBvuLFG2NT87PEn HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.009 0.008 . 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AX9sQd_puLFG2NT87Lyw HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.011 0.012 . 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AX9sQd_puLFG2NT87Lyw HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.008 0.008 . 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AYAIkaMIgBoDDGrXm8e3 HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.008 0.008 . 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AYAIkaMIgBoDDGrXm8e3 HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.010 0.008 . 10.73.56.17 - a2438de2b642823e800d43011c30512d5f895ca3 [26/May/2022:00:00:36 +0000] "GET /api/ce/task?id=AX_dp1oNgBoDDGrXm42o HTTP/1.1" 404 82 "-" "Scanner for Jenkins" 0.010 0.012 .
We see similar issue reported here: https://community.sonarsource.com/t/errors-in-the-jenkins-log-while-execution-of-sonarscan/34236
Facts
Jenkins plugin we are using:
SonarQube Scanner for Jenkins Version 2.14
Question
1. Are these expected?
2. Why is "Scanner for Jenkins" keeps acquiring non exist tasks
From logs in jenkins, it only has something like:
2022-05-25 17:25:04 WARNING hudson.plugins.sonar.client.SQProjectResolver resolve Error fetching project information org.sonarqube.ws.client.HttpException: Error 500 on https://mia-sonarqube.citrite.net/api/ce/task?id=AXyiX4IkaCRXucfOiSwv : <html>^M <head><title>500 Internal Server Error</title></head>^M <body>^M <center><h1>500 Internal Server Error</h1></center>^M <hr><center>nginx</center>^M </body>^M </html>^M at org.sonarqube.ws.client.BaseResponse.failIfNotSuccessful(BaseResponse.java:36) at hudson.plugins.sonar.client.HttpClient.getHttp(HttpClient.java:38) at hudson.plugins.sonar.client.WsClient.getCETask(WsClient.java:51) at hudson.plugins.sonar.client.SQProjectResolver.requestCETaskDetails(SQProjectResolver.java:98) at hudson.plugins.sonar.client.SQProjectResolver.resolve(SQProjectResolver.java:75) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:76) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:52) at hudson.plugins.sonar.action.SonarProjectActionFactory.createProjectPage(SonarProjectActionFactory.java:117) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:83) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:42) at hudson.model.Actionable.createFor(Actionable.java:114) at hudson.model.Actionable.getAction(Actionable.java:329)